New open models ship every day and most never get a hosted API. Anyone who wants them has to self-deploy — GPU image, inference engine, weights download, engine flags, reverse proxy, assembled differently for every model. The direction after v0.1 is to make that one command:
Drop OhMyGPU onto any machine with a GPU, point it at a model, and have an OpenAI-compatible endpoint serving it within ten minutes — for any open model, including one released today.
A promise like that is only worth stating as a budget. On a fresh GPU VPS with ~1 Gbps of network:
| Step | Target |
|---|---|
| Install the base | ≤ 1 min — static binary or one image |
| Install the inference backend | ≈ 0 pre-baked in the image; 3–5 min on bare metal |
| Download weights | bandwidth-bound — 7B FP16 ≈ 2 min · 32B ≈ 9 min · 70B AWQ ≈ 6 min |
| Start + warm-up | llama.cpp seconds; vLLM 1–3 min with good defaults |
So the honest version: ≤ ~50 GB of weights, on a ≥1 Gbps box, with the pre-baked image → under ten minutes. Bigger weights are a network fact, not a runtime defect — the runtime's job is a correct ETA, not a hidden one. Non-goals: 200B+ MoE models that need multi-GPU tuning, and architectures no engine supports yet — those must fail fast with a clear reason, not hang.
Six pillars, each shipping on its own: remote-safe mode (bind + API key), Linux packaging, multi-GPU detection with snapshot downloads, a vLLM backend behind the same trait, recipes as data, and finally the pre-baked CUDA image with a nightly, published ten-minute test matrix — the proof, not the pitch. Local-first does not change: a laptop on 127.0.0.1 with llama.cpp stays the default experience.