Roadmap

From local runtime to the ten-minute base

v0.1 made one machine boring: a laptop, 127.0.0.1, llama.cpp. The road after it makes the same binary run identically on any GPU box — six steps, each one shipping on its own. Recorded 2026-08-20 in vision.md.

v0.1 — the local runtime

SHIPPED · 2026-08

Headless daemon, two OpenAI-compatible APIs over one pipeline, full lifecycle over HTTP, supervised llama.cpp, verified catalog, thin CLI. Since then (v0.4.0): image input for vision models, speech to text through whisper.cpp, 20 catalog models. Release notes →

Remote-safe mode

PLANNED

Bind beyond loopback only with an API key — on /v1/* and the Management API both. CLI grows --host, so omg --host vps run … makes today's runtime usable on a GPU VPS. The smallest change that matters.

Linux packaging

PLANNED

Release binaries, install.sh, a Dockerfile (llama.cpp first) and a systemd unit — the fresh-VPS path becomes copy-paste.

Hardware resolver & snapshot downloads

PLANNED

Detect every GPU — count, VRAM, free memory, CUDA version — and resolve recipe × hardware → variant, with "will not fit, needs X" said before any download. Multi-file safetensors snapshots, parallel chunks, mirrors, resume.

vLLM backend

PLANNED

crates/runtime_vllm behind the same backend trait — this is what makes "any model" true: the Transformers fallback gives day-0 coverage while GGUF conversions lag. Backend registry, per-model choice; SGLang later behind the same interface.

Recipes as data

IN PROGRESS · STARTED 2026-08-20

One YAML per model: repo, quant variants mapped to VRAM, engine args, tool parser, a smoke test. Schema v1, validation and example recipes are in; the resolver and the catalog migration are next. A default recipe means most models need no recipe at all. Announcement →

Pre-baked image & the ten-minute matrix

PLANNED

One CUDA container with vLLM, llama-server and the runtime pre-baked — then a nightly SLA matrix on a real GPU: pull → start → smoke test, timed, published. The matrix is both the quality gate and the proof of the promise.

The promise, as a budget

Ten minutes, itemized

Install the base	≤ 1 min
Inference backend (pre-baked)	≈ 0
Weights — 7B FP16 / 32B / 70B AWQ	2 / 9 / 6 min
Start + warm-up	sec – 3 min

Honest form: ≤ ~50 GB of weights on a ≥1 Gbps box with the pre-baked image → under ten minutes. Bigger weights are a network fact — the runtime shows a correct ETA instead of hiding it. Out of scope: 200B+ multi-GPU MoE tuning; unsupported architectures fail fast with the reason.

Constants

What does not change

One inference pipeline — wire formats stay at the boundary.
Explicit lifecycle through the model manager.
The CLI stays a thin HTTP client.
No unimplemented API fields claimed.
Fast offline tests; real GPUs behind OHMYGPU_E2E=1.
Local-first: a laptop on 127.0.0.1 stays the default experience.

Backends

One trait, many engines

Every engine lives behind the same RuntimeBackend trait — supervised subprocess, same lifecycle, same APIs — and a recipe picks the backend per variant. The bar for adding one: it must make more models runnable without breaking the ten-minute budget.

Engine	Runs	Why — and why not yet	Status
llama.cpp	GGUF · Metal / Vulkan / CPU	Consumer hardware and laptops; auto-installed per platform. The local-first default.	SHIPPED · V0.1
whisper.cpp	ggml whisper models · Metal / CPU	Speech to text behind /v1/audio/transcriptions; the same supervised subprocess pattern, picked by a model's kind.	SHIPPED · V0.4
vLLM	safetensors + Transformers fallback · CUDA	What makes "any model" true — day-0 coverage while GGUF conversions lag; GPU-box throughput, tensor parallel.	PLANNED · STEP 4
SGLang	safetensors · CUDA	Same seat as vLLM with stronger structured output and high-concurrency serving. Waits behind the same trait.	LATER
MLX	mlx-lm · Apple Silicon	Often faster than llama.cpp on M-series. Earns its slot if the user base stays mac-heavy.	CONSIDERING
Candle	Rust-native · no Python	Closest to this project's soul — but per-model hand adaptation contradicts recipes-as-data. Kept in archive/; revisit as the ecosystem matures.	ARCHIVED
TensorRT-LLM	compiled engines · NVIDIA	Peak NVIDIA performance, but engine compilation alone can eat the ten-minute budget.	NOT PLANNED

Landscape

Honest comparisons

Every developer evaluating OhMyGPU is already looking at these. The answers, without spin:

vs Ollama

Why not just Ollama?

Chatting on your own machine? Use Ollama — it's excellent at that.
OhMyGPU serves apps that embed a runtime: headless, private data dir, graceful shutdown.
…and rented GPU boxes: vLLM backend, multi-GPU, a measured ten-minute path.
Models straight from Hugging Face via open YAML recipes — no registry, no Modelfile.
Responses API as canonical; honest subset — 400 unsupported, no pretending.

Ollama is the front door for local chat. OhMyGPU is the base under applications and GPU boxes.

vs the engines

Why not run vLLM or llama-server directly?

You can — OhMyGPU runs exactly those engines underneath.
What you'd hand-roll instead: weights download + resume, hardware fit checks, engine flags per model.
Process supervision, crash recovery, a lifecycle your app can poll.
One API that stays stable when you switch engines.
That glue is the product — the part nobody enjoys rewriting per project.

Engines compute. The base operates.

vs the cousins

GPUStack, Xinference, RamaLama?

The closest cousins — each covers parts of multi-backend model management.
OhMyGPU: one static Rust binary, no Python in the control plane.
Embeddable and headless by design.
Recipes anyone can contribute, as plain data.
Ten-minute promise measured nightly in public — not asserted.

The proof ships as a test matrix, not a benchmark chart.