The runtime between your application and any GPU.
OhMyGPU runs open models behind OpenAI-compatible APIs and owns the GPU runtime, model files and lifecycle. The same binary serves a laptop on 127.0.0.1 and a rented GPU box — your application just talks HTTP.
A headless local runtime on llama.cpp and whisper.cpp: macOS, Linux and Windows, OpenAI-compatible /v1 on 127.0.0.1 — text, tool calls, image input and speech to text — a verified catalog of 20 models plus any Hugging Face GGUF. The ten-minute GPU-box promise is the roadmap, stated with its budget — not fine print.
# install: two binaries, nothing else $ omg serve $ omg model pull qwen2.5-7b-instruct qwen2.5-7b-instruct 4.7 GB ▓▓▓▓▓▓▓▓░░ 82% $ omg run qwen2.5-7b-instruct state: running · http://127.0.0.1:10692 $ curl localhost:10692/v1/responses \ -d '{"model":"qwen2.5-7b-instruct", "input":"Why is the sky blue?"}'
The runtime is the product. Everything else is a client.
| Component | Role |
|---|---|
| ohmygpu-runtime | The product. Inference APIs, Management API, model lifecycle, backend supervision. |
| ohmygpu · omg | The CLI — an administrative client of the runtime. Thin by design. |
| future GUI | An optional client. Nothing in the runtime depends on a UI. |
| third-party apps | The primary consumers. They talk to the runtime over HTTP. |
OhMyGPU is not a chat app, a desktop app, an agent framework or a cloud platform. It is the boring, reliable local runtime underneath those things.
Your code says ensure model exists → start model, then uses either inference endpoint. Every state, download percentage and failure reason is visible over HTTP — nothing requires the CLI.
API referenceWhat your app no longer has to manage
/v1/responses is the canonical surface; /v1/chat/completions keeps the ecosystem working. Both reach the same model through one internal pipeline — verified against the official OpenAI SDK.
not_installed → downloading → installed → starting → running — plus error with the reason and the tail of the engine log. States, progress and failures are all plain HTTP.
Adapting a model is editing a YAML file — repo, quant variants, VRAM requirements, engine args, a smoke test — not writing code. Anyone can contribute one; the catalog grows in public.
Any model, any GPU box, in ten minutes.
New open models ship every day; most never get a hosted API. Using them means hand-assembling GPU image, engine, weights, flags and proxy — differently for every model. The goal after v0.1: drop OhMyGPU onto any machine with a GPU, point it at a model — including one released today — and have an OpenAI-compatible endpoint inside ten minutes.
Stated as a budget, not a slogan: base install ≤ 1 min, backend pre-baked ≈ 0, weights bandwidth-bound with an honest ETA, warm-up ≤ 3 min — and once the nightly test matrix ships, the ten minutes get measured in public, not asserted.
Verified models, ready to pull
| ID | Quant | Size | Capabilities |
|---|---|---|---|
| qwen2.5-7b-instruct | Q4_K_M | 4.7 GB | TOOLS |
| llama-3.1-8b-instruct | Q4_K_M | 4.9 GB | TOOLS |
| qwen3-4b-instruct | Q4_K_M | 2.5 GB | TOOLS |
| qwen2.5-vl-7b-instruct | Q4_K_M | 4.7 GB | VISION |
| whisper-large-v3-turbo | q8_0 | 0.87 GB | SPEECH |
Point it at a model.
Free, open source, Apache-2.0 — and staying that way.