One archive per platform

Pinned versions live at releases/download/<tag>/…, and every release carries SHA256SUMS.txt. The binaries are not code-signed yet — if macOS quarantines a browser download, run xattr -dr com.apple.quarantine <dir> (curl downloads are not quarantined). Or build from source with a Rust toolchain, no GPU toolchain needed: make build.

omg — a thin client, by design

Every command except serve and config is a call to the Management API. Add --json for machine-readable output; point at a non-default runtime with --url or OHMYGPU_URL.

omg serve [--host H] [--port P] run the runtime in the foreground
omg status runtime + backend + models summary
omg hardware detected platform, CPU, memory, GPU
omg model list | catalog installed models / curated catalog
omg model pull <ref> [--id X] catalog id, hf:owner/repo/file.gguf, or URL
 [--mmproj FILE] [--kind llm|whisper] vision projector · force the kind
omg model rm | info <id> delete / inspect
omg run <id> [--context-length N] start (also: --gpu-layers, --threads)
omg stop <id> stop a running model
omg shutdown graceful shutdown, stops all models
omg config [key [value]] read / write config.toml

An OpenAI-compatible subset, stated exactly

Only what is listed is implemented. Unknown fields are ignored; unsupported features return 400 with "code": "unsupported" — never silently wrong output.

POST /v1/responses — canonical

Not supported (400): previous_response_id, background, hosted tools, file inputs, non-text formats. Response storage is not implemented; store is always false.

POST /v1/chat/completions — ecosystem

Not supported (400): n > 1, audio content parts, non-text response_format.

POST /v1/audio/transcriptions — speech to text

Not supported (400): stream, word timestamps, /v1/audio/translations. A chat request to a whisper model, or a transcription request to an LLM, answers 400 unsupported. GET /v1/models lists installed models in the OpenAI shape with extra state and kind fields. Legacy /v1/completions is intentionally absent.

Errors — the OpenAI envelope, everywhere

The whole lifecycle over HTTP

Everything the CLI can do, your application can do — every state, download percentage and failure reason is visible. Nothing requires the CLI.

One directory, one file

Everything lives under ~/.config/ohmygpu — or $OHMYGPU_HOME / --data-dir, so a bundling application can give the runtime a private data directory.

# config.toml — the defaults, annotated
[daemon]
host = "127.0.0.1" # local only; no auth yet, keep it that way
port = 10692

[inference]
auto_start = false # 409 for stopped models, or start on demand

[backend.llamacpp]
auto_install = true # fetch the official release if not found
release = "latest" # or pin a tag like "b10437"
context_length = 8192
# server_path = "/path/to/llama-server" # or OHMYGPU_LLAMA_SERVER
startup_timeout_secs = 600

Environment overrides: OHMYGPU_HOME · OHMYGPU_HOST · OHMYGPU_PORT · OHMYGPU_LLAMA_SERVER · OHMYGPU_WHISPER_SERVER · HF_TOKEN · OHMYGPU_LOG (e.g. info,llamacpp=debug to see the engine's own logs). Managed llama.cpp builds cover Metal on macOS, CPU/Vulkan on Linux and Windows; for CUDA on Linux point server_path at your own build.

Adapting a model is editing a data file

A recipe is one YAML file per model: where the weights live, which backend runs them, quant variants mapped to hardware requirements, and a smoke test. Hand-authored recipes are YAML (comments, multi-line templates); the wire format is JSON; one schema behind both — recipe-v1.json.

schema_version: 1
id: qwen2.5-7b-instruct
family: qwen2.5
capabilities: { tools: true }
backend: llamacpp
source: { hf: bartowski/Qwen2.5-7B-Instruct-GGUF }
chat: { tool_call_parser: hermes }

variants:
 - name: q4_k_m
 source: { file: Qwen2.5-7B-Instruct-Q4_K_M.gguf }
 size_gb: 4.7
 requires: { vram_gb: 6, ram_gb: 8 }
 - name: bf16 # same model, GPU-box variant
 backend: vllm
 source: { hf: Qwen/Qwen2.5-7B-Instruct }
 requires: { vram_gb: 20, min_compute_capability: "8.0" }

tests:
 - prompt: "Reply with exactly: pong"
 expect_contains: pong

The minimal recipe is an id and a source — backend and format are inferred. A default recipe (vLLM with the Transformers fallback, once the vLLM backend lands) means most models will need no recipe at all; the compiled-in catalog migrates to recipes/*.yaml as this ships. Status: schema, validation and examples are in the tree; the hardware resolver is next.

Ship it inside your application

The runtime is built to be bundled: no UI, no global state outside its data dir, structured logs on stderr, graceful shutdown that stops its engine children.

# 1 — launch with a private data dir and port
ohmygpu-runtime --data-dir /your/app/data --port 17742
# 2 — wait for readiness
GET /ohmygpu/v1/health → {"status":"ok", …}
# 3 — ensure model exists, start it, infer
POST /ohmygpu/v1/models/pull → 202, poll download progress
POST /ohmygpu/v1/models/{id}/start?wait=true
POST /v1/responses
# 4 — on quit
POST /ohmygpu/v1/shutdown (or SIGTERM)

By default a request for a stopped model returns 409 model_not_running — deliberate, so your app owns the start moment. Set inference.auto_start = true to start on first request instead.

Free, open source — and staying that way

OhMyGPU is Apache-2.0 with no commercial edition planned. The most valuable contribution is the day-to-day one: model adaptations as recipes — and honest issue reports from real hardware.

$ make test # unit + API tests, mock backend — fast, no GPU or network
$ make check # fmt + clippy
$ OHMYGPU_E2E=1 cargo test -p ohmygpu_daemon --test e2e_llamacpp -- --ignored
 # real llama.cpp + a small model, ~480 MB download

Architecture notes and the llama.cpp-vs-Candle decision are in docs/architecture.md. To propose a model: open a PR adding one file under recipes/ that validates against the schema and passes its own smoke test. Each accepted adaptation gets its post on the news page.