Two things applications kept asking for are now in the runtime — without changing the shape of anything. Same binary, same lifecycle, same two inference APIs; the models just understand more.
Images in. The vision models in the catalog — Qwen2.5-VL 3B and 7B, Gemma 3 4B, SmolVLM 256M — take image input through both APIs: input_image parts in /v1/responses, image_url parts in /v1/chat/completions. An image is a data: URL or an http(s) URL; the runtime fetches it, inlines it, and never lets the backend fetch anything. png, jpeg, gif, webp and bmp, up to 20 MB. A non-vision model answers 400 unsupported, exactly as the catalog flag says.
omg model pull qwen2.5-vl-3b-instruct && omg run qwen2.5-vl-3b-instruct
curl http://127.0.0.1:10692/v1/chat/completions -H 'Content-Type: application/json' -d '{
"model": "qwen2.5-vl-3b-instruct",
"messages": [{"role": "user", "content": [
{"type": "text", "text": "Read all the text in this image."},
{"type": "image_url", "image_url": {"url": "data:image/png;base64,<BASE64>"}}
]}]}'
Speech to text out. whisper.cpp joins llama.cpp as a second supervised engine, behind the same RuntimeBackend trait and the same pull → start → serve lifecycle. POST /v1/audio/transcriptions takes a multipart upload — wav, mp3, m4a/aac, flac, ogg-vorbis, decoded and resampled by the runtime itself, no ffmpeg — and answers in json, text, verbose_json (segments with timestamps), srt or vtt. Four multilingual models ship in the catalog: whisper-tiny, whisper-base, whisper-small and whisper-large-v3-turbo.
omg model pull whisper-base && omg run whisper-base
curl http://127.0.0.1:10692/v1/audio/transcriptions \
-F file=@meeting.mp3 -F model=whisper-base -F language=zh -F response_format=verbose_json
What did not change is the point. Every model carries a kind — llm or whisper — that picks its backend and the API that serves it. A chat request to a whisper model, or a transcription request to an LLM, is a clean 400 unsupported, not a confusing failure three layers down. Text, tool calls and images still travel one internal pipeline; the engines only ever see inlined data.
The catalog now holds 20 models: 12 text, 4 vision, 4 speech. Also in this release line (v0.3.x): release binaries are built and published on every tag push, with version-less asset names so releases/latest/download/… always points at the newest build.