omlx

oMLX runs large language models on your Mac and manages the whole thing from the menu bar. Pick a model, hit start, and you have a local OpenAI-compatible API at localhost:8000 that Claude Code, Codex, or any OpenAI client can point at. Apache 2.0, no account, no paid tier.

The engineering underneath is more serious than the menu bar suggests. Continuous batching handles concurrent requests, and a two-tier KV cache keeps hot blocks in RAM and spills cold ones to SSD, so a long context survives a server restart instead of being recomputed from scratch. One process serves text models, vision models, OCR, embeddings, and rerankers, with model pinning, per-model idle timeouts, sampling settings, and a memory ceiling all editable from the admin dashboard.

Solo developers running a local coding assistant get the most out of it, and a small team can share one big Mac over the API. Ollama (ollama/ollama) is easier to hand to a teammate and runs on any hardware. llama.cpp (ggml-org/llama.cpp) gives you more low-level control. oMLX wins specifically on Apple Silicon.

The catch: Apple Silicon and macOS 15 or newer, full stop. And the fast native Metal kernels for the GLM and MiniMax families do not build from a plain pip install. Without full Xcode you silently fall back to a path the project measures at roughly 30x slower on prefill. Use the official DMG, which ships them precompiled.