sglang

SGLang has become one of the default engines for serving open-weight LLMs in production at scale. It runs from a single GPU up to large clusters, with prefix caching, speculative decoding, and day-zero support for new model releases. The project says it runs on hundreds of thousands of GPUs, at xAI, NVIDIA and the major clouds among others. Apache-2.0, free.

Install is a uv pip install, with builds for NVIDIA, AMD, Intel, Google TPUs and Ascend NPUs. Models span Llama, Qwen, DeepSeek, Kimi, GLM, Mistral, diffusion models, and most of Hugging Face. It is OpenAI-API compatible, so existing clients drop in without rewrites, and it doubles as a rollout backend for reinforcement learning training.

Solo developers and small teams running open-weight models: this is one of the strongest options on the shelf, especially on DeepSeek or with heavy agentic workloads. Large teams running production inference at scale: you are very likely already evaluating it.

The catch is that serious inference is still serious work. Cold starts, KV cache tuning, and multi-node setups need real engineering. SGLang gives you a faster engine, not a managed platform. RadixArk, a venture-funded company among its maintainers, is building one on top, with no public pricing yet.