sglang

SGLang has become the default engine for serving open-weight LLMs in production at scale. It runs from a single GPU up to large clusters, with prefix caching, speculative decoding, and day-zero support for new model releases. The throughput gains over the previous generation of servers are large, and the adoption backs it up: it reportedly runs across hundreds of thousands of GPUs at xAI, NVIDIA, and the major clouds. Apache-2.0, free.

Setup is the standard NVIDIA inference stack, but hardware support is broad now: NVIDIA, AMD, Intel, Google TPUs, and Ascend NPUs. Models span Llama, Qwen, DeepSeek, GLM, Mistral, diffusion models, and most of Hugging Face. It is OpenAI-API compatible, so existing clients drop in without rewrites.

Solo developers and small teams running open-weight models: this is one of the strongest options on the shelf, especially on DeepSeek or with heavy agentic workloads. Large teams running production inference at scale: you are very likely already evaluating it.

The catch is that serious inference is still serious work. Cold starts, KV cache tuning, and multi-node setups need real engineering. SGLang gives you a faster engine; it does not hand you a managed platform, and the operational burden of running one is still yours.