vLLM

vLLM is the open source server teams use to serve language models to lots of users at once. Its PagedAttention memory management and continuous batching squeeze far more requests out of a GPU than a naive setup: the original benchmarks showed 14 to 24 times the throughput of plain Hugging Face Transformers and 2.2 to 3.5 times an optimized server like TGI. Apache 2.0, and a PyTorch Foundation project.

It exposes an OpenAI-compatible API and runs on NVIDIA, AMD ROCm (with prebuilt wheels), Intel XPU, Google TPUs and Apple Silicon through a plugin. Getting the best out of it means matching CUDA versions, sizing GPU memory and tuning batch settings. A 70B model at full precision needs several data-center GPUs; quantized models shrink that.

The software is free; your bill is GPUs. On-demand A100s run roughly $1.59 to $2.79 an hour on RunPod and Lambda, and an 8-GPU AWS p4d instance is about $22 an hour. Solo devs running one model for themselves should use Ollama (ollama/ollama) or llama.cpp instead. Teams serving real traffic: this is the default choice.

The catch: it's a big, fast-moving codebase with a matching security record. 33 advisories landed between August and early October 2026, including remote code execution through request parameters, patched in 0.31.0. Pin versions, upgrade deliberately, and keep a gateway in front of it. Its creators also founded Inferact in January 2026, so watch for a commercial layer.