vLLM
High-throughput LLM inference and serving engine
The Lens
By Erik Loyd, SaaS CEO and former COO/CFO of an AWS Premier Partner.
Updated Aug 2026
vLLM is the fastest way to serve open-weight LLMs on your own hardware. It takes a model like Llama or Mistral and puts an OpenAI-compatible API in front of it, squeezing maximum throughput out of your GPUs.
What's free: everything. Apache 2.0 license. The entire inference engine, all optimizations (PagedAttention, continuous batching, tensor parallelism), the OpenAI-compatible API server. All free.
vLLM's key innovation is PagedAttention, which manages GPU memory the way operating systems manage RAM, in pages instead of contiguous blocks. The result: 2-4x more throughput than naive inference. It's become the default serving engine for self-hosted LLMs.
The catch: you need serious GPUs. Running a 70B parameter model requires 2-4 A100 GPUs ($1-2/hr on cloud, or $10K+ each to buy). Even a 7B model needs a decent GPU with 16GB+ VRAM. vLLM is free but the hardware is emphatically not. And it's optimized for NVIDIA GPUs. AMD ROCm support exists but is second-class.
Free vs Self-Hosted vs Paid
fully freeWhat's Free
Everything. Apache 2.0 license. All features, all optimizations, no restrictions.
The Hardware Bill (This Is Your Real Cost)
- 7B model (Llama 3.1 7B): 1x GPU with 16GB+ VRAM. Cloud: ~$0.50-1.00/hr. Buy: RTX 4090 ~$1,600.
- 70B model (Llama 3.1 70B): 2-4x A100 80GB GPUs. Cloud: $4-8/hr (~$3,000-6,000/mo 24/7). Buy: ~$40K-80K.
- 405B model: 8x A100 or H100. Cloud: $16-32/hr (~$12K-24K/mo). Buy: you don't want to know.
Cloud GPU Options
- RunPod: A100 80GB at ~$1.64/hr. Good for experimentation.
- Lambda Labs: A100 at ~$1.10/hr. Better for sustained use.
- AWS (p4d/p5): $12-40/hr. Enterprise-grade, enterprise-priced.
vs Paying for API Access
- OpenAI GPT-4o: $2.50-10/1M tokens. No hardware to manage.
- Self-hosted Llama 70B via vLLM: ~$0.20-0.50/1M tokens at scale. But you're managing infrastructure.
When Self-Hosting Makes Sense
When: data privacy is non-negotiable, you're processing millions of tokens/day (cost crossover), or you need custom model fine-tuning. When not: you're processing <100K tokens/day (API is cheaper), or you don't have GPU expertise.
Software is free. Hardware costs $0.50-32/hr in the cloud. Self-hosting beats API pricing only at massive scale or when data privacy is non-negotiable.
What to do by team size
- Solo
- Only if you have GPU access and a specific privacy/cost reason. Otherwise, use OpenAI/Anthropic APIs
- Small team
- Makes sense if you're building an AI product and need to control inference costs at scale
- Medium team
- Strong fit if you're processing millions of tokens/day. The cost savings vs APIs are real at this volume
- Large team
- Standard infrastructure for AI teams. Run on dedicated GPU clusters with auto-scaling
Get tools like this every Wednesday
One featured tool, three on the radar. No fluff.
Similar Tools
Get up and running with Kimi-K2.5, GLM-5, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
Unsloth is a local UI for training and running Kimi K3, Gemma 4, Qwen3.6, DeepSeek-V4, GLM and other models.
LLM inference in C/C++
Load almost any open AI model in a few lines of Python. The standard library for running and fine-tuning models from the Hugging Face Hub, now PyTorch only.
SDK and proxy to call 100+ LLM APIs in OpenAI format
Open-source AI engine, run any model locally
A low score is not a verdict on quality. Young and niche tools start low by design. How we calculate scores
Trust Signals
About
- Stars
- 92,070
- Forks
- 22,353
Explore Further
More tools in the directory
openclaw
Your own personal AI assistant. Any OS. Any Platform. The lobster way. 🦞
389.9k ★everything-claude-code
The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.
264.9k ★hermes-agent
The agent that grows with you
247.9k ★Featured in The Open Source Drop #16