LLM Inference

8 open source tools compared. Sorted by stars — scroll down for our analysis.

Tool	Stars	Velocity	Language	License	Score
ollama Get up and running with Kimi-K2.5, GLM-5, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.	166.1k	—	Go	MIT License	100
llama.cpp LLM inference in C/C++	99.3k	—	C++	MIT License	85
vLLM High-throughput LLM inference and serving engine	74.3k	—	Python	Apache License 2.0	82
text-generation-webui Local LLM interface with text, vision, and training	46.4k	—	Python	GNU Affero General Public License v3.0	71
LocalAI Open-source AI engine, run any model locally	44.4k	—	Go	MIT License	79
MLX Array framework for Apple silicon	24.8k	+176/wk	C++	MIT License	79
omlx LLM inference server with continuous batching and SSD caching for Apple Silicon, managed from the macOS menu bar.	6.8k	+1085/wk	Python	Apache License 2.0	73
flash-moe Running a big model on a small laptop	2.0k	+1962/wk	Objective-C	—	58

Our Analysis

ollama166.1k★

Ollama made running local LLMs as simple as `ollama run llama3`. One command, model downloads, inference starts. No Python environments, no CUDA debugging, no config files. It's Docker for LLMs, and that's exactly why it has 166k stars. Under the hood it's llama.cpp with a clean API layer. LM Studio gives you a GUI for model exploration but isn't built for integration. llama.cpp itself offers maximum control but requires you to enjoy compiling C++. For most indie hackers building AI features into products, Ollama's REST API is the fastest path from zero to local inference. If you're prototyping AI features, building offline-capable apps, or just tired of paying OpenAI for every test run, use Ollama. The API is OpenAI-compatible, so switching between local and cloud is a one-line change. The catch: it's single-user optimized. If you need to serve multiple concurrent users in production, vLLM or TGI are better fits. And model quantization choices are opaque — you get what Ollama packages, not fine-grained control.

llama.cpp99.3k★

llama.cpp is the project that proved you don't need a data center to run an LLM. Pure C/C++ inference for large language models — no Python, no PyTorch, no CUDA requirement. It runs Llama, Mistral, Phi, and dozens of other models on CPUs, Apple Silicon, and consumer GPUs. The engine behind nearly every local AI app. If you want to run AI models locally — for privacy, cost savings, or offline use — llama.cpp is the foundation everything else is built on. Ollama wraps it in a friendly CLI. LM Studio wraps it in a GUI. vLLM is faster for GPU serving but Python-only. ExLlamaV2 squeezes more performance from NVIDIA GPUs. Best for developers building local AI products or anyone who wants to understand how LLM inference actually works at the metal level. The catch: it's C/C++, so building from source and debugging isn't for everyone. Model quantization tradeoffs (quality vs. speed vs. memory) require experimentation. Performance tuning is hardware-specific. And the project moves so fast that tutorials from three months ago may already be outdated.

vLLM74.3k★

vLLM is the production LLM inference engine that made PagedAttention mainstream. If you're serving models to multiple users simultaneously, vLLM's memory-efficient KV cache management means you can serve 2-4x more concurrent requests on the same GPU than naive implementations. 74k stars and adopted as the default by Hugging Face. TGI (Hugging Face's own server) has a faster Rust core with 1-5ms less overhead per request — better for low-latency single-user serving. TensorRT-LLM squeezes 20-40% more throughput on NVIDIA hardware but requires hour-long compilation steps. Ollama is simpler for local dev but not built for production scale. Use vLLM if you're self-hosting LLMs for a product with multiple concurrent users. The OpenAI-compatible API means your app code doesn't change. The catch: it's Python-based, so scheduling adds 1-5ms latency per request versus TGI's Rust core. GPU required — this isn't for CPU-only setups. And model support, while broad, doesn't always include the latest architecture on day one. SGLang is emerging as a faster alternative for some workloads.

text-generation-webui46.4k★

Text-generation-webui is the Swiss Army knife for running LLMs locally. One web interface, every backend — llama.cpp, ExLlamaV2, Transformers, AutoGPTQ. Load a model, chat with it, fine-tune it, run it as an API. It's the Gradio-powered cockpit for local AI. If you want to experiment with open-weight models without touching a command line, this is your starting point. It supports model quantization, LoRA training, multimodal input, and OpenAI-compatible API endpoints. Ollama is simpler but less flexible — great for quick inference, not for training. LM Studio has a prettier UI but is closed-source. vLLM is faster for production serving but has no UI. Best for tinkerers and indie hackers who want full control over their AI stack without cloud API bills. The catch: it's AGPL-3.0, so building a commercial product on top requires care. Setup can be finicky — GPU drivers, CUDA versions, and Python dependencies love to conflict. And performance won't match purpose-built inference servers like vLLM or TGI for production workloads.

LocalAI44.4k★

The Swiss Army knife of local AI inference. LocalAI doesn't just run LLMs — it handles image generation, audio processing, embeddings, and more through a single OpenAI-compatible API. If you're migrating off cloud AI services and need a drop-in replacement that speaks the same protocol, this is it. Ollama is the simpler, faster choice for just running LLMs — 15-20% faster inference and dead-simple CLI. LM Studio gives you a desktop GUI. vLLM is the production-grade option for GPU-heavy deployments. Where LocalAI shines is flexibility. It supports GGUF, Safetensors, GPTQ, AWQ — basically every model format. It runs on CPU without a GPU, which Ollama also does but with fewer model types. The multi-modal support means one service replaces three or four specialized tools. The catch: that flexibility comes with complexity. Setup is harder than Ollama's one-liner. Performance lags behind dedicated tools for any single task. And with Ollama hitting 52 million monthly downloads in Q1 2026, the ecosystem gravity is pulling developers the other way.

MLX24.8k★

MLX is Apple's answer to "why can't I train models efficiently on my MacBook?" It's an array framework built specifically for Apple Silicon's unified memory architecture — no copying data between CPU and GPU. If you have an M-series Mac, MLX squeezes out performance that PyTorch's MPS backend can't match for inference. For indie hackers running local LLMs on a MacBook Pro, MLX is the fastest path. It ships with mlx-lm for running Llama, Mistral, and other models locally. PyTorch is the cross-platform standard but wastes cycles on Apple's non-CUDA architecture. llama.cpp is the other local inference option with broader hardware support. The catch: MLX is Apple Silicon only — your code won't run on Linux, Windows, or NVIDIA GPUs. That means no cloud deployment, no team members on non-Mac hardware, and no GPU cluster training. PyTorch is still faster for training (MLX wins on inference). The ecosystem is tiny compared to PyTorch's. Use MLX for local experimentation and inference; use PyTorch for anything that needs to run beyond your laptop.

omlx6.8k★

oMLX is the best way to run LLMs on Apple Silicon right now. A menu-bar inference server with continuous batching, SSD caching that drops TTFT from 30-90 seconds to under 5, and up to 4.14x generation speedup at 8 concurrent requests. Compatible with Claude Code, OpenClaw, Cursor, and any OpenAI-compatible client. The two-tier SSD caching is the breakthrough: hot KV cache blocks stay in RAM, cold blocks go to SSD with LRU policy, and everything persists — so previously cached contexts are always recoverable. Supports text LLMs, vision models, OCR, embeddings, and rerankers. Compared to Ollama (easier but no SSD caching), oMLX is faster for repeated contexts. Compared to llama.cpp (more portable), oMLX is Apple Silicon-optimized. Use this when you're a Mac developer running local models as coding agent backends or for inference. Skip this on any non-Apple hardware. The catch: macOS-only by design. The menu-bar UX is convenient but hides complexity — debugging inference issues requires digging into logs. And the MLX ecosystem is still maturing compared to CUDA/llama.cpp.

flash-moe2.0k★

Flash-MoE runs a 397-billion parameter model on a MacBook with 48GB RAM at 4.4+ tokens/second. No Python, no frameworks — pure C, Objective-C, and hand-tuned Metal shaders. It exploits MoE architecture by streaming only the 4 active experts (of 512) from SSD per layer, on demand. This is a technical marvel. The custom Metal compute pipeline reads expert weights directly from NVMe via parallel pread() with GCD dispatch groups. The OS page cache handles caching naturally. It runs Qwen3.5-397B-A17B with tool calling support. Compared to llama.cpp (broader model support but slower for MoE) and Ollama (easier but can't touch this model size), Flash-MoE is the only way to run a 400B model locally. Use this when you want frontier-class model quality on your laptop and have a Mac with 48GB+ RAM. Skip this if you need broad model compatibility — it's optimized for one architecture. The catch: macOS-only (Metal required), supports only MoE models with the right architecture, and 4.4 tok/s means waiting 30+ seconds for a paragraph. Patience required.