LLM Inference

ollama181.2k★

Ollama makes running an LLM on your own machine dead simple. Download it, type ollama run llama3 in your terminal, and you are chatting with a model locally. No Python environments, no CUDA wrangling, no Docker. It is the most popular local LLM tool by a wide margin, supports dozens of models like Llama, Mistral, Gemma, DeepSeek, and Qwen, and runs on Mac, Linux, and Windows. The API is OpenAI-compatible, so anything built for the OpenAI API can point at Ollama instead and keep your data on your own machine. That local engine is MIT-licensed and free, and for most people it is the whole product. The newer wrinkle is Ollama Cloud. There is now a hosted option for running larger models than your hardware can handle, with a free tier, a Pro tier at 20 dollars a month, and a Max tier at 100 dollars a month for heavier use. The local engine stays free no matter what. Solo users and anyone privacy-minded run everything locally at no cost. Reach for the cloud tiers only when you want frontier-size models without buying the GPU to match. The catch is hardware, the same as it ever was. A Mac with 16GB of RAM runs 7B models fine; 70B and up needs serious GPU power, which is exactly the gap the paid cloud now fills. And local models still trail the best hosted models like Claude and GPT on the hardest tasks, so match the model to the job.

Open WebUI152.3k★

Open WebUI gives you a ChatGPT-style interface for your own models, whether they run locally through Ollama, through OpenAI's API, or any compatible endpoint. Chat with models, upload documents for RAG (retrieval-augmented generation, meaning the AI reads your files and answers questions about them), manage conversations, and share prompts. All of it on your own server. The polish is the surprise. It feels like a commercial product rather than a weekend project: multi-user accounts, conversation history, model management, function calling, web search integration, and image generation. It is the most feature-complete self-hosted LLM front end available, and standing it up is a Docker command plus whatever inference backend you point it at. There is no first-party cloud, by design; the pitch is that you run it, on-premise, private cloud, or air-gapped. Standard internal use is free, with the branding left exactly as shipped. Solo and small teams: run it, there is no catch at your size. The catch shows up at scale, and it is mostly about whose name is on the screen. The license is not open source. It is BSD-3 plus a branding clause: you may not remove, alter, or replace the Open WebUI name and logo once a deployment passes fifty end users in a rolling thirty-day window, unless you buy an enterprise license or get written permission. A few enterprise-exclusive capabilities sit behind that same license too, so "everything is free" is not quite right. Read the license before building a business on top of it. The other cost is hardware, since a 7B model wants 8GB of RAM or a decent GPU and Open WebUI itself is the light part.

llama.cpp128.5k★

A server without a GPU. llama.cpp makes it possible. It runs quantized versions of open models (Llama, Mistral, Phi, Qwen, and dozens more) in pure C/C++ with optional GPU acceleration. No Python, no PyTorch, no CUDA dependency hell. Everything is free under MIT. No paid tier, no cloud, no account. Download a model file (GGUF format), point llama.cpp at it, and you're running inference. It includes a built-in HTTP server that exposes an OpenAI-compatible API, so your existing code that talks to GPT can talk to a local model with one URL change. The catch: you need hardware. A 7B parameter model needs ~4GB RAM (quantized). A 70B model needs ~40GB. Quality depends entirely on the model and quantization level; a heavily quantized model on a laptop won't match GPT-4. But for privacy-sensitive workloads, offline use, or just not wanting to pay per token, nothing else comes close.

vLLM92.1k★

vLLM is the fastest way to serve open-weight LLMs on your own hardware. It takes a model like Llama or Mistral and puts an OpenAI-compatible API in front of it, squeezing maximum throughput out of your GPUs. What's free: everything. Apache 2.0 license. The entire inference engine, all optimizations (PagedAttention, continuous batching, tensor parallelism), the OpenAI-compatible API server. All free. vLLM's key innovation is PagedAttention, which manages GPU memory the way operating systems manage RAM, in pages instead of contiguous blocks. The result: 2-4x more throughput than naive inference. It's become the default serving engine for self-hosted LLMs. The catch: you need serious GPUs. Running a 70B parameter model requires 2-4 A100 GPUs ($1-2/hr on cloud, or $10K+ each to buy). Even a 7B model needs a decent GPU with 16GB+ VRAM. vLLM is free but the hardware is emphatically not. And it's optimized for NVIDIA GPUs. AMD ROCm support exists but is second-class.

CLIProxyAPI52.4k★

CLIProxyAPI wraps existing AI coding CLIs, Gemini CLI, Claude Code, ChatGPT Codex, and others, and exposes them as OpenAI/Gemini/Claude-compatible API endpoints. The pitch is that you get access to models like Gemini 2.5 Pro and GPT-5 through their free CLI tiers, served as a standard API you can plug into any app. Let me be direct: this is a proxy that routes around pricing by using free CLI tools as backends, and exploding because free model access is irresistible. The homepage points to a subscription service at z.ai. The catch: this sits in a gray area. You're wrapping free CLI tools and serving them as APIs, which likely violates the terms of service for most of those CLIs. The sustainability of this approach depends entirely on providers not shutting it down. The MIT license covers the code, but the underlying model access is not yours to redistribute. Use at your own risk.

LocalAI49.1k★

LocalAI runs your own AI models locally and exposes them through an OpenAI-compatible API. LLMs, image and 3D generation, speech in both directions: all from a single server. No cloud, no API keys, no data leaving your machine. MIT-licensed, free. Docker-based setup handles most of the complexity. A config file defines which models to load and which backends to use (llama.cpp, whisper, stable diffusion, and more). CPU inference is supported, which means any machine can run it. GPU acceleration is faster but not required. Models download at first startup. Developers who want to swap out OpenAI API calls with local models point their existing code at LocalAI's endpoint and change nothing else. Good for privacy-sensitive applications, air-gapped environments, and teams that want to control costs without changing application code. The catch: local inference is slower than cloud for most hardware setups. Model selection lags the frontier. You get privacy and cost control; you give up raw performance and convenience.

text-generation-webui47.7k★

Text-generation-webui gives you a browser-based interface to do it. Load a model, chat with it, fine-tune it, generate images. It's the Swiss Army knife for local AI. The entire project is free under AGPL-3.0. Every feature (chat, notebook mode, model loading, LoRA training, multimodal/vision support, extensions) ships at $0. The developer sells some extension packs on Gumroad, but those are optional add-ons, not core features. Self-hosting is the only option, and the setup complexity depends on your GPU situation. If you have an NVIDIA card with 8GB+ VRAM, the one-click installers work well. AMD and Apple Silicon support exists but can be finicky. Expect 30-60 minutes for first-time setup including downloading a model. Solo developers: this is your playground. Run models locally, experiment with fine-tuning, keep your data private. Small teams: share a beefy GPU server running the API mode. Beyond that, look at dedicated inference servers like vLLM. The catch: GPU hardware requirements are real. You need a decent GPU to run anything useful. A 7B parameter model needs ~6GB VRAM. Anything bigger needs proportionally more. No GPU, no party.

llmfit36.9k★

llmfit answers the question everyone running local AI asks first: will this model fit on my machine? It reads your CPU, RAM, and GPU, then scores every model in its catalog on memory fit, estimated speed, quality, and context length. No more downloading a huge model to find out it will not load. MIT licensed, free, one Rust binary. Install with Homebrew, Scoop, MacPorts, uv, Docker, or a curl script, on macOS, Linux, or Windows. It detects NVIDIA, Apple Silicon, AMD ROCm, and Intel GPUs, understands GGUF, AWQ, GPTQ, and EXL2 quantization, and works with Ollama, llama.cpp, MLX, Docker Model Runner, and LM Studio. No account, and the README says nothing leaves your machine unless you ask. There is also a web dashboard and a REST API through llmfit serve. Free at every size with no paid tier. Solo builders get the most out of it. Teams can run the API on shared GPU boxes to see what fits where. Once you have picked a model, ollama/ollama or ggml-org/llama.cpp does the actual running. The catch: speeds are estimates from a formula unless you run the benchmark against a live runtime, and the GPU bandwidth table only covers about 80 cards. Windows and Intel Macs get weaker detection. The model catalog is baked in at compile time, so new models only show up when you update.

sglang36.1k★

SGLang has become one of the default engines for serving open-weight LLMs in production at scale. It runs from a single GPU up to large clusters, with prefix caching, speculative decoding, and day-zero support for new model releases. The project says it runs on hundreds of thousands of GPUs, at xAI, NVIDIA and the major clouds among others. Apache-2.0, free. Install is a `uv pip install`, with builds for NVIDIA, AMD, Intel, Google TPUs and Ascend NPUs. Models span Llama, Qwen, DeepSeek, Kimi, GLM, Mistral, diffusion models, and most of Hugging Face. It is OpenAI-API compatible, so existing clients drop in without rewrites, and it doubles as a rollout backend for reinforcement learning training. Solo developers and small teams running open-weight models: this is one of the strongest options on the shelf, especially on DeepSeek or with heavy agentic workloads. Large teams running production inference at scale: you are very likely already evaluating it. The catch is that serious inference is still serious work. Cold starts, KV cache tuning, and multi-node setups need real engineering. SGLang gives you a faster engine, not a managed platform. RadixArk, a venture-funded company among its maintainers, is building one on top, with no public pricing yet.

ds422.6k★

DwarfStar (the repo is still named ds4) is antirez's local inference engine for running a short list of big open models on hardware you own. The creator of Redis built it first for DeepSeek V4 Flash, and it now also runs DeepSeek V4.1 Flash, V4 PRO, the GLM 5.x family, and Qwen3.8 Flash Next on Metal, behind an OpenAI- and Anthropic-compatible server. It only loads the GGUF files the project itself produces. MIT, free. The price is hardware, not software. Macs with 96 GB or more are the primary target, and smaller machines can stream weights off SSD. On Linux it runs on CUDA, including older Ada Lovelace cards like the L40S, plus DGX Spark and Strix Halo boxes. Setup is a download script and `make`. Weight files run to hundreds of gigabytes, so budget a big, fast SSD too. Beyond the chat CLI it ships `ds4-agent`, a built-in coding agent, plus image input, disk KV caching and inference split across machines. A solo developer with a loaded Mac Studio gets a near-frontier model with nobody else in the loop. Everyone else should stay on llama.cpp or vLLM until they own the metal. The catch is churn. antirez calls it beta quality, warns that regressions are possible and that models get dropped when better ones arrive, and says openly it was built with strong help from AI coding agents. Run it to own your inference stack, not to ship a product on it.

omlx21.8k★

oMLX runs large language models on your Mac and manages the whole thing from the menu bar. Pick a model, hit start, and you have a local OpenAI-compatible API at localhost:8000 that Claude Code, Codex, or any OpenAI client can point at. Apache 2.0, no account, no paid tier. The engineering underneath is more serious than the menu bar suggests. Continuous batching handles concurrent requests, and a two-tier KV cache keeps hot blocks in RAM and spills cold ones to SSD, so a long context survives a server restart instead of being recomputed from scratch. One process serves text models, vision models, OCR, embeddings, and rerankers, with model pinning, per-model idle timeouts, sampling settings, and a memory ceiling all editable from the admin dashboard. Solo developers running a local coding assistant get the most out of it, and a small team can share one big Mac over the API. Ollama (ollama/ollama) is easier to hand to a teammate and runs on any hardware. llama.cpp (ggml-org/llama.cpp) gives you more low-level control. oMLX wins specifically on Apple Silicon. The catch: Apple Silicon and macOS 15 or newer, full stop. And the fast native Metal kernels for the GLM and MiniMax families do not build from a plain pip install. Without full Xcode you silently fall back to a path the project measures at roughly 30x slower on prefill. Use the official DMG, which ships them precompiled.

TensorRT-LLM14.7k★

TensorRT-LLM squeezes maximum inference performance out of NVIDIA GPUs for large language models. It handles quantization (FP8, FP4, INT4), custom attention kernels, paged KV caching, and multi-GPU deployment through a Python API. If you are serving LLMs at scale on NVIDIA hardware, this is the optimization layer that makes the economics work. Running it yourself means you need NVIDIA GPUs, full stop. No AMD, no Apple Silicon, no CPU fallback. You will also need CUDA installed and compatible driver versions. The setup is not trivial, but NVIDIA provides containers and Docker images that smooth out the worst of it. Once running, the performance gains over naive PyTorch inference are substantial, often 2-4x throughput improvements. For teams already committed to NVIDIA hardware, TensorRT-LLM is the right call over vLLM when you need every last token per second. vLLM is easier to set up and supports more hardware. llama.cpp is better for local, single-GPU experimentation. TensorRT-LLM is for production serving where GPU cost is a real line item. The catch: you are locked to NVIDIA forever. The library only works on their GPUs, and if your cloud costs push you toward AMD or custom silicon, you are rewriting your inference stack from scratch.

kimi-k3-in-c8.1k★

kimi-k3-in-c runs a 2.78-trillion-parameter model on a machine with 8 GB of RAM, and the claim holds up. It's a from-scratch inference engine in portable C99, no BLAS, no GPU, no framework. Kimi K3 is a mixture-of-experts model where only a handful of experts fire per token, so the engine streams weights from disk and caches the few it needs. Apache 2.0, free. The headline number is real but it's not the one that matters. You need roughly 1.7 TB of fast NVMe for the checkpoint, Linux on x86-64 with AVX2, and patience: generation runs seconds to tens of seconds per token depending on hardware and memory budget. More RAM buys real speed, but nothing close to interactive. Output is byte-identical at every memory budget, validated against a PyTorch reference, which is the kind of engineering discipline most projects skip. Use this to learn how MoE inference actually works, to benchmark against, or to prove a point. Don't use it to serve anything: Moonshot's hosted API is faster and cheaper for real workloads, and llama.cpp is the right tool for models that fit your hardware. The catch: it's a weeks-old, essentially solo reference implementation, and the model weights carry Moonshot's own license, not Apache. Treat it as a brilliant teaching artifact, not infrastructure.

rikkahub7.7k★

RikkaHub is a native Android app for chatting with any LLM through one interface. Instead of separate apps for ChatGPT, Claude and Gemini, you point it at any OpenAI, Anthropic or Google-compatible endpoint, including your own, and switch providers between conversations. It handles markdown, image, PDF and document input, web search, message branching, memory and MCP. AGPL-3.0, free. Install it from Google Play or the project site, add your API keys and go. There is nothing to self-host: the app talks directly to the providers you configure. Newer builds add a Workspace, a lightweight Linux environment built on proot that runs on the phone without root so agents can gather information and assemble documents, plus a web interface you can open in a browser. For anyone on Android juggling multiple AI providers, this is the clean way to do it. You pay each provider for tokens, or point it at a local model endpoint and pay nothing. It is a personal app, and there is no team tier. The catch is trust in the chain. A multi-provider client is only as private as the endpoints you configure, and the README itself warns that many forks exist, some leaking privacy or asking for excessive permissions. Install the official build and point it at providers you trust.

mlx-lm7.0k★

mlx-lm runs and fine-tunes large language models directly on a Mac. Point it at a model on Hugging Face and one command pulls it down and runs it locally, using Apple's own MLX engine instead of a cloud API or a separate GPU rig. MIT licensed, free, and built by Apple's own ml-explore team, the same group behind MLX itself. It does more than run models. You can quantize them down to 4-bit, fine-tune with LoRA or full-model training, serve with streaming and prompt caching, and even split work across multiple machines. Setup is close to trivial: pip install mlx-lm, then a single command chats with a model. The real constraint is memory. MLX uses the Mac's unified memory, so the model has to roughly fit in RAM, and pushing past that needs macOS 15 or newer plus some system tuning. And it's Apple Silicon only. No M-series chip, no mlx-lm. The honest framing on competition: this is a building block, not a finished app. llama.cpp is the closest peer and runs on more hardware; Ollama and LM Studio are more packaged and app-like, and increasingly use MLX under the hood anyway; vLLM is for datacenter GPUs, a different world. mlx-lm's edge is being the MLX-native option, which means the best raw performance on a Mac and the cleanest fine-tuning story. Solo developers and researchers on Apple Silicon: this is the fast path. Small teams can build on it; larger production serving will want something server-side. The catch is that you're trading convenience and reach for Mac-native speed. It's lower-level than Ollama, locked to Apple hardware, and capped by how much RAM you bought. Within those lines, nothing runs models on a Mac better.

LiteRT-LM6.5k★

LiteRT-LM runs language models directly on a device, no cloud and no internet required. The model lives on the phone, laptop, smartwatch, or even in the browser, so data never leaves the hardware, it works offline, and there's no per-query bill. This is Google's own framework, and Google uses it to power on-device AI in Chrome, Chromebook Plus, and the Pixel Watch. Apache 2.0, completely free. It's cross-platform by design, targeting Android, iOS, desktop, the web via WebGPU, and small boards like Raspberry Pi, and it taps GPU and NPU acceleration instead of grinding on the CPU. It runs open models like Gemma, Llama, Phi, and Qwen. The work isn't running a server, because there is no server. The work is on the build side: you obtain and convert models into the right format, then wire up the native SDK for each platform you ship to, and manage on-device memory per device class. Heavier than calling a cloud API, far lighter than operating an inference cluster. The real competition is other on-device runtimes. llama.cpp has broader model coverage and a bigger community; Meta's ExecuTorch is the closest vendor-backed rival; Apple's MLX wins on Macs but only on Macs. LiteRT-LM's edge is tight, official integration with Android and Google silicon. It doesn't replace a paid product so much as move certain workloads off the paid-API meter: the small and mid-size models you'd otherwise rent from a cloud. Solo and small teams shipping mobile or edge apps: this is the Google-blessed path. Larger teams already on Android get first-party support. The catch is maturity. The core runtime is production-ready and shipping in real Google products, but some bindings, Swift and JavaScript among them, are still early preview, and the project is young. And on-device models are not frontier models. If you need GPT-class quality, this isn't that. It's for when private, offline, free, and good-enough beats cloud-quality.

flash-moe4.1k★

Big models want big hardware, which puts most of them out of reach on a laptop. Flash-moe makes that possible anyway. It uses a technique called Mixture of Experts (MoE) to run only the parts of the model that matter for each request, dramatically cutting the memory and compute needed. The pitch is simple: big model intelligence on small hardware. Models that normally need 32GB+ of VRAM can run on a laptop with 8-16GB of regular RAM. It's slower than running on a GPU, but it works. The catch: it is growing explosively and is still very early. The 'runs on a laptop' promise depends heavily on the model and your hardware. And MoE optimization is an active research area. Expect the approach to evolve fast.

Rapid-MLX3.8k★

Rapid-MLX runs open models locally on Apple Silicon, and speed is the whole pitch. One command installs it (Homebrew, pip, or a curl script), one command serves an OpenAI-compatible API, and it also speaks the Anthropic messages format, so Claude Code, Codex CLI, and Aider can point at it as a drop-in backend. Apache 2.0, completely free. The engineering focus is on the things that make local agent workflows fail in practice. A prompt cache brings claimed time-to-first-token under a tenth of a second on repeat prompts, and tool calling is handled across 17 parser formats, which matters because local models break agent loops on malformed tool calls far more often than on reasoning. It's in Homebrew core and publishes to PyPI, with releases landing near daily. Mac-only by design. If you run agents against local models on an M-series machine, try it head-to-head with Ollama or LM Studio; the performance claims ship with benchmarks in the repo rather than vibes. On Linux or Windows, look at Ollama or vLLM instead. The catch: it's young and moves extremely fast, the headline numbers are the project's own, and independent benchmarks are still thin. And no inference engine fixes the ceiling: a fast small model is still a small model.

waste2.4k★

waste does something that shouldn't be possible on a laptop: run the full 2.8-trillion-parameter Kimi K3 model on modest RAM by streaming the activated expert weights straight off an NVMe drive. It's a dependency-free, embeddable C inference engine with documented multimodal (vision) support, from the team behind sqlite.ai. The headline isn't hype, Kimi K3 is a real Moonshot model released in mid-2026. The trick is that a mixture-of-experts model only lights up a fraction of its weights per token, so waste keeps the cold weights on disk and streams what it needs. That's how a 3-trillion-class model fits on consumer hardware. Free under Apache 2.0, for anyone at any scale who wants to poke at frontier-size models without a datacenter. The catch is speed, and it's a big one. You're looking at roughly half a token per second on a 64GB MacBook, and you need close to a terabyte of NVMe to hold the model. It's explicitly pre-production, the format and API aren't frozen. This is a fascinating proof that disk-streaming inference works, not something you'd build a product on. Run it to learn, not to serve.

tokenspeed2.1k★

TokenSpeed is an LLM inference engine aimed at agentic workloads. The claim is TensorRT-LLM-level performance with vLLM-level usability, which is bold positioning if it holds up. The architecture uses a local-SPMD modeling layer with static compilation and a C++ control plane with type-safe KV cache management. The team has shipped benchmarks against TensorRT-LLM on Kimi K2.5 that look favorable. Hardware target is NVIDIA Blackwell (B200) right now, with Hopper and AMD MI350 optimization in progress. Setup involves the usual NVIDIA stack: CUDA, drivers, the lightseek.org/tokenspeed getting-started guide, and Blackwell-class hardware you almost certainly don't own personally. Currently it runs Kimi K2.5; Qwen, DeepSeek, and MiniMax support is in progress. If you're standing up an inference service for agent workloads on Blackwell GPUs, this is worth evaluating against vLLM and TensorRT-LLM. Solo and small teams: stick with vLLM until TokenSpeed matures. Large teams running serious agent workloads on B200s: benchmark it, the agentic optimizations look real. The catch: explicitly preview/beta. The README says "do not use this preview release for production deployments." Model coverage is thin and the runtime is still gaining features like KV store and VLM support. Watch it, don't bet your inference layer on it yet.

laya-mlx2.1k★

laya-mlx runs Laya decision models natively on Apple Silicon. Laya is an open-weights model from Convai Innovations that answers multiple choice, rating, and yes or no questions with probabilities instead of generated text. It is the open counterpart to TypeSafe AI's closed Jev API. This port drops PyTorch and runs on Apple's MLX, returning a short decision in roughly 7 to 14 milliseconds on an M3 Max by the author's own measurement. Apache 2.0. Setup is pip install laya-mlx on Python 3.11 or later and macOS 14 or later. Weights download from Hugging Face on first load, and peak memory for a short question stays under 1 GB. No cloud API and no account. It covers inference and conversion only. Training and fine-tuning stay in the upstream Laya project. Free at every size. Use it for local classification, routing, or guardrail checks on a Mac where a full LLM call is too slow. TheoLeeCJ/SemIf gets a similar interface out of general open models and also has an MLX backend. The catch: this is an unofficial port from a young pseudonymous account, at version 0.1.0 with five commits. Context tops out at 512 to 1,024 tokens. And an open issue reports that swapping the order of the options flipped the answer 27 to 33 percent of the time on a Chinese benchmark. Test option-order sensitivity before trusting it.

nativ1.4k★

nativ runs AI models locally on your Mac, built for Apple Silicon specifically using MLX, Apple's own ML framework. Chat with local models, do vision and image tasks, manage what you've downloaded, and expose an OpenAI or Anthropic-compatible endpoint so your coding tools can hit a model running on your own machine. MIT, free, no account. Setup is about as easy as local AI gets: download the DMG, drag it to Applications, launch. It auto-updates. The only real gate is hardware. You need an Apple Silicon Mac and macOS 26, which is a very new OS to require, plus enough unified memory for whatever model you load. For privacy-sensitive work, or just dodging API bills, running models locally is the move, and on a modern Mac nativ makes it painless, solo or small team, at no cost. It competes with LM Studio and Ollama front-ends; the difference is it's MLX-native and Mac-only by design. The catch: that Mac-only, macOS 26 requirement locks out everyone else, and a couple of the headline features (audio models, image generation as a chat tool) are still "coming soon" rather than shipped. Solid today, with asterisks.

MoonEP1.1k★

MoonEP is a low-level building block for running Mixture-of-Experts AI models across many GPUs. Its whole job is keeping the token load balanced across GPU ranks, using dynamic redundant experts and fused kernels, so no single GPU becomes the bottleneck during MoE inference or training. It comes from Moonshot AI, MIT-licensed and free. This is infrastructure, not an app. There's nothing to click. You need a datacenter-grade setup: a multi-GPU NVIDIA cluster with NVLink, since it relies on symmetric memory and remote reads between GPUs. It was tested on eight H20s. If that sentence didn't apply to you, this isn't your tool. For the specific teams training or serving large MoE models on GPU clusters, MoonEP is a serious performance primitive that competes with libraries like DeepEP. Everyone else can skip it. There's no team-size question here, it's for infrastructure engineers at labs and companies running big models. The catch is reach and maturity. It's NVIDIA-only today, with other hardware listed as coming soon, and it's very new, benchmarked against DeepEP but not yet widely battle-tested. It's a sharp tool for a narrow, expensive job, not something you casually adopt.

deltafin817★

Deltafin runs Moonshot's Kimi K3, a massive trillion-parameter model, locally on a single machine and puts an OpenAI-compatible API in front of it. That means your local chat or coding agent can talk to a frontier-scale model running entirely on your own hardware, no API keys, no data leaving the box. It's MIT-licensed and free. The honesty starts with the requirements. You need at least 64GB of RAM and about 1.7TB of storage for the full model, or roughly 215GB in streaming mode. It'll use Metal or CUDA if you have it and fall back to CPU if you don't. This is a heavy lift, firmly in enthusiast-with-a-workstation territory. For a hobbyist who wants to run a giant model fully private and doesn't mind the setup, it's a fascinating way to do it for free instead of paying for hosted API calls. It's a proof of concept, not a team server, so treat it as an experiment rather than infrastructure. The catch is speed and maturity, and the README is refreshingly blunt about it: set your client timeouts to hours, not seconds. It does greedy decoding only, handles one request at a time, and re-processes the full context every turn. It's an impressive hack, not a practical replacement for a hosted API when you need answers now.