Best LLM Inference Tools

Ollama makes running an LLM on your own machine dead simple. Download it, type ollama run llama3 in your terminal, and you are chatting with a model locally. No Python environments, no CUDA wrangling, no Docker. It is the most popular local LLM tool by a wide margin, supports dozens of models like Llama, Mistral, Gemma, DeepSeek, and Qwen, and runs on Mac, Linux, and Windows. The API is OpenAI-compatible, so anything built for the OpenAI API can point at Ollama instead and keep your data on your own machine. That local engine is MIT-licensed and free, and for most people it is the whole product. The newer wrinkle is Ollama Cloud. There is now a hosted option for running larger models than your hardware can handle, with a free tier, a Pro tier at 20 dollars a month, and a Max tier at 100 dollars a month for heavier use. The local engine stays free no matter what. Solo users and anyone privacy-minded run everything locally at no cost. Reach for the cloud tiers only when you want frontier-size models without buying the GPU to match. The catch is hardware, the same as it ever was. A Mac with 16GB of RAM runs 7B models fine; 70B and up needs serious GPU power, which is exactly the gap the paid cloud now fills. And local models still trail the best hosted models like Claude and GPT on the hardest tasks, so match the model to the job.

Local engine is free and open source. Ollama Cloud adds hosted tiers at $20/mo (Pro) and $100/mo (Max).

RAGFlow is a retrieval-augmented generation engine built around deep document understanding, and it has grown into an agent platform sitting on top of that. It still handles the messy parts of RAG that other frameworks hand-wave past: PDFs with tables, slides, spreadsheets, scanned images, mixed-language docs, now with MinerU and Docling parsing. On top of that it added agent memory, an MCP server, and delivery into chat channels like Discord, Telegram, and Feishu. Apache 2.0. Self-hosting is a real project, not a `docker run`. You need 4-plus CPU cores, 16-plus GB RAM, 50-plus GB disk, and Docker. The compose file spins up the RAGFlow service plus a search backend (Elasticsearch, or their in-house Infinity vector DB) and MinIO for blob storage. The template-based chunking is still the killer feature: you can see and tweak how documents are segmented before they ever hit the index. Solo developers and small teams that want full control of their RAG pipeline can self-host this and feel good about it. Larger teams, or anyone without infra time, should look at RAGFlow Cloud, which gives you the same engine without the ops. The catch is operational weight. If you just need to chat with a few documents, LlamaIndex or LangChain over a simple vector store ships faster. Pick RAGFlow when document parsing quality, citation grounding, and now agent workflows over your own documents are the thing you are actually buying.

Apache 2.0. Self-host free if you have the compute; RAGFlow Cloud is the managed path.

A server without a GPU. llama.cpp makes it possible. It runs quantized versions of open models (Llama, Mistral, Phi, Qwen, and dozens more) in pure C/C++ with optional GPU acceleration. No Python, no PyTorch, no CUDA dependency hell. Everything is free under MIT. No paid tier, no cloud, no account. Download a model file (GGUF format), point llama.cpp at it, and you're running inference. It includes a built-in HTTP server that exposes an OpenAI-compatible API, so your existing code that talks to GPT can talk to a local model with one URL change. The catch: you need hardware. A 7B parameter model needs ~4GB RAM (quantized). A 70B model needs ~40GB. Quality depends entirely on the model and quantization level; a heavily quantized model on a laptop won't match GPT-4. But for privacy-sensitive workloads, offline use, or just not wanting to pay per token, nothing else comes close.

Software is free. Cost is hardware: $0 on a laptop for small models, $700+ for serious inference.

vLLM is the fastest way to serve open-weight LLMs on your own hardware. It takes a model like Llama or Mistral and puts an OpenAI-compatible API in front of it, squeezing maximum throughput out of your GPUs. What's free: everything. Apache 2.0 license. The entire inference engine, all optimizations (PagedAttention, continuous batching, tensor parallelism), the OpenAI-compatible API server. All free. vLLM's key innovation is PagedAttention, which manages GPU memory the way operating systems manage RAM, in pages instead of contiguous blocks. The result: 2-4x more throughput than naive inference. It's become the default serving engine for self-hosted LLMs. The catch: you need serious GPUs. Running a 70B parameter model requires 2-4 A100 GPUs ($1-2/hr on cloud, or $10K+ each to buy). Even a 7B model needs a decent GPU with 16GB+ VRAM. vLLM is free but the hardware is emphatically not. And it's optimized for NVIDIA GPUs. AMD ROCm support exists but is second-class.

Software is free. Hardware costs $0.50-32/hr in the cloud. Self-hosting beats API pricing only at massive scale or when data privacy is non-negotiable.

SGLang has become the default engine for serving open-weight LLMs in production at scale. It runs from a single GPU up to large clusters, with prefix caching, speculative decoding, and day-zero support for new model releases. The throughput gains over the previous generation of servers are large, and the adoption backs it up: it reportedly runs across hundreds of thousands of GPUs at xAI, NVIDIA, and the major clouds. Apache-2.0, free. Setup is the standard NVIDIA inference stack, but hardware support is broad now: NVIDIA, AMD, Intel, Google TPUs, and Ascend NPUs. Models span Llama, Qwen, DeepSeek, GLM, Mistral, diffusion models, and most of Hugging Face. It is OpenAI-API compatible, so existing clients drop in without rewrites. Solo developers and small teams running open-weight models: this is one of the strongest options on the shelf, especially on DeepSeek or with heavy agentic workloads. Large teams running production inference at scale: you are very likely already evaluating it. The catch is that serious inference is still serious work. Cold starts, KV cache tuning, and multi-node setups need real engineering. SGLang gives you a faster engine; it does not hand you a managed platform, and the operational burden of running one is still yours.

Free Apache 2.0 inference engine, now the production standard. Hardware and ops are the cost.

Open WebUI gives you a ChatGPT-like interface for your own AI models, whether they're running locally with Ollama, through OpenAI's API, or any compatible endpoint. Chat with models, upload documents for RAG (retrieval-augmented generation, meaning the AI can read your files and answer questions about them), manage conversations, and share prompts. All running on your own server. community-maintained. The UI is polished. It feels like a commercial product. Multi-user support, conversation history, model management, function calling, web search integration, and image generation. It's the most feature-rich self-hosted LLM frontend available. Everything is free for self-hosting. No premium features, no gated functionality. They recently launched a cloud-hosted version, but the self-hosted version is the full product. The catch: the license is technically "Other." It uses a custom license that's permissive for personal and organizational use but restricts commercial redistribution. Read it before building a product on top of it. Also, running LLMs locally requires serious hardware. A 7B model needs 8GB+ RAM (or a decent GPU). Open WebUI itself is lightweight, but the models it talks to are not. And updates ship fast, which means occasional breaking changes.

Free. Self-host and point it at your own models. Hardware costs are the real expense.

LiteLLM is a proxy and Python library that puts a unified OpenAI-compatible API in front of 100+ LLM providers: OpenAI, Anthropic, Gemini, Cohere, Azure, Bedrock, Ollama, and more. Write your code once using the OpenAI format and switch providers by changing one line. Run it as a proxy server and you get rate limiting, cost tracking, fallback routing, and load balancing across providers. Teams use it to control which models engineers can call, track spend per team, and add retry logic without touching application code. MIT-licensed, free to self-host. Engineering teams building on multiple LLMs or managing costs across a company get the most value from the proxy. Individual developers using it as a Python library just want to avoid rewriting LLM calls when switching providers. Both use cases are free. The catch: the proxy adds latency. Not much, usually under 10ms, but it is a network hop. And the feature set moves fast enough that staying current requires attention.

SDK is free. Proxy is free to self-host. Enterprise pricing is custom. Your real costs are the LLM API bills themselves.

LocalAI runs your own AI models locally and exposes them through an OpenAI-compatible API. LLMs, image generation, speech-to-text: all from a single server. No cloud, no API keys, no data leaving your machine. MIT-licensed, free. Docker-based setup handles most of the complexity. A config file defines which models to load and which backends to use (llama.cpp, whisper, stable diffusion, and more). CPU inference is supported, which means any machine can run it. GPU acceleration is faster but not required. Models download at first startup. Developers who want to swap out OpenAI API calls with local models point their existing code at LocalAI's endpoint and change nothing else. Good for privacy-sensitive applications, air-gapped environments, and teams that want to control costs without changing application code. The catch: local inference is slower than cloud for most hardware setups. Model selection lags the frontier. You get privacy and cost control; you give up raw performance and convenience.

Free. Your hardware is the only cost.

CLIProxyAPI wraps existing AI coding CLIs, Gemini CLI, Claude Code, ChatGPT Codex, and others, and exposes them as OpenAI/Gemini/Claude-compatible API endpoints. The pitch is that you get access to models like Gemini 2.5 Pro and GPT-5 through their free CLI tiers, served as a standard API you can plug into any app. Let me be direct: this is a proxy that routes around pricing by using free CLI tools as backends, and exploding because free model access is irresistible. The homepage points to a subscription service at z.ai. The catch: this sits in a gray area. You're wrapping free CLI tools and serving them as APIs, which likely violates the terms of service for most of those CLIs. The sustainability of this approach depends entirely on providers not shutting it down. The MIT license covers the code, but the underlying model access is not yours to redistribute. Use at your own risk.

Free to self-host but built on borrowed time. The upstream free tiers can disappear.

Omlx puts an LLM inference server in your macOS menu bar. Click the icon, pick a model, and you have a local AI API running. It uses continuous batching (handles multiple requests efficiently) and SSD caching (models load faster after the first time) optimized specifically for Apple Silicon. This is the easiest way to run local LLMs on a Mac right now. No Docker, no Python environments, no config files. Menu bar app, one click, done. The API is OpenAI-compatible so any tool that talks to OpenAI can point at your local omlx instead. Apache 2.0 licensed, Python. The catch: Mac only. Apple Silicon specifically; Intel Macs are either unsupported or severely limited. The performance depends on your Mac's unified memory; 8GB will run small models, you need 32GB+ for anything serious. And 'menu bar simplicity' means less control over advanced settings like quantization, context length, and memory allocation.

Free. Runs entirely on your Mac hardware.

Langfuse is the observability platform for LLM applications: traces every call, measures latency and cost, tracks quality over time. It traces every LLM call, shows you the prompts, completions, latency, costs, and lets you evaluate output quality over time. Basically Datadog for LLM applications. You instrument your code (a few lines with their SDK), and suddenly you can see every prompt, every token count, every dollar spent, every hallucination, across your entire pipeline. This is one of the fastest-growing tools in the LLM infrastructure space. Self-hosting is free with no feature limits. Their cloud tier has a generous free plan (50K observations/mo) and paid plans starting at $59/mo for 1M observations. The catch: Langfuse is LLM-specific. If you need general application monitoring, use Datadog or SigNoz. And the space is moving fast. New competitors appear monthly. The evaluation features (scoring, human feedback) are good but still maturing compared to dedicated evaluation platforms.

Self-host for free, cloud free tier is generous at 50K observations. Pay $59/mo when you scale past that.

DwarfStar (the repo is still named ds4) is antirez's local inference engine for running one open model family at a time on serious hardware. Right now that is DeepSeek V4, both the smaller Flash weights and the larger PRO. It is not a generic GGUF runner; the creator of Redis built it to run a frontier-class model on your own machine behind an OpenAI- and Anthropic-compatible server. MIT, free. The price here is the hardware, not the software. The floor is a 96GB Mac on Metal, and the PRO weights want much more; on Linux it runs on CUDA, DGX Spark, and Strix Halo (ROCm) boxes like the Framework Desktop. It now streams models larger than RAM off SSD, caches KV state to disk, and can split inference across multiple machines. Setup is a download script and a make, then a CLI with think and no-think modes plus that drop-in API server. For a solo developer with a loaded Mac Studio, this is a genuinely cool way to run a frontier model with full sovereignty over it. For everyone else, it asks for hardware most people do not have, so llama.cpp or vLLM stays the practical path until you own the metal. The catch is that this is alpha, by antirez's own admission, leaning heavily on AI assistance and openly indebted to llama.cpp. One model family, no production claims, breaking changes expected. Run it to learn and to own your inference stack, not to ship a product on top of.

Free software with a steep hardware floor (96GB Mac and up). Built for sovereignty, not portability.

headroom strips the bloat out of everything an LLM agent reads before it hits the prompt, tool outputs, logs, RAG chunks, raw file dumps, conversation history. The reported cuts are large, 60 to 95 percent fewer tokens with the same answers, and the compression is reversible, so the original is cached and fetchable when the model actually needs it. Apache-2.0, local-first, and free: your content never leaves your machine. You can adopt it at whatever depth you want. Drop the Python or TypeScript SDK inline, run it as a zero-code local proxy, or wrap an agent directly, Claude Code, Cursor, Copilot, and Aider all supported, plus an MCP server. It also trims the model's output, not just its input, through verbosity steering and effort routing, and headroom learn mines your failed sessions for auto-corrections. Development is fast-moving, this is not a stale side project. For anyone running agents on metered tokens, the math is simple: less context in and out means a smaller bill, and the local proxy makes trying it nearly free of effort. Solo developers and small teams: the open Apache-2.0 build is the whole tool, use it. Larger teams wanting shared deployment get a managed enterprise offering on top, which is where the money is, cross-team memory and central management rather than a better compressor. The catch is trust in the compression. Cutting 90 percent of the tokens only works if the 10 percent kept is the right 10 percent, and reversible caching exists precisely because sometimes it is not. Watch the first few runs on real work before you wire it into production blind.

The open Apache-2.0 build is the complete tool and free; pay only for managed team deployment and shared memory.

OpenMAIC creates multi-agent conversations where AI agents collaborate, debate, and teach. It's an open multi-agent interactive classroom where AI agents play different roles (teacher, student, devil's advocate) and you learn through their interactions, not just a chatbot Q&A. Built by Tsinghua University researchers, it creates immersive learning sessions with one click. The agents debate concepts, ask each other questions, and you can jump in anytime. It's backed by a published academic paper, so the pedagogy is research-grounded. AGPL-3.0 licensed. If you modify it and offer it as a service, you must open source your changes. The catch: this is an academic research project, not a polished product. The 'one click' setup still requires you to bring your own LLM API keys and the experience quality depends heavily on which model you use. AGPL-3.0 makes commercial deployment complicated. And the multi-agent approach burns through tokens fast. A 30-minute session with 3 agents is 3x the API cost of a single chatbot.

Free for personal/internal use. API costs scale with agent count. AGPL restricts hosted commercial use.

Unlimited-OCR is Baidu's open model for turning images and PDFs into text. Point it at a scanned contract, a multi-page report, or a screenshot full of text and it reads the whole thing in one pass, even very long documents. It handles single images, batches, and full PDFs, with streaming output so you see results as they parse. The weights are on Hugging Face and ModelScope under the MIT license, which means free to use, including commercially, with no strings. Running it yourself is where the cost shows up. This needs an NVIDIA GPU, CUDA 12.9, Python 3.12, and a recent PyTorch and transformers stack. You can run it through plain transformers or through SGLang for faster batch serving. It builds on DeepSeek-OCR, so the lineage is solid, but you are still standing up GPU infrastructure and a model server. There is no hosted API here, no dashboard, no support line. You bring the hardware. For solo work, the Hugging Face demo is free to try and you can run it on a rented GPU when you have a real job. Small teams that already process documents will want a dedicated GPU box or a cloud GPU. Larger teams treating OCR as a pipeline should put SGLang behind a queue. If you need OCR as a managed service with an SLA, look at a cloud vendor instead. The catch: MIT covers the code and weights, but accuracy on messy real-world scans is the thing you have to test on your own documents before trusting it. No benchmark replaces running it on your actual files.

Free and MIT licensed, including for commercial use. Your only cost is the GPU you run it on.

mlx-lm runs and fine-tunes large language models directly on a Mac. Point it at a model on Hugging Face and one command pulls it down and runs it locally, using Apple's own MLX engine instead of a cloud API or a separate GPU rig. MIT licensed, free, and built by Apple's own ml-explore team, the same group behind MLX itself. It does more than run models. You can quantize them down to 4-bit, fine-tune with LoRA or full-model training, serve with streaming and prompt caching, and even split work across multiple machines. Setup is close to trivial: pip install mlx-lm, then a single command chats with a model. The real constraint is memory. MLX uses the Mac's unified memory, so the model has to roughly fit in RAM, and pushing past that needs macOS 15 or newer plus some system tuning. And it's Apple Silicon only. No M-series chip, no mlx-lm. The honest framing on competition: this is a building block, not a finished app. llama.cpp is the closest peer and runs on more hardware; Ollama and LM Studio are more packaged and app-like, and increasingly use MLX under the hood anyway; vLLM is for datacenter GPUs, a different world. mlx-lm's edge is being the MLX-native option, which means the best raw performance on a Mac and the cleanest fine-tuning story. Solo developers and researchers on Apple Silicon: this is the fast path. Small teams can build on it; larger production serving will want something server-side. The catch is that you're trading convenience and reach for Mac-native speed. It's lower-level than Ollama, locked to Apple hardware, and capped by how much RAM you bought. Within those lines, nothing runs models on a Mac better.

Free and open source. Your only cost is owning an Apple Silicon Mac with enough RAM.

Civitai is the community hub for sharing Stable Diffusion models, LoRAs, and checkpoints. Think of it as GitHub for AI image generation models: browse what others have trained, download fine-tuned versions, and share your own. Self-hosting requires Docker, Node.js 20+, PostgreSQL, and a fair bit of setup. The docker-compose gets you running but managing model storage at scale is the real operational challenge. Most people just use civitai.com directly. Useful for solo creators experimenting with image generation or teams building products on top of community models. The platform itself is free and open source. The catch: the self-hosted version is meant for development, not production. The hosted platform at civitai.com is where the community and model library actually live.

Completely free and open source. Civitai.com offers optional paid memberships for faster downloads.

LiteRT-LM runs language models directly on a device, no cloud and no internet required. The model lives on the phone, laptop, smartwatch, or even in the browser, so data never leaves the hardware, it works offline, and there's no per-query bill. This is Google's own framework, and Google uses it to power on-device AI in Chrome, Chromebook Plus, and the Pixel Watch. Apache 2.0, completely free. It's cross-platform by design, targeting Android, iOS, desktop, the web via WebGPU, and small boards like Raspberry Pi, and it taps GPU and NPU acceleration instead of grinding on the CPU. It runs open models like Gemma, Llama, Phi, and Qwen. The work isn't running a server, because there is no server. The work is on the build side: you obtain and convert models into the right format, then wire up the native SDK for each platform you ship to, and manage on-device memory per device class. Heavier than calling a cloud API, far lighter than operating an inference cluster. The real competition is other on-device runtimes. llama.cpp has broader model coverage and a bigger community; Meta's ExecuTorch is the closest vendor-backed rival; Apple's MLX wins on Macs but only on Macs. LiteRT-LM's edge is tight, official integration with Android and Google silicon. It doesn't replace a paid product so much as move certain workloads off the paid-API meter: the small and mid-size models you'd otherwise rent from a cloud. Solo and small teams shipping mobile or edge apps: this is the Google-blessed path. Larger teams already on Android get first-party support. The catch is maturity. The core runtime is production-ready and shipping in real Google products, but some bindings, Swift and JavaScript among them, are still early preview, and the project is young. And on-device models are not frontier models. If you need GPT-class quality, this isn't that. It's for when private, offline, free, and good-enough beats cloud-quality.

Free and open source. The only cost is your device's compute and the engineering to integrate it.

TensorRT-LLM squeezes maximum inference performance out of NVIDIA GPUs for large language models. It handles quantization (FP8, FP4, INT4), custom attention kernels, paged KV caching, and multi-GPU deployment through a Python API. If you are serving LLMs at scale on NVIDIA hardware, this is the optimization layer that makes the economics work. Running it yourself means you need NVIDIA GPUs, full stop. No AMD, no Apple Silicon, no CPU fallback. You will also need CUDA installed and compatible driver versions. The setup is not trivial, but NVIDIA provides containers and Docker images that smooth out the worst of it. Once running, the performance gains over naive PyTorch inference are substantial, often 2-4x throughput improvements. For teams already committed to NVIDIA hardware, TensorRT-LLM is the right call over vLLM when you need every last token per second. vLLM is easier to set up and supports more hardware. llama.cpp is better for local, single-GPU experimentation. TensorRT-LLM is for production serving where GPU cost is a real line item. The catch: you are locked to NVIDIA forever. The library only works on their GPUs, and if your cloud costs push you toward AMD or custom silicon, you are rewriting your inference stack from scratch.

Software is free. The real cost is NVIDIA GPU hardware or cloud GPU rental.

This is a maintained list of permanently free LLM API endpoints with API keys. Not trials, not 'free for 30 days,' but free-tier APIs you can actually build on. The list is organized by provider with rate limits, model availability, and key details. CC0 licensed, so you can do whatever you want with the information. The catch: 'permanently free' is a strong claim. Free tiers change. Rate limits tighten. Providers shut down. This is a living document that's only as good as its last update. And free LLM APIs often have significant rate limits. Fine for prototyping, not for production traffic. Always have a paid fallback for anything customer-facing.

Free. It's a reference list. The APIs listed have their own terms.

This is the 'awesome list' for AI. Models, tools, infrastructure, datasets, organized by category with brief descriptions and links to the actual projects. Awesome lists live or die by curation quality. This one focuses on 'truly open source,' not source-available, not 'open weights with a restrictive license.' That distinction matters when you're building on top of these tools. The list is maintained on GitHub and follows the awesome-re standards. The catch: awesome lists are snapshots. They go stale unless someone actively maintains them, and the growth spike suggests this was recently featured somewhere. The real question is whether it'll be maintained in 6 months. Also, 'curated' means one person's opinion of what's worth including; your needs might differ. Use it as a starting point, not a definitive source.

Free. It's a curated list on GitHub.

EdgeQuake is a GraphRAG engine: it turns a pile of documents into a knowledge graph and combines vector search with graph traversal so an AI can answer multi-hop questions, the kind where the answer is spread across several documents. It's a Rust reimplementation of the LightRAG approach with a React UI, Apache-licensed and free. Running it is a real project. You need Postgres 16 with the pgvector and Apache AGE extensions, plus Docker Compose to wire it together. It also needs an LLM and embedding provider: point it at OpenAI or Anthropic and you'll pay per call, or run it against a local Ollama model for no API cost at all. For a developer building retrieval over their own documents who specifically wants the graph-plus-vector approach, EdgeQuake is a solid, self-hostable option, mostly solo and small teams comfortable with Postgres. The Rust core is fast and the code is documented. The catch is honesty about the payoff. On accuracy it's roughly a tie with LightRAG, so you're not buying a quality leap, you're buying a Rust implementation and a UI. And the pgvector-plus-AGE Postgres setup is a genuine dependency lift. If you just need basic RAG, a simple vector store ships faster. Reach for this when the graph structure is the point.

Free and open (Apache 2.0). You supply an LLM and embedding provider: pay per call with OpenAI/Anthropic, or run it free against local Ollama.

MoonEP is a low-level building block for running Mixture-of-Experts AI models across many GPUs. Its whole job is keeping the token load balanced across GPU ranks, using dynamic redundant experts and fused kernels, so no single GPU becomes the bottleneck during MoE inference or training. It comes from Moonshot AI, MIT-licensed and free. This is infrastructure, not an app. There's nothing to click. You need a datacenter-grade setup: a multi-GPU NVIDIA cluster with NVLink, since it relies on symmetric memory and remote reads between GPUs. It was tested on eight H20s. If that sentence didn't apply to you, this isn't your tool. For the specific teams training or serving large MoE models on GPU clusters, MoonEP is a serious performance primitive that competes with libraries like DeepEP. Everyone else can skip it. There's no team-size question here, it's for infrastructure engineers at labs and companies running big models. The catch is reach and maturity. It's NVIDIA-only today, with other hardware listed as coming soon, and it's very new, benchmarked against DeepEP but not yet widely battle-tested. It's a sharp tool for a narrow, expensive job, not something you casually adopt.

Completely free and open source (MIT). No paid component. The cost is the hardware: a multi-GPU NVLink cluster.

MOSS-TTS-Nano is a multilingual text-to-speech model with only 100 million parameters that runs in real time on a CPU. No GPU required. It now handles 20 languages, including English, Chinese, German, Spanish, French, Japanese, and Korean, with solid quality for its size. Small enough to embed in desktop apps, local demos, or lightweight web services. Apache 2.0. Runs via Python with standard ML dependencies, and there is now an ONNX CPU build that nearly doubles throughput. Models are on HuggingFace and ModelScope, finetuning code is published for custom voices, and recent additions include mlx-audio support for Apple Silicon and a browser reader extension. The deployment stack stays simple: no CUDA, no heavy inference server, a basic machine handles it. Free for everyone under a permissive license. Solo developers building voice features get real-time TTS without paying per-character API fees. Teams shipping products can embed it directly without usage limits. The catch: 100M parameters means tradeoffs in naturalness and expressiveness. If you need the best possible voice quality, larger models or paid APIs like ElevenLabs will sound better. This is the right call when you want speed, low cost, and local execution over peak fidelity.

Completely free. Trade peak voice quality for zero cost and local execution.

waste does something that shouldn't be possible on a laptop: run the full 2.8-trillion-parameter Kimi K3 model on modest RAM by streaming the activated expert weights straight off an NVMe drive. It's a dependency-free, embeddable C inference engine with documented multimodal (vision) support, from the team behind sqlite.ai. The headline isn't hype, Kimi K3 is a real Moonshot model released in mid-2026. The trick is that a mixture-of-experts model only lights up a fraction of its weights per token, so waste keeps the cold weights on disk and streams what it needs. That's how a 3-trillion-class model fits on consumer hardware. Free under Apache 2.0, for anyone at any scale who wants to poke at frontier-size models without a datacenter. The catch is speed, and it's a big one. You're looking at roughly half a token per second on a 64GB MacBook, and you need close to a terabyte of NVMe to hold the model. It's explicitly pre-production, the format and API aren't frozen. This is a fascinating proof that disk-streaming inference works, not something you'd build a product on. Run it to learn, not to serve.

Free and open source. A research curiosity, not a workhorse.

llm-agents.nix packages every AI coding agent worth using as a Nix flake. Claude Code, Codex, Cursor Agent, Crush, Aider, GitHub Copilot CLI, Droid, Amp, Gemini CLI, OpenCode, and a dozen others all run via `nix run github:numtide/llm-agents.nix#claude-code`. MIT licensed, automatically updated daily. Nix users already know why this matters. Most AI coding agents ship as npm packages, curl-bash installers, or proprietary binaries that conflict with each other and bypass your shell environment. Wrapping them as Nix derivations means reproducible installs, no global pollution, and easy version pinning. The repo updates from upstream sources every day, so versions stay current without manual intervention. For Nix users running multiple AI agents, this replaces a directory of brittle install scripts with one flake. Solo: install it. Teams on NixOS: add it to your shared flake. Anyone else: not for you, this is Nix-specific tooling. The catch: most of these agents are themselves proprietary (Claude Code, Cursor, Copilot, Droid, Codex), so the Nix package is just a packaging convenience. You still need API keys, accounts, and licenses for the underlying services.

The Nix packaging is free and MIT. The bundled AI agents have their own pricing (most require paid subscriptions or API keys).

RikkaHub is a native Android app for chatting with any LLM you want, through one interface. Instead of installing separate apps for ChatGPT, Claude, and Gemini, you point it at any OpenAI, Anthropic, or Google-compatible endpoint (including your own) and switch providers mid-conversation. It handles markdown, image and PDF input, web access, message branching, and MCP, and it is open source and free. Running it is as easy as any Play Store install: download it, drop in your API keys, and go. There is nothing to self-host because the app is the whole thing; it talks directly to whichever provider's API you configure. Built in Kotlin with Jetpack Compose, it feels like a proper native app, not a web wrapper. For anyone on Android who juggles multiple AI providers, this is the clean way to do it, and it costs nothing to run. You still pay each provider for the tokens you use, or point it at a local model endpoint and pay nothing at all. There is no team story here; it is a personal app on your phone, which is exactly what it should be. The catch is that you bring your own keys and your own judgment. RikkaHub does not shield you from a provider's pricing or rate limits, and a multi-provider client is only as private as the endpoints you point it at. Configure a sketchy proxy and your conversations flow through it. Point it at the real vendors and you are fine.

Free app. You pay the LLM providers you connect it to, or nothing if you use a local endpoint.

TokenSpeed is an LLM inference engine aimed at agentic workloads. The claim is TensorRT-LLM-level performance with vLLM-level usability, which is bold positioning if it holds up. The architecture uses a local-SPMD modeling layer with static compilation and a C++ control plane with type-safe KV cache management. The team has shipped benchmarks against TensorRT-LLM on Kimi K2.5 that look favorable. Hardware target is NVIDIA Blackwell (B200) right now, with Hopper and AMD MI350 optimization in progress. Setup involves the usual NVIDIA stack: CUDA, drivers, the lightseek.org/tokenspeed getting-started guide, and Blackwell-class hardware you almost certainly don't own personally. Currently it runs Kimi K2.5; Qwen, DeepSeek, and MiniMax support is in progress. If you're standing up an inference service for agent workloads on Blackwell GPUs, this is worth evaluating against vLLM and TensorRT-LLM. Solo and small teams: stick with vLLM until TokenSpeed matures. Large teams running serious agent workloads on B200s: benchmark it, the agentic optimizations look real. The catch: explicitly preview/beta. The README says "do not use this preview release for production deployments." Model coverage is thin and the runtime is still gaining features like KV store and VLM support. Watch it, don't bet your inference layer on it yet.

Free MIT inference engine if you can afford Blackwell GPUs and accept the beta label.

nativ runs AI models locally on your Mac, built for Apple Silicon specifically using MLX, Apple's own ML framework. Chat with local models, do vision and image tasks, manage what you've downloaded, and expose an OpenAI or Anthropic-compatible endpoint so your coding tools can hit a model running on your own machine. MIT, free, no account. Setup is about as easy as local AI gets: download the DMG, drag it to Applications, launch. It auto-updates. The only real gate is hardware. You need an Apple Silicon Mac and macOS 26, which is a very new OS to require, plus enough unified memory for whatever model you load. For privacy-sensitive work, or just dodging API bills, running models locally is the move, and on a modern Mac nativ makes it painless, solo or small team, at no cost. It competes with LM Studio and Ollama front-ends; the difference is it's MLX-native and Mac-only by design. The catch: that Mac-only, macOS 26 requirement locks out everyone else, and a couple of the headline features (audio models, image generation as a chat tool) are still "coming soon" rather than shipped. Solid today, with asterisks.

Completely free and open source (MIT). No paid tier, no required keys.

TimeGPT predicts where your numbers are headed. Feed it a history of anything measured over time, daily sales, hourly server load, weekly signups, and it forecasts what comes next and flags the weird spikes. No training a model, no feature engineering, three lines of Python. The SDK is open source under Apache-2.0. Here is the important part. The SDK is open, the model is not. TimeGPT runs on Nixtla's servers, and your data goes to their API to get a forecast back. There is nothing to self-host, which is why setup is trivial, but it also means you are sending your numbers to a third party and paying per call. A free trial key lets you kick the tires. For prototyping and one-off forecasts, the free trial plus a few lines of code is impressive, you get a working forecast in minutes. For production, you are on a usage-based bill and a hard dependency on their uptime. Solo and small teams testing an idea: start free. Teams putting forecasts in front of customers: price the API against running your own model with Nixtla's open StatsForecast or Prophet. The catch is the closed model. You cannot audit it, you cannot run it offline, and if Nixtla changes pricing or pulls the service, your forecasts go with it. Open SDK, closed brain.

Free to prototype, usage-based in production; the model is closed and cloud-only, so budget for the API and the dependency.

claude-real-video lets an LLM actually watch a video instead of guessing from a title. It pulls the frames that matter, using scene-change detection so it grabs a new frame when the picture changes, not every few seconds, deduplicates the near-identical ones, and optionally transcribes the audio with Whisper. Out comes a set of frames, a transcript, and a manifest you feed to Claude or any model. The core is MIT and free, and it all runs on your machine. Setup is a notch above pip-install because it leans on ffmpeg, the video-processing workhorse, which you install once at the system level. After that it is Python 3.10+ and a package install, with Whisper optional if you want transcription. Nothing leaves your laptop, which matters if the footage is sensitive. The free tier is the useful 90 percent: frame extraction, dedup, transcript. A one-time Pro unlock (founder price around $19, not a subscription) adds analysis on top, camera-movement classification, editing-rhythm detection, and a perception timeline of gestures and sound events. For most people wanting to hand a video to an LLM, free is plenty. Pay the $19 only if you are doing real video teardowns and want the extra structure. The catch is scope. This prepares video for a model, it does not understand it, the understanding still happens in whatever LLM you pipe the output into. It is a very good front door, not the whole house.

Free core covers frame extraction and transcription; the one-time $19 Pro is only worth it for detailed video teardowns.

Mike is an open-source AI legal platform for document processing and review. Built on Next.js with an Express API, Supabase for auth and storage, S3-compatible object storage for files, and your choice of LLM provider for the analysis. AGPL-3.0. Setup is significant. You need a Supabase project, an S3-compatible bucket (Cloudflare R2 works), API keys for at least one model provider, and LibreOffice for DOC/DOCX-to-PDF conversion. The README is sparse, so plan for some reverse-engineering. Pick this if you run a small practice or a legal-ops team and want AI document review without sending client files to Harvey, Spellbook, or Casetext. Solo: free, you pay your model bill, the setup curve is real. Small teams: same plus shared infra. Large firms: they pay for Harvey because the integrations matter and they don't want to maintain this themselves. The catch: legal AI is a confidence and accuracy game, and Mike is early. Docs are thin, the feature set is whatever the README implies, and there is no SOC 2 posture out of the box. Use this because you want to own the stack, not because it's a turnkey product.

Open source. Real cost is your Supabase plan, R2 storage, and LLM usage.

Text-generation-webui gives you a browser-based interface to do it. Load a model, chat with it, fine-tune it, generate images. It's the Swiss Army knife for local AI. The entire project is free under AGPL-3.0. Every feature (chat, notebook mode, model loading, LoRA training, multimodal/vision support, extensions) ships at $0. The developer sells some extension packs on Gumroad, but those are optional add-ons, not core features. Self-hosting is the only option, and the setup complexity depends on your GPU situation. If you have an NVIDIA card with 8GB+ VRAM, the one-click installers work well. AMD and Apple Silicon support exists but can be finicky. Expect 30-60 minutes for first-time setup including downloading a model. Solo developers: this is your playground. Run models locally, experiment with fine-tuning, keep your data private. Small teams: share a beefy GPU server running the API mode. Beyond that, look at dedicated inference servers like vLLM. The catch: GPU hardware requirements are real. You need a decent GPU to run anything useful. A 7B parameter model needs ~6GB VRAM. Anything bigger needs proportionally more. No GPU, no party.

Software is completely free. The real cost is GPU hardware: $300-900 for a capable NVIDIA card.

Gemma Tuner lets you fine-tune Google's Gemma 4 and 3n models on your Mac, no cloud GPU required. Text, images, and audio, all via Apple Silicon's MPS backend. You bring a CSV of training data, point the wizard at a HuggingFace checkpoint, and watch the training run on your local GPU with a real-time visualizer showing loss curves, attention heatmaps, and memory pressure. The 2B and 4B parameter models are the sweet spot for consumer hardware. 16GB RAM minimum, 32GB recommended. It streams training data from Google Cloud Storage or BigQuery for datasets larger than your SSD. Exports land in HuggingFace SafeTensors format with guides for Core ML and GGUF conversion if you want to deploy on-device. Solo ML practitioners get local fine-tuning without paying $2-5/hr for cloud GPUs. Small teams prototyping custom Gemma models can iterate locally before scaling to cloud training. The wizard CLI makes the setup approachable even if you're not a PyTorch expert. The catch: Gemma only. No Llama, no Mistral, no other model families. Larger Gemma weights (26B+) are not supported. Audio fine-tuning on non-Mac platforms requires CUDA. And you still need a HuggingFace account with Gemma's license accepted before you can download the weights.

Free local fine-tuning on Apple Silicon. Saves $2-5/hr vs cloud GPU alternatives.

DaVinci-MagiHuman does it in one model. No separate video generation, no separate voice synthesis, no stitching. One 15-billion-parameter transformer takes text and a reference image and jointly produces video and audio. The numbers are real: 5-second 1080p video in 38 seconds on a single H100. Supports Mandarin, Cantonese, English, Japanese, Korean, German, and French. Beats Ovi 1.1 (80% win rate) and LTX 2.3 (60.9% win rate) in human evaluation. The full model stack is released: base model, distilled model, super-resolution model, and inference code. From Shanghai's GAIR Lab and Sand.ai. The catch: you need serious hardware. An H100 for the fast inference numbers, and the 15B parameter model isn't running on a consumer GPU. No license file listed; check before commercial use. And 'joint audio-video generation' is still early. The 5-second clip limit means this is for avatars and short-form content, not video production.

Free to use. You pay for GPU compute, and you'll need a lot of it.

Boogu-Image is an open source image-generation and editing model you run yourself. It comes in Base, Turbo, and Edit variants for text-to-image, fast photorealistic generation, and image-to-image editing, with notably strong Chinese and English text rendering. Code is Apache-2.0 and the weights are on Hugging Face and ModelScope, all free. Self-hosting a model like this means real hardware. You need a capable GPU and the patience to set up inference, and there is no official hosted API to fall back on. At version 0.1 it is early, so expect rough edges. The payoff is full control and no per-image cloud bill, which adds up fast once you generate at volume. This is for ML practitioners and builders who want a permissively-licensed image model they can run on their own GPUs, especially if bilingual text rendering matters to them. Solo with a good GPU: free to run. Teams generating at scale: free software, but budget for the hardware. If you want a turnkey API and no ops, the proprietary options like DALL-E, Midjourney, Firefly, and FLUX Pro are where you pay for convenience. The catch is the hardware and the early stage. This is weights and code, not a service, so the GPU and the setup are on you, and at v0.1 it is more a capable starting point than a finished product. If you have the silicon and want to own your image pipeline, it is a strong free option. If you do not, an API is less hassle.

Completely free and open source. The GPU hardware to run it is the real cost.

OnnxOCR is a fast, multilingual OCR engine, the technology that turns images of text into actual text you can use. It is a rebuild of the popular PaddleOCR that strips out the heavy PaddlePaddle training framework and runs on ONNX Runtime instead, which makes it lean and quick. It reads Simplified and Traditional Chinese, English, Japanese, and more, and handles tables, document layout, and even license plates. Apache-2.0 and free. Setup is a pip install with Python 3.8 or newer. You can run it locally with a test script, stand it up as a JSON API, or launch a browser UI, and Docker support is included. Because it dropped the training framework, it runs well on edge devices and on both ARM and x86, which is the whole point: OCR without dragging a deep learning stack along. Accuracy is reported to match PaddleOCR 3.0. Developers who need to pull text from images or scanned documents and would rather not pay per-call cloud fees should look here. Solo and small teams get production-capable OCR for free, running on their own hardware. Larger teams processing high volumes save the most, since cloud OCR APIs bill per image. It is free at every scale. The catch is that you own the deployment and the accuracy tuning. Cloud OCR services like Google Vision or AWS Textract hand you an API and a support line; here you manage the models and the edge cases yourself. For high volume or privacy-sensitive work, that trade is usually worth it.

Completely free and open source under Apache-2.0. The savings come from running OCR yourself instead of paying per-image cloud fees.

JoyAI-Echo is JD.com's open release for generating long video with synced audio from a text prompt. Not the usual five-second clip. It targets multi-shot sequences up to about five minutes and keeps the same character and voice consistent across shots. The weights are actually released, not just a paper and a demo page, which is the difference between a real tool and a press release. The catch arrives fast on hardware. You're looking at roughly 46GB of VRAM for the full setup and around 70GB of weights to download. That's an H100 or A100, data-center territory. There are reduced settings for smaller cards, with the obvious tradeoffs in quality and length. Read the license before you build anything on this. It ships under the LTX-2 Community License, which forbids commercial use. So it substitutes for Sora, Runway, Kling, and Veo for research, prototyping, and personal projects, but not for anything you plan to sell. For commercial long-form video you're back to the paid services, or to a permissively licensed model like the Wan and Hunyuan families. Between a non-commercial license and data-center GPU requirements, this is a research showcase for most people, not a production tool. Impressive that it exists and runs. Just know what you can and can't do with it.

Free to download, but the LTX-2 Community License forbids commercial use. Heavy GPU needs (40GB+ VRAM).

Flash-moe makes that possible. It uses a technique called Mixture of Experts (MoE) to run only the parts of the model that matter for each request, dramatically cutting the memory and compute needed. The pitch is simple: big model intelligence on small hardware. Models that normally need 32GB+ of VRAM can run on a laptop with 8-16GB of regular RAM. It's slower than running on a GPU, but it works. The catch: growing explosively but very early. The 'runs on a laptop' promise depends heavily on the model and your hardware. And MoE optimization is an active research area. Expect the approach to evolve fast.

Free. Check the license for commercial use.

Pyre-Code is a self-hosted coding practice platform for machine learning. 68 problems that walk you from implementing ReLU to building attention mechanisms, RLHF pipelines, and diffusion models. Browser-based editor with instant test feedback and reference solutions. No GPU required. Running it yourself takes a Next.js frontend and a FastAPI backend with SQLite for progress tracking. Standard Docker setup, nothing exotic. The whole point is to run it locally or on a team server so you can practice without leaking code to a third-party platform. ML engineers prepping for interviews or deepening their understanding of model internals will get the most from this. It's not a course or a tutorial: it's hands-on implementation practice where you write the code and the tests tell you if you're right. The catch: 68 problems is a fixed set. No community contributions yet, no problem editor, and the difficulty curve assumes you already know Python and basic linear algebra.

Completely free. MIT licensed, self-hosted, no catches.