Run and serve large language models: local inference, production serving, and model management.
Ranked by Discovery Score — our 0-100 composite of adoption, maintenance, community, and license health. How scoring works.
By Erik Loyd, SaaS CEO and former COO/CFO of an AWS Premier Partner.
Ollama makes running an LLM on your own machine dead simple. Download it, type ollama run llama3 in your terminal, and you are chatting with a model locally. No Python environments, no CUDA wrangling, no Docker. It is the most popular local LLM tool by a wide margin, supports dozens of models like Llama, Mistral, Gemma, DeepSeek, and Qwen, and runs on Mac, Linux, and Windows. The API is OpenAI-compatible, so anything built for the OpenAI API can point at Ollama instead and keep your data on your own machine. That local engine is MIT-licensed and free, and for most people it is the whole product. The newer wrinkle is Ollama Cloud. There is now a hosted option for running larger models than your hardware can handle, with a free tier, a Pro tier at 20 dollars a month, and a Max tier at 100 dollars a month for heavier use. The local engine stays free no matter what. Solo users and anyone privacy-minded run everything locally at no cost. Reach for the cloud tiers only when you want frontier-size models without buying the GPU to match. The catch is hardware, the same as it ever was. A Mac with 16GB of RAM runs 7B models fine; 70B and up needs serious GPU power, which is exactly the gap the paid cloud now fills. And local models still trail the best hosted models like Claude and GPT on the hardest tasks, so match the model to the job.
Local engine is free and open source. Ollama Cloud adds hosted tiers at $20/mo (Pro) and $100/mo (Max).
**Free / self-hosted:** The local engine is MIT-licensed and free. Run any supported model on your own hardware, OpenAI-compatible API, full privacy. The Cloud free tier also lets you touch hosted models to start. **Pro ($20/mo, or $200/yr):** Larger cloud models, run 3 cloud models at once, ~50x more cloud usage than free. **Max ($100/mo):** Run 10 cloud models at once, 5x more usage than Pro. A Team plan is coming. All cloud tiers are optional; the local engine never costs anything.
RAGFlow is a retrieval-augmented generation engine built around deep document understanding, and it has grown into an agent platform on top of that. It handles the messy parts of RAG other frameworks hand-wave past: PDFs with tables, slides, spreadsheets and scanned images, with MinerU and Docling parsing. Recent releases added knowledge compilation into wikis, graphs and mind maps, agent memory, and delivery into chat apps like Discord, Telegram and Line. Apache 2.0.
Self-hosting is a real project, not a docker run. You need 4+ CPU cores, 16+ GB RAM, 50+ GB disk, and x86 hardware, since there are no ARM64 images. The compose file brings up RAGFlow plus MySQL, Redis, object storage and a search backend: Elasticsearch by default, with Infinity, OpenSearch and OceanBase among the options. Template-based chunking is still the killer feature: you see and adjust how documents are split before they reach the index.
Solo developers and small teams that want full control can self-host for free. Everyone else can use RAGFlow Cloud, currently $29/mo for 5 seats on Starter or $129/mo for 20 seats on Pro, with Enterprise adding on-premises or bring-your-own-cloud deployment.
The catch is operational weight. Just chatting with a few documents? LlamaIndex or LangChain over a simple vector store ships faster. Note that v0.27 pulled the old GraphRAG and RAPTOR features from the UI, so check tutorials against your version.
Free under Apache 2.0 if you can run the stack. RAGFlow Cloud is currently $29/mo for 5 seats or $129/mo for 20.
**Free tier:** Apache 2.0 when self-hosted: deep-document parsing (tables, slides, spreadsheets, scans) with MinerU and Docling, template chunking with visual review, traceable citations, agentic RAG, knowledge compilation (wiki, graph, tree, mind map, timeline), agent memory, data connectors, chat-channel delivery, and an MCP server that is off by default. RAGFlow Cloud also has a free plan: 5 apps, 1 seat, 0.1 GB storage and 500 credits a month, without API access. **Self-hosted:** Real ops. 4+ cores, 16+ GB RAM, 50+ GB disk, Docker 24+ with Compose 2.26.1+, `vm.max_map_count` of at least 262144, and x86 hardware only. The stack runs RAGFlow with MySQL, Redis, object storage and a search engine (Elasticsearch by default; Infinity, OpenSearch, OceanBase and others supported). gVisor is needed only for the code executor sandbox. **Paid:** RAGFlow Cloud Starter is currently $29/mo (listed down from $59) for 50 apps, 5 seats, 5 GB storage, 5,000 credits and API access. Pro is currently $129/mo (down from $259) for unlimited apps, 20 seats, 50 GB and 20,000 credits. Enterprise is custom, with bring-your-own-cloud or on-premises deployment, dedicated support and a custom SLA.
Onyx is the self-hosted answer to enterprise ChatGPT, and its value is in where it plugs in. It indexes Slack, Google Drive, Confluence, GitHub, and 50-odd other sources, then answers questions about your own material with citations pointing back at the source document. The Community Edition is MIT licensed and covers chat, agentic RAG, custom agents, web search, code execution, and artifacts. Self-hosting comes in two shapes and the gap between them matters. Lite runs in under 1GB of memory on a simplified stack. Standard is the full system: vector and keyword indexing, background workers, ML inference servers, Redis, and MinIO blob storage, deployed through Docker, Kubernetes, or Helm. Standard is a real deployment with real operational weight, not a weekend project. The Enterprise Edition is where the money sits, and the split is defensible: SSO and SAML, role-based access control, analytics, query auditing, and whitelabeling. Solo and small teams get everything meaningful for free. Onyx Cloud has a free tier if you would rather not run it at all. The catch is that RBAC line. Permission-aware search, where the assistant only surfaces documents a given person is allowed to see, is exactly what a company with sensitive material needs, and it sits on the paid side. That is fair monetization, but price it before you roll this out company-wide.
MIT core covers chat, RAG, and agents. SSO, RBAC, and auditing are Enterprise.
**Free tier:** Community Edition under MIT. Chat, agentic RAG, custom agents, actions, web search, code execution, artifacts, voice mode, image generation, and 50+ indexing connectors. Onyx Cloud also offers a free tier. **Self-hosted:** Two profiles. Lite runs under 1GB of memory. Standard needs vector and keyword indexing, background workers, ML inference servers, Redis, and MinIO. Docker, Kubernetes, Helm, and Terraform are all supported. **Paid:** Onyx Cloud paid plans, plus the Enterprise Edition, which gates SSO/SAML, role-based access control, analytics, query auditing, custom code execution, and whitelabeling. Enterprise pricing is not published; you talk to them.
A server without a GPU. llama.cpp makes it possible. It runs quantized versions of open models (Llama, Mistral, Phi, Qwen, and dozens more) in pure C/C++ with optional GPU acceleration. No Python, no PyTorch, no CUDA dependency hell. Everything is free under MIT. No paid tier, no cloud, no account. Download a model file (GGUF format), point llama.cpp at it, and you're running inference. It includes a built-in HTTP server that exposes an OpenAI-compatible API, so your existing code that talks to GPT can talk to a local model with one URL change. The catch: you need hardware. A 7B parameter model needs ~4GB RAM (quantized). A 70B model needs ~40GB. Quality depends entirely on the model and quantization level; a heavily quantized model on a laptop won't match GPT-4. But for privacy-sensitive workloads, offline use, or just not wanting to pay per token, nothing else comes close.
Software is free. Cost is hardware: $0 on a laptop for small models, $700+ for serious inference.
Fully open source under MIT. The software is free; your cost is hardware. **Hardware math:** - 7B model (good for coding, chat): Runs on a modern laptop with 8GB RAM. Free if you already own one. - 13B model (better quality): Needs 8-10GB RAM. Still laptop-friendly. - 70B model (approaching GPT-4 quality): Needs ~40GB RAM or a GPU with 24GB+ VRAM. A used RTX 3090 runs ~$700. **Compared to API costs:** If you make 100K API calls/month to Claude or GPT-4, you're spending $500+/mo. A one-time $700 GPU investment pays for itself in 6 weeks for high-volume inference. **The hidden cost:** Your time. Setting up, choosing the right model, tuning quantization, and debugging performance issues is hours of work that an API call handles in milliseconds.
vLLM is the fastest way to serve open-weight LLMs on your own hardware. It takes a model like Llama or Mistral and puts an OpenAI-compatible API in front of it, squeezing maximum throughput out of your GPUs. What's free: everything. Apache 2.0 license. The entire inference engine, all optimizations (PagedAttention, continuous batching, tensor parallelism), the OpenAI-compatible API server. All free. vLLM's key innovation is PagedAttention, which manages GPU memory the way operating systems manage RAM, in pages instead of contiguous blocks. The result: 2-4x more throughput than naive inference. It's become the default serving engine for self-hosted LLMs. The catch: you need serious GPUs. Running a 70B parameter model requires 2-4 A100 GPUs ($1-2/hr on cloud, or $10K+ each to buy). Even a 7B model needs a decent GPU with 16GB+ VRAM. vLLM is free but the hardware is emphatically not. And it's optimized for NVIDIA GPUs. AMD ROCm support exists but is second-class.
Software is free. Hardware costs $0.50-32/hr in the cloud. Self-hosting beats API pricing only at massive scale or when data privacy is non-negotiable.
### What's Free Everything. Apache 2.0 license. All features, all optimizations, no restrictions. ### The Hardware Bill (This Is Your Real Cost) - **7B model (Llama 3.1 7B)**: 1x GPU with 16GB+ VRAM. Cloud: ~$0.50-1.00/hr. Buy: RTX 4090 ~$1,600. - **70B model (Llama 3.1 70B)**: 2-4x A100 80GB GPUs. Cloud: $4-8/hr (~$3,000-6,000/mo 24/7). Buy: ~$40K-80K. - **405B model**: 8x A100 or H100. Cloud: $16-32/hr (~$12K-24K/mo). Buy: you don't want to know. ### Cloud GPU Options - **RunPod**: A100 80GB at ~$1.64/hr. Good for experimentation. - **Lambda Labs**: A100 at ~$1.10/hr. Better for sustained use. - **AWS (p4d/p5)**: $12-40/hr. Enterprise-grade, enterprise-priced. ### vs Paying for API Access - OpenAI GPT-4o: $2.50-10/1M tokens. No hardware to manage. - Self-hosted Llama 70B via vLLM: ~$0.20-0.50/1M tokens at scale. But you're managing infrastructure. ### When Self-Hosting Makes Sense When: data privacy is non-negotiable, you're processing millions of tokens/day (cost crossover), or you need custom model fine-tuning. When not: you're processing <100K tokens/day (API is cheaper), or you don't have GPU expertise.
CLIProxyAPI wraps existing AI coding CLIs, Gemini CLI, Claude Code, ChatGPT Codex, and others, and exposes them as OpenAI/Gemini/Claude-compatible API endpoints. The pitch is that you get access to models like Gemini 2.5 Pro and GPT-5 through their free CLI tiers, served as a standard API you can plug into any app. Let me be direct: this is a proxy that routes around pricing by using free CLI tools as backends, and exploding because free model access is irresistible. The homepage points to a subscription service at z.ai. The catch: this sits in a gray area. You're wrapping free CLI tools and serving them as APIs, which likely violates the terms of service for most of those CLIs. The sustainability of this approach depends entirely on providers not shutting it down. The MIT license covers the code, but the underlying model access is not yours to redistribute. Use at your own risk.
Free to self-host but built on borrowed time. The upstream free tiers can disappear.
The CLI proxy itself is MIT-licensed and free to self-host. However, the homepage links to z.ai which offers a subscription-based hosted service. Self-hosted: free, but you're responsible for maintaining the CLI tool installations and handling rate limits from upstream providers. The 'free' model access depends on the free tiers of Gemini CLI, Claude Code, etc.; those providers can change terms at any time. Hosted (z.ai): subscription pricing, details on their site. The real cost is risk. If Google, Anthropic, or OpenAI crack down on CLI-to-API proxying, your integration breaks overnight.
llmfit answers the question everyone running local AI asks first: will this model fit on my machine? It reads your CPU, RAM, and GPU, then scores every model in its catalog on memory fit, estimated speed, quality, and context length. No more downloading a huge model to find out it will not load. MIT licensed, free, one Rust binary. Install with Homebrew, Scoop, MacPorts, uv, Docker, or a curl script, on macOS, Linux, or Windows. It detects NVIDIA, Apple Silicon, AMD ROCm, and Intel GPUs, understands GGUF, AWQ, GPTQ, and EXL2 quantization, and works with Ollama, llama.cpp, MLX, Docker Model Runner, and LM Studio. No account, and the README says nothing leaves your machine unless you ask. There is also a web dashboard and a REST API through llmfit serve. Free at every size with no paid tier. Solo builders get the most out of it. Teams can run the API on shared GPU boxes to see what fits where. Once you have picked a model, ollama/ollama or ggml-org/llama.cpp does the actual running. The catch: speeds are estimates from a formula unless you run the benchmark against a live runtime, and the GPU bandwidth table only covers about 80 cards. Windows and Intel Macs get weaker detection. The model catalog is baked in at compile time, so new models only show up when you update.
Completely free and open source.
**Free:** Everything. MIT licensed CLI and terminal UI, a web dashboard, a REST API, and a built-in catalog of hundreds of models sourced from Hugging Face. **Self-hosted:** A single binary on your own machine. No account, no model download needed to get recommendations. The benchmark command needs a local runtime like Ollama already running. **Paid:** None. No hosted version, no paid tier, no sponsor-gated features.
SGLang has become one of the default engines for serving open-weight LLMs in production at scale. It runs from a single GPU up to large clusters, with prefix caching, speculative decoding, and day-zero support for new model releases. The project says it runs on hundreds of thousands of GPUs, at xAI, NVIDIA and the major clouds among others. Apache-2.0, free.
Install is a uv pip install, with builds for NVIDIA, AMD, Intel, Google TPUs and Ascend NPUs. Models span Llama, Qwen, DeepSeek, Kimi, GLM, Mistral, diffusion models, and most of Hugging Face. It is OpenAI-API compatible, so existing clients drop in without rewrites, and it doubles as a rollout backend for reinforcement learning training.
Solo developers and small teams running open-weight models: this is one of the strongest options on the shelf, especially on DeepSeek or with heavy agentic workloads. Large teams running production inference at scale: you are very likely already evaluating it.
The catch is that serious inference is still serious work. Cold starts, KV cache tuning, and multi-node setups need real engineering. SGLang gives you a faster engine, not a managed platform. RadixArk, a venture-funded company among its maintainers, is building one on top, with no public pricing yet.
Free Apache 2.0 inference engine used widely in production. Hardware and ops are the cost.
**Free:** Apache 2.0. High-throughput LLM serving from one GPU to clusters, prefix caching, speculative decoding, day-zero model support, an OpenAI-compatible API, and use as an RL rollout backend. The project is hosted under the non-profit LMSYS organization. **Self-hosted:** The way it runs. Broad hardware: NVIDIA, AMD, Intel, Google TPU and Ascend NPU. Real ops for cold starts, KV tuning, and multi-node. **Paid:** No paid tier for SGLang itself. RadixArk, a company that co-maintains SGLang, sells managed inference infrastructure built on it, with no public pricing. Hardware is otherwise the only cost.
LiteLLM is a proxy and Python library that puts a unified OpenAI-compatible API in front of 100+ LLM providers: OpenAI, Anthropic, Gemini, Cohere, Azure, Bedrock, Ollama, and more. Write your code once using the OpenAI format and switch providers by changing one line. Run it as a proxy server and you get rate limiting, cost tracking, fallback routing, and load balancing across providers. Teams use it to control which models engineers can call, track spend per team, and add retry logic without touching application code. MIT-licensed, free to self-host. Engineering teams building on multiple LLMs or managing costs across a company get the most value from the proxy. Individual developers using it as a Python library just want to avoid rewriting LLM calls when switching providers. Both use cases are free. The catch: the proxy adds latency. Not much, usually under 10ms, but it is a network hop. And the feature set moves fast enough that staying current requires attention.
SDK is free. Proxy is free to self-host. Enterprise pricing is custom. Your real costs are the LLM API bills themselves.
### Free (SDK) The Python SDK is fully open source. Call 100+ LLM providers using the OpenAI format. Streaming, function calling, vision, embeddings, all supported. `pip install litellm` and you're running. ### Free (Self-Hosted Proxy) The proxy server adds centralized API key management, load balancing across providers, caching, rate limiting, spend tracking per user/team, and a management UI. Self-host with Docker. Apache 2.0 for core features. ### Paid (Enterprise) SSO/SAML, advanced audit logs, priority support, custom SLAs. Pricing is not public; contact sales. Likely $1,000+/mo based on comparable tools. ### Self-Hosted Costs The proxy itself is lightweight; a $10-20/mo VPS handles most workloads. Your real costs are the LLM API bills: OpenAI, Anthropic, etc. LiteLLM helps you track and optimize those costs but doesn't reduce them directly. ### What LiteLLM Saves You Without it: maintaining separate API integrations for each provider, custom retry logic, manual spend tracking. A team calling 3+ providers saves 20-40 hours of integration work upfront and ongoing maintenance. ### Verdict The SDK is free and worth using even for a single provider. The proxy is free to self-host and pays for itself in spend visibility.
LocalAI runs your own AI models locally and exposes them through an OpenAI-compatible API. LLMs, image and 3D generation, speech in both directions: all from a single server. No cloud, no API keys, no data leaving your machine. MIT-licensed, free. Docker-based setup handles most of the complexity. A config file defines which models to load and which backends to use (llama.cpp, whisper, stable diffusion, and more). CPU inference is supported, which means any machine can run it. GPU acceleration is faster but not required. Models download at first startup. Developers who want to swap out OpenAI API calls with local models point their existing code at LocalAI's endpoint and change nothing else. Good for privacy-sensitive applications, air-gapped environments, and teams that want to control costs without changing application code. The catch: local inference is slower than cloud for most hardware setups. Model selection lags the frontier. You get privacy and cost control; you give up raw performance and convenience.
Free. Your hardware is the only cost.
Fully open source under MIT. No paid tier, no hosted offering. All model types (text, image, audio, embeddings) run for free. Your only cost is hardware. A decent development setup: existing laptop/desktop with 16GB+ RAM. Production inference: GPU server ($200-2,000/mo cloud or buy your own).
oMLX runs large language models on your Mac and manages the whole thing from the menu bar. Pick a model, hit start, and you have a local OpenAI-compatible API at localhost:8000 that Claude Code, Codex, or any OpenAI client can point at. Apache 2.0, no account, no paid tier. The engineering underneath is more serious than the menu bar suggests. Continuous batching handles concurrent requests, and a two-tier KV cache keeps hot blocks in RAM and spills cold ones to SSD, so a long context survives a server restart instead of being recomputed from scratch. One process serves text models, vision models, OCR, embeddings, and rerankers, with model pinning, per-model idle timeouts, sampling settings, and a memory ceiling all editable from the admin dashboard. Solo developers running a local coding assistant get the most out of it, and a small team can share one big Mac over the API. Ollama (ollama/ollama) is easier to hand to a teammate and runs on any hardware. llama.cpp (ggml-org/llama.cpp) gives you more low-level control. oMLX wins specifically on Apple Silicon. The catch: Apple Silicon and macOS 15 or newer, full stop. And the fast native Metal kernels for the GLM and MiniMax families do not build from a plain pip install. Without full Xcode you silently fall back to a path the project measures at roughly 30x slower on prefill. Use the official DMG, which ships them precompiled.
Free under Apache 2.0. Your only cost is the Mac, and it has to be Apple Silicon on macOS 15 or newer.
### Free Everything. Apache 2.0, every feature, no account, no license key, no paid tier. The macOS app, the Homebrew formula, and the source build are the same software. ### Paid There is no paid tier. The only ask anywhere in the project is a Buy Me a Coffee link in the README. ### Self-Hosted Costs Your Mac and your electricity. Hard requirements: Apple Silicon (M1 through M4), macOS 15.0 or newer, and Python 3.11 to 3.13 for source installs. Memory is the real budget. 8GB runs small quantized models, 32GB and up is where the larger models stop thrashing. Model weights come from HuggingFace at no cost. Building the native Metal kernels needs full Xcode, not just Command Line Tools, or you take the precompiled DMG. ### When to Pay Never, to this project. Spend money when you outgrow a single machine: renting a GPU or calling a hosted API beats buying a second Mac for a workload you only run occasionally.
Langfuse is the observability platform for LLM applications. Instrument your code with a few SDK lines and every call becomes a trace: prompts, completions, latency, token costs, quality scores over time. Basically Datadog for your LLM stack, and one of the standard picks in the space. The core is open source and self-hostable for free. Version 4 landed as a breaking major, and it moved real capability into the self-hosted edition: full-text search across traces, a filter search bar, monitors and alerts, faster observation and metrics APIs, and the in-app agent is no longer enterprise-only. Upgrading from v3 means walking their migration guide, so plan the jump rather than winging it. Self-hosting runs on Docker Compose or Kubernetes with Postgres and ClickHouse behind it. Cloud pricing restructured: the free Hobby tier gets you started, Core is $29/mo, and Pro jumps to $199/mo. Solo: Hobby or self-host. Small teams: Core covers most needs, or self-host to skip per-observation math. At scale, self-hosting wins on cost long before the Pro tier stops stinging. The catch: Langfuse only watches LLMs. General app monitoring still needs Datadog or SigNoz, the space breeds new competitors monthly, and the breaking major means every v3 self-hoster owes an upgrade project before they see any of this.
Self-host free with no feature limits. Cloud starts free, Core is $29/mo, and the jump to Pro at $199/mo is where self-hosting starts paying for itself.
**Self-hosted (free):** The open source core, which v4 expanded: full-text search, monitors and alerts, and the in-app agent are no longer paywalled. Docker Compose or Kubernetes; Postgres plus ClickHouse at scale. Migrating from v3 requires the official upgrade guide, it's a breaking major. **Cloud Hobby (free):** Enough to evaluate and run side projects. **Cloud Core ($29/mo):** The workhorse tier for small teams. **Cloud Pro ($199/mo):** Higher limits and support. **Enterprise ($2,499/mo):** Compliance and scale features. **The math:** Metering is by observation volume, and busy LLM apps generate a lot of observations. Past Core, a $20-50/mo VPS running the self-hosted stack usually wins on cost; cloud wins on not operating ClickHouse yourself.
DwarfStar (the repo is still named ds4) is antirez's local inference engine for running a short list of big open models on hardware you own. The creator of Redis built it first for DeepSeek V4 Flash, and it now also runs DeepSeek V4.1 Flash, V4 PRO, the GLM 5.x family, and Qwen3.8 Flash Next on Metal, behind an OpenAI- and Anthropic-compatible server. It only loads the GGUF files the project itself produces. MIT, free.
The price is hardware, not software. Macs with 96 GB or more are the primary target, and smaller machines can stream weights off SSD. On Linux it runs on CUDA, including older Ada Lovelace cards like the L40S, plus DGX Spark and Strix Halo boxes. Setup is a download script and make. Weight files run to hundreds of gigabytes, so budget a big, fast SSD too.
Beyond the chat CLI it ships ds4-agent, a built-in coding agent, plus image input, disk KV caching and inference split across machines. A solo developer with a loaded Mac Studio gets a near-frontier model with nobody else in the loop. Everyone else should stay on llama.cpp or vLLM until they own the metal.
The catch is churn. antirez calls it beta quality, warns that regressions are possible and that models get dropped when better ones arrive, and says openly it was built with strong help from AI coding agents. Run it to own your inference stack, not to ship a product on it.
Free under MIT. The bill is a 96 GB-plus Mac or serious GPUs, plus hundreds of gigabytes of fast SSD.
**Free tier:** MIT licensed, with retained llama.cpp and GGML pieces also under MIT. Supports DeepSeek V4 Flash (including an experimental vision model), DeepSeek V4.1 Flash, DeepSeek V4 PRO, GLM 5.2, GLM 5.3 and GLM 5.3 Flash, and Qwen3.8 Flash Next on Metal. Includes the CLI, the `ds4-agent` coding agent, an OpenAI- and Anthropic-compatible HTTP server, SSD weight streaming, disk KV cache, and distributed inference across machines. It only runs the project's own GGUF files, not arbitrary models. **Self-hosted:** The only way it runs. Metal on Macs with 96 GB or more is the primary target; a 64 GB Mac can run Flash Q2 with SSD streaming. CUDA covers DGX Spark and multi-GPU Ada Lovelace systems such as L40S cards, and ROCm covers Strix Halo machines like the Framework Desktop. Two 128 GB Macs over RDMA can split a 4-bit Flash model with tensor parallelism. Weight files are very large (the DeepSeek V4.1 Flash Q2 download is 341 GiB), so fast storage is part of the bill. **Paid:** None. The cost is the hardware.
mlx-lm runs and fine-tunes large language models directly on a Mac. Point it at a model on Hugging Face and one command pulls it down and runs it locally, using Apple's own MLX engine instead of a cloud API or a separate GPU rig. MIT licensed, free, and built by Apple's own ml-explore team, the same group behind MLX itself. It does more than run models. You can quantize them down to 4-bit, fine-tune with LoRA or full-model training, serve with streaming and prompt caching, and even split work across multiple machines. Setup is close to trivial: pip install mlx-lm, then a single command chats with a model. The real constraint is memory. MLX uses the Mac's unified memory, so the model has to roughly fit in RAM, and pushing past that needs macOS 15 or newer plus some system tuning. And it's Apple Silicon only. No M-series chip, no mlx-lm. The honest framing on competition: this is a building block, not a finished app. llama.cpp is the closest peer and runs on more hardware; Ollama and LM Studio are more packaged and app-like, and increasingly use MLX under the hood anyway; vLLM is for datacenter GPUs, a different world. mlx-lm's edge is being the MLX-native option, which means the best raw performance on a Mac and the cleanest fine-tuning story. Solo developers and researchers on Apple Silicon: this is the fast path. Small teams can build on it; larger production serving will want something server-side. The catch is that you're trading convenience and reach for Mac-native speed. It's lower-level than Ollama, locked to Apple hardware, and capped by how much RAM you bought. Within those lines, nothing runs models on a Mac better.
Free and open source. Your only cost is owning an Apple Silicon Mac with enough RAM.
**Free:** MIT-licensed, fully open, no paid tier. Running, quantizing, fine-tuning, and serving, all free. **Cost you'll actually pay:** A Mac with enough RAM. The software is free; the hardware is the spend, and RAM is the limiting factor for model size. **The trade:** Self-hosting on a Mac you already own means no per-token cloud bill. The ceiling is your RAM, not your budget.
headroom strips the bloat out of everything an LLM agent reads before it hits the prompt, tool outputs, logs, RAG chunks, raw file dumps, conversation history. The reported cuts are large, 60 to 95 percent fewer tokens with the same answers, and the compression is reversible, so the original is cached and fetchable when the model actually needs it. Apache-2.0, local-first, and free: your content never leaves your machine. You can adopt it at whatever depth you want. Drop the Python or TypeScript SDK inline, run it as a zero-code local proxy, or wrap an agent directly, Claude Code, Cursor, Copilot, and Aider all supported, plus an MCP server. It also trims the model's output, not just its input, through verbosity steering and effort routing, and headroom learn mines your failed sessions for auto-corrections. Development is fast-moving, this is not a stale side project. For anyone running agents on metered tokens, the math is simple: less context in and out means a smaller bill, and the local proxy makes trying it nearly free of effort. Solo developers and small teams: the open Apache-2.0 build is the whole tool, use it. Larger teams wanting shared deployment get a managed enterprise offering on top, which is where the money is, cross-team memory and central management rather than a better compressor. The catch is trust in the compression. Cutting 90 percent of the tokens only works if the 10 percent kept is the right 10 percent, and reversible caching exists precisely because sometimes it is not. Watch the first few runs on real work before you wire it into production blind.
The open Apache-2.0 build is the complete tool and free; pay only for managed team deployment and shared memory.
**Free (open source, Apache-2.0):** The full compressor, local-first. SDK, proxy, agent wrapping (Claude Code, Cursor, Copilot, Aider), MCP server, reversible compression, output shaping, and headroom learn. Your data stays local. **Self-hosted:** It is local by default, nothing leaves your machine. **Paid (managed enterprise):** A hosted and managed offering for team deployments, adding shared cross-agent memory and central management. Priced for organizations.
OpenMAIC turns a topic into a classroom of AI agents that teach and argue while you watch. AI teachers lecture while AI peers discuss and push back, so what you take away comes out of the exchange instead of one flat chatbot answer. Built by Tsinghua University researchers, grounded in a paper published in the Journal of Computer Science and Technology, and free to run. You run it yourself and bring your own model keys. Recent releases added a Pro Mode that turns any generated slide into an editable canvas, plus MP4 export, so a session can leave as a video file instead of living in a browser tab. Setup is a build step and API keys, not a one click install, and output quality tracks whichever model you point it at. The big change since spring: the project relicensed from AGPL-3.0 to MIT in June 2026. That takes the legal question off the table for anyone who wanted to embed this in a commercial product or offer it as a hosted service. Solo learners and teachers: run it locally and pay only for tokens. Teams building training content: usable now, and MIT means you can fork it into your own stack without a lawyer. The catch: multiple agents means multiplied token bills. A thirty minute session with three agents costs roughly three times what the same ground covered by a single chatbot would, and video export adds render time on your own machine. It is still an academic codebase, so expect rough edges where a product would have polish.
Free and MIT licensed since June 2026. Your only cost is model tokens, and three agents burns roughly three times as many.
**Free tier:** The whole project. MIT licensed as of v0.3.0 (June 2026), relicensed from AGPL-3.0. No feature gating, no hosted upsell, no account required. **Self-hosted:** The only way to run it. Clone, build, supply your own LLM API keys. Runs on a laptop. The cost that matters is not infrastructure, it is tokens: every agent in a session is a separate conversation with the model, so a three agent classroom bills roughly three times a solo chat of the same length. Video export renders locally and costs CPU time, not money. **Paid:** None. There is no commercial tier and no vendor to buy from. Your bill comes from OpenAI, Anthropic, DeepSeek, or whichever provider you point it at, or from nothing at all if you run a local model. **The MIT change matters:** under AGPL-3.0, offering a modified OpenMAIC as a service obligated you to publish your changes. That blocked commercial deployment for a lot of teams. As of June 2026 that restriction is gone.
Civitai is the community hub for sharing Stable Diffusion models, LoRAs, and checkpoints. Think of it as GitHub for AI image generation models: browse what others have trained, download fine-tuned versions, and share your own. Self-hosting requires Docker, Node.js 20+, PostgreSQL, and a fair bit of setup. The docker-compose gets you running but managing model storage at scale is the real operational challenge. Most people just use civitai.com directly. Useful for solo creators experimenting with image generation or teams building products on top of community models. The platform itself is free and open source. The catch: the self-hosted version is meant for development, not production. The hosted platform at civitai.com is where the community and model library actually live.
Completely free and open source. Civitai.com offers optional paid memberships for faster downloads.
## Free Tier The platform and codebase are completely free and open source. Browse, download, and share AI models at no cost on civitai.com. ## Self-Hosted Self-hosting requires Docker, Node.js 20+, PostgreSQL. The self-hosted version is intended for development, not production use. Model storage at scale is the main operational challenge. ## Paid Optional Buzz membership on civitai.com provides faster downloads and priority generation. The open source code itself has no paid tier.
LiteRT-LM runs language models directly on a device, no cloud and no internet required. The model lives on the phone, laptop, smartwatch, or even in the browser, so data never leaves the hardware, it works offline, and there's no per-query bill. This is Google's own framework, and Google uses it to power on-device AI in Chrome, Chromebook Plus, and the Pixel Watch. Apache 2.0, completely free. It's cross-platform by design, targeting Android, iOS, desktop, the web via WebGPU, and small boards like Raspberry Pi, and it taps GPU and NPU acceleration instead of grinding on the CPU. It runs open models like Gemma, Llama, Phi, and Qwen. The work isn't running a server, because there is no server. The work is on the build side: you obtain and convert models into the right format, then wire up the native SDK for each platform you ship to, and manage on-device memory per device class. Heavier than calling a cloud API, far lighter than operating an inference cluster. The real competition is other on-device runtimes. llama.cpp has broader model coverage and a bigger community; Meta's ExecuTorch is the closest vendor-backed rival; Apple's MLX wins on Macs but only on Macs. LiteRT-LM's edge is tight, official integration with Android and Google silicon. It doesn't replace a paid product so much as move certain workloads off the paid-API meter: the small and mid-size models you'd otherwise rent from a cloud. Solo and small teams shipping mobile or edge apps: this is the Google-blessed path. Larger teams already on Android get first-party support. The catch is maturity. The core runtime is production-ready and shipping in real Google products, but some bindings, Swift and JavaScript among them, are still early preview, and the project is young. And on-device models are not frontier models. If you need GPT-class quality, this isn't that. It's for when private, offline, free, and good-enough beats cloud-quality.
Free and open source. The only cost is your device's compute and the engineering to integrate it.
**Free:** Everything. Apache 2.0 framework with no paid tier, no hosted upsell, no license fee. The models it runs (Gemma and friends) are downloaded separately and are themselves free. **Cost you'll actually pay:** Your own device compute and the engineering time to integrate it per platform. That's it. **The trade:** Running inference on-device means you avoid paid cloud-API costs entirely for the workloads it can handle. The ceiling is model quality, not price.
Open WebUI gives you a ChatGPT-style interface for your own models, whether they run locally through Ollama, through OpenAI's API, or any compatible endpoint. Chat with models, upload documents for RAG (retrieval-augmented generation, meaning the AI reads your files and answers questions about them), manage conversations, and share prompts. All of it on your own server. The polish is the surprise. It feels like a commercial product rather than a weekend project: multi-user accounts, conversation history, model management, function calling, web search integration, and image generation. It is the most feature-complete self-hosted LLM front end available, and standing it up is a Docker command plus whatever inference backend you point it at. There is no first-party cloud, by design; the pitch is that you run it, on-premise, private cloud, or air-gapped. Standard internal use is free, with the branding left exactly as shipped. Solo and small teams: run it, there is no catch at your size. The catch shows up at scale, and it is mostly about whose name is on the screen. The license is not open source. It is BSD-3 plus a branding clause: you may not remove, alter, or replace the Open WebUI name and logo once a deployment passes fifty end users in a rolling thirty-day window, unless you buy an enterprise license or get written permission. A few enterprise-exclusive capabilities sit behind that same license too, so "everything is free" is not quite right. Read the license before building a business on top of it. The other cost is hardware, since a 7B model wants 8GB of RAM or a decent GPU and Open WebUI itself is the light part.
Free for standard internal use with branding intact. Above fifty users, white-labeling and a few enterprise-only capabilities require a paid license.
**Free tier:** Standard internal use, with the original branding intact, is free. Multi-user accounts, RAG, function calling, web search, image generation, and model management all ship with no plan gate and no seat count. **Self-hosted:** The only way to run it, and that is deliberate. A Docker command gets the front end up; deploy on-premise, in a private cloud, or fully air-gapped. The real bill is inference: a 7B model wants roughly 8GB of RAM or a decent GPU, and larger models scale from there. Open WebUI itself is lightweight next to whatever it talks to. Updates ship fast, which is good for features and means occasional breaking changes, so pin your image tag rather than tracking latest in production. Managed hosting exists from third parties like Elestio and Railway, but that is someone else's service wrapped around the same open project. **Paid:** An enterprise license covers two things. First, branding: above fifty end users in a rolling thirty-day window you must keep the Open WebUI name and logo intact unless you buy it or obtain written permission. Second, a handful of enterprise-exclusive capabilities such as Terminals. Pricing is not published. Most of what you would pay for is the right to take their name off the screen, but it is not only that.
Unlimited-OCR is Baidu's open model for turning images and PDFs into text. Point it at a scanned contract, a multi-page report, or a screenshot full of text and it reads the whole thing in one pass, even very long documents. It handles single images, batches, and full PDFs, with streaming output so you see results as they parse. The weights are on Hugging Face and ModelScope under the MIT license, which means free to use, including commercially, with no strings. Running it yourself is where the cost shows up. This needs an NVIDIA GPU, CUDA 12.9, Python 3.12, and a recent PyTorch and transformers stack. You can run it through plain transformers or through SGLang for faster batch serving. It builds on DeepSeek-OCR, so the lineage is solid, but you are still standing up GPU infrastructure and a model server. There is no hosted API here, no dashboard, no support line. You bring the hardware. For solo work, the Hugging Face demo is free to try and you can run it on a rented GPU when you have a real job. Small teams that already process documents will want a dedicated GPU box or a cloud GPU. Larger teams treating OCR as a pipeline should put SGLang behind a queue. If you need OCR as a managed service with an SLA, look at a cloud vendor instead. The catch: MIT covers the code and weights, but accuracy on messy real-world scans is the thing you have to test on your own documents before trusting it. No benchmark replaces running it on your actual files.
Free and MIT licensed, including for commercial use. Your only cost is the GPU you run it on.
**Free:** The model weights are published on Hugging Face and ModelScope under the MIT license. Free to download, run, and use commercially with no restrictions. A live demo runs on Hugging Face Spaces for testing without any setup. **Self-hosted:** This is the only real way to run it in production. You supply the hardware: an NVIDIA GPU, CUDA 12.9, Python 3.12, and a recent PyTorch and transformers stack. Inference runs through plain transformers or through SGLang for faster batch serving. No license fee, but you pay for the GPU, whether that is a box you own or a rented cloud instance. **Paid:** There is no hosted API, paid tier, or commercial offering from Baidu here. If you want managed OCR with an SLA, that means a different vendor, not this project.
TimeGPT predicts where your numbers are headed. Feed it a history of anything measured over time, daily sales, hourly server load, weekly signups, and it forecasts what comes next and flags the weird spikes. No training a model, no feature engineering, three lines of Python. The SDK is open source under Apache-2.0. Here is the important part. The SDK is open, the model is not. TimeGPT runs on Nixtla's servers, and your data goes to their API to get a forecast back. There is nothing to self-host, which is why setup is trivial, but it also means you are sending your numbers to a third party and paying per call. A free trial key lets you kick the tires. For prototyping and one-off forecasts, the free trial plus a few lines of code is impressive, you get a working forecast in minutes. For production, you are on a usage-based bill and a hard dependency on their uptime. Solo and small teams testing an idea: start free. Teams putting forecasts in front of customers: price the API against running your own model with Nixtla's open StatsForecast or Prophet. The catch is the closed model. You cannot audit it, you cannot run it offline, and if Nixtla changes pricing or pulls the service, your forecasts go with it. Open SDK, closed brain.
Free to prototype, usage-based in production; the model is closed and cloud-only, so budget for the API and the dependency.
**Free:** A trial API key and the fully open-source SDK (Apache-2.0). Enough to prototype, benchmark, and run small forecasts. **Self-hosted:** Not available for the TimeGPT model itself, it is closed and cloud-only (an Azure private deployment is listed as coming). If you need local, Nixtla's own StatsForecast and NeuralForecast libraries are fully open and run on your hardware. **Paid:** Usage-based API pricing after the trial, billed per forecast call. Snowflake integration available via stored procedures.
TensorRT-LLM squeezes maximum inference performance out of NVIDIA GPUs for large language models. It handles quantization (FP8, FP4, INT4), custom attention kernels, paged KV caching, and multi-GPU deployment through a Python API. If you are serving LLMs at scale on NVIDIA hardware, this is the optimization layer that makes the economics work. Running it yourself means you need NVIDIA GPUs, full stop. No AMD, no Apple Silicon, no CPU fallback. You will also need CUDA installed and compatible driver versions. The setup is not trivial, but NVIDIA provides containers and Docker images that smooth out the worst of it. Once running, the performance gains over naive PyTorch inference are substantial, often 2-4x throughput improvements. For teams already committed to NVIDIA hardware, TensorRT-LLM is the right call over vLLM when you need every last token per second. vLLM is easier to set up and supports more hardware. llama.cpp is better for local, single-GPU experimentation. TensorRT-LLM is for production serving where GPU cost is a real line item. The catch: you are locked to NVIDIA forever. The library only works on their GPUs, and if your cloud costs push you toward AMD or custom silicon, you are rewriting your inference stack from scratch.
Software is free. The real cost is NVIDIA GPU hardware or cloud GPU rental.
### Free Tier Free under Apache 2.0. Requires NVIDIA GPUs (no AMD/Intel support). ### Self-Hosted Heavy setup. Requires NVIDIA GPU with sufficient VRAM, CUDA toolkit, and Docker. The optimization pipeline involves model conversion and compilation steps. ### Paid None for the software. The cost is NVIDIA hardware. A single A100 80GB runs K+, though cloud GPU instances start around -3/hr.
kimi-k3-in-c runs a 2.78-trillion-parameter model on a machine with 8 GB of RAM, and the claim holds up. It's a from-scratch inference engine in portable C99, no BLAS, no GPU, no framework. Kimi K3 is a mixture-of-experts model where only a handful of experts fire per token, so the engine streams weights from disk and caches the few it needs. Apache 2.0, free. The headline number is real but it's not the one that matters. You need roughly 1.7 TB of fast NVMe for the checkpoint, Linux on x86-64 with AVX2, and patience: generation runs seconds to tens of seconds per token depending on hardware and memory budget. More RAM buys real speed, but nothing close to interactive. Output is byte-identical at every memory budget, validated against a PyTorch reference, which is the kind of engineering discipline most projects skip. Use this to learn how MoE inference actually works, to benchmark against, or to prove a point. Don't use it to serve anything: Moonshot's hosted API is faster and cheaper for real workloads, and llama.cpp is the right tool for models that fit your hardware. The catch: it's a weeks-old, essentially solo reference implementation, and the model weights carry Moonshot's own license, not Apache. Treat it as a brilliant teaching artifact, not infrastructure.
Free to run if you have 1.7 TB of NVMe to spare. For actual work, the hosted Kimi API wins on both speed and cost.
**Free:** Everything. The engine is Apache 2.0; the Kimi K3 weights download free from Hugging Face under Moonshot's separate model license. **Self-hosted:** The only mode. Budget about 1.7 TB of fast NVMe for the 1.56 TB checkpoint plus repacked trunk, a Linux x86-64 box with AVX2, and expect tens of seconds per token. More RAM buys speed, never different output. **Paid:** Nothing to buy here. If you want Kimi K3 at usable speed, Moonshot's hosted API is the paid path, and the project itself concedes it's faster and cheaper for real work.
GPT-Load puts one base URL and one access key in front of every model provider you use. Your app stops caring whether a request lands on OpenAI, Anthropic, Gemini, Bedrock, or Groq. Channels, credentials, models, and routing policy all live in a web console. MIT, Go, free. Deployment is Docker Compose with application-managed SQLite by default, MySQL or Postgres if you outgrow it. It binds to localhost, the right default for something holding every API key you own, so budget time for TLS and firewall work before exposing it. One file matters most: the encryption.key file decrypts all stored credentials, this version cannot rotate it, and losing it makes every saved credential unrecoverable. Solo devs with a single API key do not need this. Small teams running several providers who want per-key usage and cost visibility are the sweet spot. Above that, read the fine print: 2.0 runs as a single instance and shares no state, so horizontal scaling is off the table. LiteLLM is the better call at that size. The catch is the feature that draws the traffic. Alongside API keys, GPT-Load pools subscription accounts through OAuth, Codex, Claude, Antigravity, and Grok, behind one endpoint. Those flows are unofficial and can break whenever an upstream changes, and the README itself tells you to connect only accounts you are entitled to use. That is the real constraint, not boilerplate.
Free and MIT with nothing to buy. Your cost is operator time plus the upstream provider bills it routes to.
**Free tier** Everything. MIT license, no paid edition, no hosted cloud, no enterprise tier. The official site sells nothing. **Self-hosted** The only mode. Docker Compose with application-managed SQLite is the default; `DATABASE_DSN` switches it to MySQL or Postgres. Native binaries ship for Linux, macOS on amd64 and arm64, and Windows, plus a Windows installer that registers a low-privilege service. The real cost is operator time: TLS and firewall work before any remote exposure, and a backup routine that keeps the database and `encryption.key` together, because credentials are unrecoverable without the key and this version has no master key rotation. **Paid** Nothing to buy from the project. Your bill is whatever the upstream providers charge. Note that GPT-Load's own usage and cost figures are estimates derived from upstream responses, not invoices, so do not reconcile finances against them.
waste does something that shouldn't be possible on a laptop: run the full 2.8-trillion-parameter Kimi K3 model on modest RAM by streaming the activated expert weights straight off an NVMe drive. It's a dependency-free, embeddable C inference engine with documented multimodal (vision) support, from the team behind sqlite.ai. The headline isn't hype, Kimi K3 is a real Moonshot model released in mid-2026. The trick is that a mixture-of-experts model only lights up a fraction of its weights per token, so waste keeps the cold weights on disk and streams what it needs. That's how a 3-trillion-class model fits on consumer hardware. Free under Apache 2.0, for anyone at any scale who wants to poke at frontier-size models without a datacenter. The catch is speed, and it's a big one. You're looking at roughly half a token per second on a 64GB MacBook, and you need close to a terabyte of NVMe to hold the model. It's explicitly pre-production, the format and API aren't frozen. This is a fascinating proof that disk-streaming inference works, not something you'd build a product on. Run it to learn, not to serve.
Free and open source. A research curiosity, not a workhorse.
Free and open source under Apache 2.0, no paid tier. The cost is hardware and patience: it needs roughly 982GB of fast NVMe to hold the Kimi K3 container and runs at around 0.45 to 0.62 tokens per second on a 64GB Mac. Treat it as a research engine, not a serving stack.
This is the 'awesome list' for AI. Models, tools, infrastructure, datasets, organized by category with brief descriptions and links to the actual projects. Awesome lists live or die by curation quality. This one focuses on 'truly open source,' not source-available, not 'open weights with a restrictive license.' That distinction matters when you're building on top of these tools. The list is maintained on GitHub and follows the awesome-re standards. The catch: awesome lists are snapshots. They go stale unless someone actively maintains them, and the growth spike suggests this was recently featured somewhere. The real question is whether it'll be maintained in 6 months. Also, 'curated' means one person's opinion of what's worth including; your needs might differ. Use it as a starting point, not a definitive source.
Free. It's a curated list on GitHub.
### Free Tier Free curated list. Reference resource, not software. ### Self-Hosted Not applicable. ### Paid None.
Rapid-MLX runs open models locally on Apple Silicon, and speed is the whole pitch. One command installs it (Homebrew, pip, or a curl script), one command serves an OpenAI-compatible API, and it also speaks the Anthropic messages format, so Claude Code, Codex CLI, and Aider can point at it as a drop-in backend. Apache 2.0, completely free. The engineering focus is on the things that make local agent workflows fail in practice. A prompt cache brings claimed time-to-first-token under a tenth of a second on repeat prompts, and tool calling is handled across 17 parser formats, which matters because local models break agent loops on malformed tool calls far more often than on reasoning. It's in Homebrew core and publishes to PyPI, with releases landing near daily. Mac-only by design. If you run agents against local models on an M-series machine, try it head-to-head with Ollama or LM Studio; the performance claims ship with benchmarks in the repo rather than vibes. On Linux or Windows, look at Ollama or vLLM instead. The catch: it's young and moves extremely fast, the headline numbers are the project's own, and independent benchmarks are still thin. And no inference engine fixes the ceiling: a fast small model is still a small model.
Completely free and open source. Your only spend is the Mac it runs on.
**Free:** Everything. Apache 2.0, installable via Homebrew, pip, or the project's install script. Chat CLI, OpenAI-compatible server, Anthropic-format endpoint, tool calling, prompt cache, and optional vision/audio extras all ship in the open package. **Self-hosted:** The only mode, and it's one binary on your Mac. No accounts, no keys, no telemetry toll. **Paid:** Nothing. There's no cloud tier or commercial edition. The spend is hardware: an Apple Silicon Mac with enough unified memory for the models you want to run.
TokenSpeed is an LLM inference engine aimed at agentic workloads. The claim is TensorRT-LLM-level performance with vLLM-level usability, which is bold positioning if it holds up. The architecture uses a local-SPMD modeling layer with static compilation and a C++ control plane with type-safe KV cache management. The team has shipped benchmarks against TensorRT-LLM on Kimi K2.5 that look favorable. Hardware target is NVIDIA Blackwell (B200) right now, with Hopper and AMD MI350 optimization in progress. Setup involves the usual NVIDIA stack: CUDA, drivers, the lightseek.org/tokenspeed getting-started guide, and Blackwell-class hardware you almost certainly don't own personally. Currently it runs Kimi K2.5; Qwen, DeepSeek, and MiniMax support is in progress. If you're standing up an inference service for agent workloads on Blackwell GPUs, this is worth evaluating against vLLM and TensorRT-LLM. Solo and small teams: stick with vLLM until TokenSpeed matures. Large teams running serious agent workloads on B200s: benchmark it, the agentic optimizations look real. The catch: explicitly preview/beta. The README says "do not use this preview release for production deployments." Model coverage is thin and the runtime is still gaining features like KV store and VLM support. Watch it, don't bet your inference layer on it yet.
Free MIT inference engine if you can afford Blackwell GPUs and accept the beta label.
**Free:** MIT license, full source. No usage limits, no telemetry tier. **Self-hosted:** Designed to run on your own NVIDIA Blackwell hardware. B200s start around $30,000 per card, so the GPU cost dwarfs everything else. **Paid:** None. It's an open source inference engine; you pay for the hardware it runs on.
RikkaHub is a native Android app for chatting with any LLM through one interface. Instead of separate apps for ChatGPT, Claude and Gemini, you point it at any OpenAI, Anthropic or Google-compatible endpoint, including your own, and switch providers between conversations. It handles markdown, image, PDF and document input, web search, message branching, memory and MCP. AGPL-3.0, free. Install it from Google Play or the project site, add your API keys and go. There is nothing to self-host: the app talks directly to the providers you configure. Newer builds add a Workspace, a lightweight Linux environment built on proot that runs on the phone without root so agents can gather information and assemble documents, plus a web interface you can open in a browser. For anyone on Android juggling multiple AI providers, this is the clean way to do it. You pay each provider for tokens, or point it at a local model endpoint and pay nothing. It is a personal app, and there is no team tier. The catch is trust in the chain. A multi-provider client is only as private as the endpoints you configure, and the README itself warns that many forks exist, some leaking privacy or asking for excessive permissions. Install the official build and point it at providers you trust.
Free under AGPL-3.0. You pay the model providers you connect, or nothing with a local endpoint.
**Free tier:** Everything, under AGPL-3.0. The project moved to plain AGPL-3.0 in July 2026, replacing an earlier split license that required a commercial agreement for commercial use or more than 10 users. The Android app is free on Google Play and the project website; donations go through Patreon and Afdian. **Self-hosted:** Not applicable. The app runs on your phone and calls whatever LLM and search endpoints you configure. The Workspace feature runs a proot Linux environment locally without root. **Paid:** None for the app. You pay each LLM or search provider you connect, or nothing with a local endpoint.
MOSS-TTS-Nano is a multilingual text-to-speech model with only 100 million parameters that runs in real time on a CPU. No GPU required. It now handles 20 languages, including English, Chinese, German, Spanish, French, Japanese, and Korean, with solid quality for its size. Small enough to embed in desktop apps, local demos, or lightweight web services. Apache 2.0. Runs via Python with standard ML dependencies, and there is now an ONNX CPU build that nearly doubles throughput. Models are on HuggingFace and ModelScope, finetuning code is published for custom voices, and recent additions include mlx-audio support for Apple Silicon and a browser reader extension. The deployment stack stays simple: no CUDA, no heavy inference server, a basic machine handles it. Free for everyone under a permissive license. Solo developers building voice features get real-time TTS without paying per-character API fees. Teams shipping products can embed it directly without usage limits. The catch: 100M parameters means tradeoffs in naturalness and expressiveness. If you need the best possible voice quality, larger models or paid APIs like ElevenLabs will sound better. This is the right call when you want speed, low cost, and local execution over peak fidelity.
Completely free. Trade peak voice quality for zero cost and local execution.
**Free tier:** Everything. Full model, finetuning code, all languages. **Self-hosted:** Python, CPU-only. No GPU needed. HuggingFace model downloads. **Paid:** No paid tier. Open source with permissive licensing.
claude-real-video lets an LLM actually watch a video instead of guessing from a title. It pulls the frames that matter, using scene-change detection so it grabs a new frame when the picture changes, not every few seconds, deduplicates the near-identical ones, and optionally transcribes the audio with Whisper. Out comes a set of frames, a transcript, and a manifest you feed to Claude or any model. The core is MIT and free, and it all runs on your machine. Setup is a notch above pip-install because it leans on ffmpeg, the video-processing workhorse, which you install once at the system level. After that it is Python 3.10+ and a package install, with Whisper optional if you want transcription. Nothing leaves your laptop, which matters if the footage is sensitive. The free tier is the useful 90 percent: frame extraction, dedup, transcript. A one-time Pro unlock (founder price around $19, not a subscription) adds analysis on top, camera-movement classification, editing-rhythm detection, and a perception timeline of gestures and sound events. For most people wanting to hand a video to an LLM, free is plenty. Pay the $19 only if you are doing real video teardowns and want the extra structure. The catch is scope. This prepares video for a model, it does not understand it, the understanding still happens in whatever LLM you pipe the output into. It is a very good front door, not the whole house.
Free core covers frame extraction and transcription; the one-time $19 Pro is only worth it for detailed video teardowns.
**Free (core, MIT):** Scene-aware keyframe extraction, deduplication, Whisper transcription, and a local HTML viewer. Enough to feed any LLM a real look at a video. **Self-hosted:** It is local by default, ffmpeg plus Python on your machine, no cloud step. **Paid (Pro, one-time ~$19):** Adds camera-movement classification, editing-rhythm analysis, a perception timeline (gestures, emotion, sound events), and teardown reports. One-time founder price, not a subscription.
EdgeQuake is a GraphRAG engine: it turns a pile of documents into a knowledge graph and combines vector search with graph traversal so an AI can answer multi-hop questions, the kind where the answer is spread across several documents. It's a Rust reimplementation of the LightRAG approach with a React UI, Apache-licensed and free. Running it is a real project. You need Postgres 16 with the pgvector and Apache AGE extensions, plus Docker Compose to wire it together. It also needs an LLM and embedding provider: point it at OpenAI or Anthropic and you'll pay per call, or run it against a local Ollama model for no API cost at all. For a developer building retrieval over their own documents who specifically wants the graph-plus-vector approach, EdgeQuake is a solid, self-hostable option, mostly solo and small teams comfortable with Postgres. The Rust core is fast and the code is documented. The catch is honesty about the payoff. On accuracy it's roughly a tie with LightRAG, so you're not buying a quality leap, you're buying a Rust implementation and a UI. And the pgvector-plus-AGE Postgres setup is a genuine dependency lift. If you just need basic RAG, a simple vector store ships faster. Reach for this when the graph structure is the point.
Free and open (Apache 2.0). You supply an LLM and embedding provider: pay per call with OpenAI/Anthropic, or run it free against local Ollama.
**Free:** The entire engine and UI (Apache 2.0). **Self-hosted:** Postgres 16 with pgvector + Apache AGE, plus Docker Compose. Bring an LLM/embedding provider (Ollama for zero cost). **Paid:** None from the project. External API providers cost per call if you use them.
This is a maintained list of permanently free LLM API endpoints with API keys. Not trials, not 'free for 30 days,' but free-tier APIs you can actually build on. The list is organized by provider with rate limits, model availability, and key details. CC0 licensed, so you can do whatever you want with the information. The catch: 'permanently free' is a strong claim. Free tiers change. Rate limits tighten. Providers shut down. This is a living document that's only as good as its last update. And free LLM APIs often have significant rate limits. Fine for prototyping, not for production traffic. Always have a paid fallback for anything customer-facing.
Free. It's a reference list. The APIs listed have their own terms.
### Free Tier The list itself is free. It catalogs permanently free LLM API endpoints. ### Self-Hosted Not applicable. It's a curated list, not software. ### Paid None. This is a reference resource.
Mike is an open-source AI legal platform for document processing and review. Built on Next.js with an Express API, Supabase for auth and storage, S3-compatible object storage for files, and your choice of LLM provider for the analysis. AGPL-3.0. Setup is significant. You need a Supabase project, an S3-compatible bucket (Cloudflare R2 works), API keys for at least one model provider, and LibreOffice for DOC/DOCX-to-PDF conversion. The README is sparse, so plan for some reverse-engineering. Pick this if you run a small practice or a legal-ops team and want AI document review without sending client files to Harvey, Spellbook, or Casetext. Solo: free, you pay your model bill, the setup curve is real. Small teams: same plus shared infra. Large firms: they pay for Harvey because the integrations matter and they don't want to maintain this themselves. The catch: legal AI is a confidence and accuracy game, and Mike is early. Docs are thin, the feature set is whatever the README implies, and there is no SOC 2 posture out of the box. Use this because you want to own the stack, not because it's a turnkey product.
Open source. Real cost is your Supabase plan, R2 storage, and LLM usage.
**Free tier:** AGPL-3.0, no licensing cost. **Self-hosted:** Self-host the Next.js + Express stack. Supabase free tier covers small usage; Cloudflare R2 is cheap; LibreOffice is free. Real cost is the LLM provider bill plus engineering time to operate it. **Paid:** No paid tier. Cost equals your Supabase plan, R2 storage, and model API usage.
H3.c generates video with synchronized audio locally on a Mac. It's a from-scratch C inference engine for MiniMax's H3 model, written by the creator of Redis, with an interactive REPL that previews output right in your terminal. The code is MIT and a pleasure to read; local, private video generation with no cloud API is the pitch. The hardware is the gate. You want Apple Silicon with serious unified memory: peak usage runs into the tens of gigabytes depending on canvas size, plus about 37 GB of disk for the checkpoint. An SSD-streaming mode drops memory use to around 2 GB, but at a steep speed penalty at small canvas sizes. Build is a simple make with FFmpeg on PATH. This is for Mac owners with serious unified memory who want local generation, and for anyone who wants to read a modern diffusion pipeline in C. ComfyUI is the mainstream local workbench; Runway and Pika are the hosted alternatives. The catch: the MIT badge covers only the code. The model weights ship under MiniMax's custom community license, non-OSI, with an application process for US, EU, UK, and South Korea users, and the engine is useless without them. The author also frames the repo as working vertical slices, not a finished product, and canvas sizes and frame counts have sharp constraints.
Free MIT code, but the required model weights carry MiniMax's non-OSI community license, and the hardware bar is high.
**Free (the code):** MIT, no commercial offering, builds with make. **Not open (the weights):** MiniMax-H3 ships under a custom community license, not OSI-approved, with a separate application process for users in the US, EU, UK, and South Korea. Commercial evaluation means reading that agreement. **The real cost:** Hardware. Peak memory runs into the tens of gigabytes depending on canvas size (an SSD-streaming mode runs in ~2 GB at a steep speed cost), ~37 GB storage, and thermal headroom on laptops. **The trade:** No per-generation API fees, in exchange for owning serious silicon and accepting a vendor model license.
Text-generation-webui gives you a browser-based interface to do it. Load a model, chat with it, fine-tune it, generate images. It's the Swiss Army knife for local AI. The entire project is free under AGPL-3.0. Every feature (chat, notebook mode, model loading, LoRA training, multimodal/vision support, extensions) ships at $0. The developer sells some extension packs on Gumroad, but those are optional add-ons, not core features. Self-hosting is the only option, and the setup complexity depends on your GPU situation. If you have an NVIDIA card with 8GB+ VRAM, the one-click installers work well. AMD and Apple Silicon support exists but can be finicky. Expect 30-60 minutes for first-time setup including downloading a model. Solo developers: this is your playground. Run models locally, experiment with fine-tuning, keep your data private. Small teams: share a beefy GPU server running the API mode. Beyond that, look at dedicated inference servers like vLLM. The catch: GPU hardware requirements are real. You need a decent GPU to run anything useful. A 7B parameter model needs ~6GB VRAM. Anything bigger needs proportionally more. No GPU, no party.
Software is completely free. The real cost is GPU hardware: $300-900 for a capable NVIDIA card.
### Free Everything core is free. Chat interface, notebook mode, model management, LoRA/QLoRA fine-tuning, multimodal support, API server mode, extensions framework, all AGPL-3.0. ### Optional Paid Extensions The developer sells extension packs on Gumroad (e.g., "Deep Reason" for enhanced reasoning). These are optional convenience add-ons, not gated features. The core project doesn't depend on them. ### Self-Hosted (Only Option) This runs on your hardware. Minimum useful setup: NVIDIA GPU with 8GB VRAM (~$200-400 used for an RTX 3060 12GB). Ideal setup: 24GB VRAM card ($400-900 used for RTX 3090/4090). Running costs are electricity. Expect $5-15/month for regular use. ### Hardware Cost Reality Check - 7B models (Mistral, Llama 3): 6-8GB VRAM. Runs on an RTX 3060. - 13B models: 10-16GB VRAM. Needs RTX 3090 or better. - 70B models: 40GB+ VRAM. Multi-GPU territory or heavy quantization. - CPU-only: possible with llama.cpp backend but painfully slow for anything useful. ### Verdict The software is free. The hardware isn't. Budget $300-900 for a capable GPU if you don't have one already.
DaVinci-MagiHuman does it in one model. No separate video generation, no separate voice synthesis, no stitching. One 15-billion-parameter transformer takes text and a reference image and jointly produces video and audio. The numbers are real: 5-second 1080p video in 38 seconds on a single H100. Supports Mandarin, Cantonese, English, Japanese, Korean, German, and French. Beats Ovi 1.1 (80% win rate) and LTX 2.3 (60.9% win rate) in human evaluation. The full model stack is released: base model, distilled model, super-resolution model, and inference code. From Shanghai's GAIR Lab and Sand.ai. The catch: you need serious hardware. An H100 for the fast inference numbers, and the 15B parameter model isn't running on a consumer GPU. No license file listed; check before commercial use. And 'joint audio-video generation' is still early. The 5-second clip limit means this is for avatars and short-form content, not video production.
Free to use. You pay for GPU compute, and you'll need a lot of it.
Open source research release. No paid tier, no hosted version. You need your own GPU infrastructure: an H100 or equivalent for reasonable inference times. The model weights are on Hugging Face.
Kev answers typed questions about a block of text and returns probabilities your code can branch on. Route this ticket to billing or shipping. Is the customer angry. How urgent is this. No prose, one forward pass, one answer per question. Three sizes (0.8B, 4B, 9B) on Qwen3.5 bases, Apache-2.0, weights and training code included. The API copies TypeSafe's System One contract, so their Python SDK points at a local server with a URL change.
Install is uv sync and one serve command. The 4B wants about 9 GB of GPU memory in bf16, the 9B about 19 GB, and both fit a 32 GB Mac. Five questions take tens of milliseconds on an H100, roughly 780 ms for the 4B on an M5, two seconds for the 9B. On Apple Silicon the older Qwen3 builds are still the quicker option.
Solo developers and small teams doing ticket routing, moderation, or intent classification can run the 4B on hardware they already own. Be honest about why, though: hosted Jev costs $42 per billion input tokens, which is cheaper than any GPU you would rent to replace it. Run Kev for data residency, offline operation, or a fine-tune on your own labels. Not to save money.
The catch is what Kev does not know. On one 900-ticket support suite the 4B loses badly to hosted Jev at spotting an angry customer, and on MMLU-Pro it scores 0.490 against Jev's 0.840, because the base model sets that ceiling. The 9B closes the tone gap to roughly a tie and the knowledge gap not at all. Narrow, well-specified decisions are the job. The server also binds to localhost with no authentication.
Free and Apache-2.0 with the weights included. But hosted Jev publishes $42 per billion input tokens, so self-host for data residency or a custom fine-tune, not to save money.
**Free: everything.** Apache-2.0. All three checkpoints are on Hugging Face and mirrored as GitHub release tarballs with SHA-256 checksums. Training code, the frozen eval suites, the research log, and a local playground ship in the repo. No account, no API key, no rate limit. A hosted demo of the 4B and 0.8B runs on Hugging Face Spaces if you want to try it before installing anything. **Self-hosted: you pay in compute, not license.** Serving the 4B in bf16 needs roughly 9 GB of GPU memory, the 9B roughly 19 GB, and both run on a 32 GB Mac. Training your own adapter is cheaper than expected: 56 minutes on a single H100 for the 4B, 91 minutes for the 9B, and a 9-minute delta fine-tune for the 4B when you start from a released checkpoint. Budget engineering time for what the project leaves out: the server ships with no authentication and binds to localhost, so a proxy, keys, and deployment are on you. **Paid: hosted Jev, and it is cheap.** Kev has no commercial tier. The paid comparison is TypeSafe's hosted Jev, the model Kev is built to imitate, and TypeSafe puts a headline rate on its homepage: $42 per billion input tokens, marketed as 238x lower than Claude Fable 5.1. There is no pricing page and no tier table, so anything beyond that number means contacting sales. Do the arithmetic before you self-host. At that rate 100 million input tokens a month runs about $4, which no rented GPU is going to beat. Self-hosting Kev buys you data residency, offline operation, and a model you can fine-tune. It does not buy you a lower bill.
laya-mlx runs Laya decision models natively on Apple Silicon. Laya is an open-weights model from Convai Innovations that answers multiple choice, rating, and yes or no questions with probabilities instead of generated text. It is the open counterpart to TypeSafe AI's closed Jev API. This port drops PyTorch and runs on Apple's MLX, returning a short decision in roughly 7 to 14 milliseconds on an M3 Max by the author's own measurement. Apache 2.0. Setup is pip install laya-mlx on Python 3.11 or later and macOS 14 or later. Weights download from Hugging Face on first load, and peak memory for a short question stays under 1 GB. No cloud API and no account. It covers inference and conversion only. Training and fine-tuning stay in the upstream Laya project. Free at every size. Use it for local classification, routing, or guardrail checks on a Mac where a full LLM call is too slow. TheoLeeCJ/SemIf gets a similar interface out of general open models and also has an MLX backend. The catch: this is an unofficial port from a young pseudonymous account, at version 0.1.0 with five commits. Context tops out at 512 to 1,024 tokens. And an open issue reports that swapping the order of the options flipped the answer 27 to 33 percent of the time on a Chinese benchmark. Test option-order sensitivity before trusting it.
Completely free and open source. Runs locally on an Apple Silicon Mac.
**Free:** Everything. Apache 2.0 inference runtime and conversion tools. The Laya weights are a separate download from Hugging Face under upstream's Apache 2.0 license. **Self-hosted:** The only way it runs: locally on an Apple Silicon Mac with Python 3.11 or later and macOS 14 or later. Under 1 GB of memory for a short question. **Paid:** None. No cloud API, no account, no hosted version.
Big models want big hardware, which puts most of them out of reach on a laptop. Flash-moe makes that possible anyway. It uses a technique called Mixture of Experts (MoE) to run only the parts of the model that matter for each request, dramatically cutting the memory and compute needed. The pitch is simple: big model intelligence on small hardware. Models that normally need 32GB+ of VRAM can run on a laptop with 8-16GB of regular RAM. It's slower than running on a GPU, but it works. The catch: it is growing explosively and is still very early. The 'runs on a laptop' promise depends heavily on the model and your hardware. And MoE optimization is an active research area. Expect the approach to evolve fast.
Free. Check the license for commercial use.
Open source, no paid tier. You clone it and run it. The license isn't specified in the repo metadata. Check the repo directly before commercial use.