LLM Inference

ollama186.1k★

Ollama is the easiest way to run open models on your own machine: one install, then ollama run and a model name, and you're chatting locally with an OpenAI-compatible API on localhost. It runs on Mac, Windows and Linux, uses NVIDIA, AMD and Intel GPUs, and Metal on Apple Silicon. The app is MIT licensed and free. Local use needs no account. Under the hood it now reads standard GGUF files through llama.cpp, Vulkan support brings in AMD and Intel cards, and the next release makes Apple's MLX engine the default on Apple Silicon. NVIDIA cards need compute capability 5.0+ and a recent driver. Ollama Cloud, its hosted side, runs bigger open models you can't fit locally, from the same CLI. Cloud moved to per-token pricing with monthly credits on August 31, 2026. Free gets starter credits for starter models, one request at a time. Pro is $20 a month ($200 a year) with $60 of usage credits and 3 concurrent requests. Max is $100 a month with $300 of credits and 10 concurrent requests. Team is $500 a month in early access with $1,000 of shared credits. Credits don't roll over. Solo devs: local stays free. Pay only when you need models your hardware can't run. The catch: convenience costs control. Ollama picks quantizations and defaults for you, which is great until you need to tune something llama.cpp (ggml-org/llama.cpp) exposes directly. And now that it's venture-funded, watch how much of the roadmap tilts toward the cloud.

llama.cpp133.5k★

llama.cpp is the engine that runs large language models on ordinary hardware: a laptop CPU, an Apple Silicon Mac, a gaming GPU, even a phone. It loads GGUF model files, the format most local-AI tools now share, and sits underneath a lot of the apps people use to run models locally. MIT licensed, and the founding ggml team joined Hugging Face in February 2026. You can still build from source, but you don't have to: versioned releases started in August 2026, there's an installer at llama.app, a llama serve command, and a free desktop app for Mac and Windows. The built-in server speaks the OpenAI API and now Anthropic's Messages API too. Hardware support is wide, from CUDA, ROCm and Metal to Vulkan, SYCL and WebGPU. Free at every size. Solo devs who want a local model with no middleman get the most control here; Ollama (ollama/ollama) is the friendlier wrapper. Teams serving many users at once should look at vLLM (vllm-project/vllm), which is built for throughput. The catch: model size meets hardware fast. A 70B model at the common 4-bit quantization is a roughly 43 GB file, so it won't fit on a single 24 GB GPU and has to split between GPU and system RAM, which is slow. Expect to spend real time picking quantizations and tuning flags.

vLLM96.4k★

vLLM is the open source server teams use to serve language models to lots of users at once. Its PagedAttention memory management and continuous batching squeeze far more requests out of a GPU than a naive setup: the original benchmarks showed 14 to 24 times the throughput of plain Hugging Face Transformers and 2.2 to 3.5 times an optimized server like TGI. Apache 2.0, and a PyTorch Foundation project. It exposes an OpenAI-compatible API and runs on NVIDIA, AMD ROCm (with prebuilt wheels), Intel XPU, Google TPUs and Apple Silicon through a plugin. Getting the best out of it means matching CUDA versions, sizing GPU memory and tuning batch settings. A 70B model at full precision needs several data-center GPUs; quantized models shrink that. The software is free; your bill is GPUs. On-demand A100s run roughly $1.59 to $2.79 an hour on RunPod and Lambda, and an 8-GPU AWS p4d instance is about $22 an hour. Solo devs running one model for themselves should use Ollama (ollama/ollama) or llama.cpp instead. Teams serving real traffic: this is the default choice. The catch: it's a big, fast-moving codebase with a matching security record. 33 advisories landed between August and early October 2026, including remote code execution through request parameters, patched in 0.31.0. Pin versions, upgrade deliberately, and keep a gateway in front of it. Its creators also founded Inferact in January 2026, so watch for a commercial layer.

CLIProxyAPI57.8k★

CLIProxyAPI turns coding-agent subscriptions you already pay for (ChatGPT Codex, Claude Code, Antigravity, Grok Build, Devin, Kimi, Meta's Muse Code) into an OpenAI-, Gemini- and Claude-compatible API you can point other tools at. Log in once per account and it serves those quotas as an endpoint, with round-robin load balancing across multiple accounts. MIT licensed, a Go binary or Docker. Setup is a single binary or docker compose, a config file, and OAuth logins per provider. It stores those tokens on the proxy host and exposes a management API, so lock it down if it ever leaves localhost. Releases ship almost daily, and Gemini CLI support moved into a separate plugin. There's no paid tier; the cost is the subscriptions you proxy, and the risk is those accounts. Anthropic's terms say Claude plan credentials can't be routed through third-party apps, and Google's Antigravity terms call third-party access grounds for suspension. Google already ran a ban wave in February 2026 that also locked people out of Gemini CLI. Use the official APIs for anything in production. The catch: this exists to stretch consumer plans past their intended use, and the README's sponsor wall now advertises residential proxies with one IP per account for multi-account operations. That's account-ban territory, and you're the one holding the account.

LocalAI50.2k★

LocalAI is a self-hosted stand-in for the OpenAI API. It runs text, vision, speech, image and video models on your own hardware, CPU-only if that's all you have, and answers on OpenAI, Anthropic, Ollama and ElevenLabs-compatible endpoints, so most apps only need a new URL. MIT-licensed and free. One docker run or the macOS app gets it going. The core stays small: each engine (llama.cpp, vLLM, whisper.cpp, MLX and more) is pulled as a separate backend only when a model needs it, and models install from a gallery, Hugging Face or the Ollama registry. It's well past a hobby server now: API keys, OIDC logins, per-user quotas and a distributed mode that spreads models across GPU machines. There's no paid tier; hardware is the bill. Solo: free, though Ollama is simpler for one laptop. Small and larger teams: free, and the quotas and cluster mode make it a real internal AI gateway. The catch: out of the box there's no login. Unless you set an API key or turn on user auth, anyone who can reach the port can use it. A critical unauthenticated SSRF bug was patched in July. Set LOCALAI_API_KEY, stay current, and never expose it raw.

text-generation-webui47.7k★

Text-generation-webui gives you a browser-based interface to do it. Load a model, chat with it, fine-tune it, generate images. It's the Swiss Army knife for local AI. The entire project is free under AGPL-3.0. Every feature (chat, notebook mode, model loading, LoRA training, multimodal/vision support, extensions) ships at $0. The developer sells some extension packs on Gumroad, but those are optional add-ons, not core features. Self-hosting is the only option, and the setup complexity depends on your GPU situation. If you have an NVIDIA card with 8GB+ VRAM, the one-click installers work well. AMD and Apple Silicon support exists but can be finicky. Expect 30-60 minutes for first-time setup including downloading a model. Solo developers: this is your playground. Run models locally, experiment with fine-tuning, keep your data private. Small teams: share a beefy GPU server running the API mode. Beyond that, look at dedicated inference servers like vLLM. The catch: GPU hardware requirements are real. You need a decent GPU to run anything useful. A 7B parameter model needs ~6GB VRAM. Anything bigger needs proportionally more. No GPU, no party.

Strata40.8k★

Strata runs Qwen3.8-Flash-Next, a very large open-weight model, on a gaming PC. The model has 125B parameters but only 6B do work on each token, so Strata keeps the busiest parts on your graphics card and the rest in system RAM, where the CPU handles them. It serves an OpenAI- and Anthropic-compatible API on localhost. The engine is MIT and free; the model weights carry Qwen's own community license. RAM is the real requirement, not the GPU. You need an NVIDIA RTX 20 to 50 series card or a recent AMD Radeon with 12 GB of VRAM or more, at least 32 GB of system RAM (enough for the coding variant only; 64 GB fits every size), and about 80 GB of free disk. On the author's RTX 5070 with 64 GB of RAM, a short chat writes roughly 50 to 95 tokens per second depending on model size. By default it answers one request at a time; you can raise that, but each answer gets slower on a 12 GB card. Solo developers with the hardware: a way to run a frontier-sized open model locally at $0 per token. For general local models, llama.cpp (ggml-org/llama.cpp) and Ollama (ollama/ollama) are the mature picks. The catch: the project is under three weeks old, one author writes nearly all of it, and engine releases land every day or two. The installer downloads a prebuilt engine and checks it against the SHA-256 GitHub publishes, which catches a tampered download but not a bad release, and it installs anyway if it can't get a hash. Cautious users should build from source (--build). Also read the setup prompts: the "experimental speed projection" it offers is, by its own docs, a vector that strips the model's refusals, not a speed boost.

llmfit38.7k★

llmfit answers the question everyone running local AI asks first: will this model fit on my machine? It reads your CPU, RAM, and GPU, then scores every model in its catalog on memory fit, estimated speed, quality, and context length. No more downloading a huge model to find out it will not load. MIT licensed, free, one Rust binary. Install with Homebrew, Scoop, MacPorts, uv, Docker, or a curl script, on macOS, Linux, or Windows. It detects NVIDIA, Apple Silicon, AMD ROCm, and Intel GPUs, understands GGUF, AWQ, GPTQ, and EXL2 quantization, and works with Ollama, llama.cpp, MLX, Docker Model Runner, and LM Studio. No account, and the README says nothing leaves your machine unless you ask. There is also a web dashboard and a REST API through llmfit serve. Free at every size with no paid tier. Solo builders get the most out of it. Teams can run the API on shared GPU boxes to see what fits where. Once you have picked a model, ollama/ollama or ggml-org/llama.cpp does the actual running. The catch: speeds are estimates from a formula unless you run the benchmark against a live runtime, and the GPU bandwidth table only covers about 80 cards. Windows and Intel Macs get weaker detection. The model catalog is baked in at compile time, so new models only show up when you update.

sglang38.5k★

SGLang has become one of the default engines for serving open-weight LLMs in production at scale. It runs from a single GPU up to large clusters, with prefix caching, speculative decoding, and day-zero support for new model releases. The project says it runs on hundreds of thousands of GPUs, at xAI, NVIDIA and the major clouds among others. Apache-2.0, free. Install is a uv pip install, with builds for NVIDIA, AMD, Intel, Google TPUs and Ascend NPUs. Models span Llama, Qwen, DeepSeek, Kimi, GLM, Mistral, diffusion models, and most of Hugging Face. It is OpenAI-API compatible, so existing clients drop in without rewrites, and it doubles as a rollout backend for reinforcement learning training. Solo developers and small teams running open-weight models: this is one of the strongest options on the shelf, especially on DeepSeek or with heavy agentic workloads. Large teams running production inference at scale: you are very likely already evaluating it. The catch is that serious inference is still serious work. Cold starts, KV cache tuning, and multi-node setups need real engineering. SGLang gives you a faster engine, not a managed platform. RadixArk, a venture-funded company among its maintainers, is building one on top, with no public pricing yet.

DeepInfer25.5k★

DeepInfer is the inference engine behind DeepSeek's own production API, now open source under Apache-2.0. That's the software that loads a huge model onto GPUs and turns prompts into answers fast and cheap. It sits in DeepSeek's official GitHub org next to FlashMLA and DeepGEMM, so it's the real thing, and it's free. Free to download is not the same as runnable. It supports one model today, DeepSeek-V4.1-Flash, on two kinds of hardware: NVIDIA SM100 GPUs (Blackwell datacenter parts like the B200) and Huawei Ascend 950 NPUs. No H100s, no gaming cards. You build from source on Linux with CUDA 13, examples assume an 8-GPU node, and DeepSeek's own guide puts the smallest efficient production unit at 128 GPUs. There's no paid tier. The bill is the hardware. Solo and small teams: skip it and use DeepSeek's API, vLLM or SGLang. Larger teams serving DeepSeek models on Blackwell or Ascend: benchmark it against vLLM. The catch: this release is offline inference only. The README says a KV cache system, prefill/decode disaggregation and load balancing are not included, which is most of what turns an engine into a service. You get DeepSeek's engine, not DeepSeek's platform.

ds424.2k★

DwarfStar (the repo is still named ds4) is antirez's local inference engine for running a short list of big open models on hardware you own. The creator of Redis built it first for DeepSeek V4 Flash, and it now also runs DeepSeek V4.1 Flash, V4 PRO, the GLM 5.x family, and Qwen3.8 Flash Next on Metal, behind an OpenAI- and Anthropic-compatible server. It only loads the GGUF files the project itself produces. MIT, free. The price is hardware, not software. Macs with 96 GB or more are the primary target, and smaller machines can stream weights off SSD. On Linux it runs on CUDA, including older Ada Lovelace cards like the L40S, plus DGX Spark and Strix Halo boxes. Setup is a download script and make. Weight files run to hundreds of gigabytes, so budget a big, fast SSD too. Beyond the chat CLI it ships ds4-agent, a built-in coding agent, plus image input, disk KV caching and inference split across machines. A solo developer with a loaded Mac Studio gets a near-frontier model with nobody else in the loop. Everyone else should stay on llama.cpp or vLLM until they own the metal. The catch is churn. antirez calls it beta quality, warns that regressions are possible and that models get dropped when better ones arrive, and says openly it was built with strong help from AI coding agents. Run it to own your inference stack, not to ship a product on it.

omlx23.4k★

oMLX runs large language models on your Mac and manages them from the menu bar. You get a local API that speaks both the OpenAI and Anthropic formats, so Claude Code, Codex or Cursor can point at your own machine instead of a paid cloud model. Apache 2.0, no account, no paid tier. The reason to pick it over Ollama is the cache. Coding agents keep resending the same long prompt, and oMLX saves that work to your SSD, so a returning context loads from disk instead of being recomputed, even after a restart. The signed DMG auto-updates and ships the fast native kernels prebuilt. Solo developers running a local coding assistant get the most from it. Small teams can share one big Mac over the API; it refuses network binds without an API key. Larger teams belong on vLLM and real GPUs. The catch: Apple Silicon, macOS 15 or newer, and the project calls 16GB of RAM the minimum with 64GB recommended. Homebrew and source installs without full Xcode skip the native kernels, and GLM-5.2 prefill then runs roughly 30x slower. Take the DMG. It's also one person's after-hours project.

TensorRT-LLM15.0k★

TensorRT-LLM squeezes maximum inference performance out of NVIDIA GPUs for large language models. It handles quantization (FP8, FP4, INT4), custom attention kernels, paged KV caching, and multi-GPU deployment through a Python API. If you are serving LLMs at scale on NVIDIA hardware, this is the optimization layer that makes the economics work. Running it yourself means you need NVIDIA GPUs, full stop. No AMD, no Apple Silicon, no CPU fallback. You will also need CUDA installed and compatible driver versions. The setup is not trivial, but NVIDIA provides containers and Docker images that smooth out the worst of it. Once running, the performance gains over naive PyTorch inference are substantial, often 2-4x throughput improvements. For teams already committed to NVIDIA hardware, TensorRT-LLM is the right call over vLLM when you need every last token per second. vLLM is easier to set up and supports more hardware. llama.cpp is better for local, single-GPU experimentation. TensorRT-LLM is for production serving where GPU cost is a real line item. The catch: you are locked to NVIDIA forever. The library only works on their GPUs, and if your cloud costs push you toward AMD or custom silicon, you are rewriting your inference stack from scratch.

awesome-free-llm-apis12.4k★

awesome-free-llm-apis is a curated list of LLM APIs with ongoing free tiers, not one-off trial credits: Gemini, Groq, Mistral, OpenRouter, Cloudflare Workers AI, NVIDIA NIM and ten more. Each entry gives the base URL, free models, context sizes, rate limits and a link to the API key page. It's CC0. Nothing to install. It's a README maintained by Manifest, an LLM routing company that plugs its own product up top. The footnotes beat most lists: who may train on free-tier prompts, who demands real-name ID checks, which models are being retired. Our spot check of Groq, OpenRouter and Cloudflare limits matched their own docs. Solo builders and students: start here for a free prototype. Small teams: use it to pick a fallback, then pay. Larger teams: OpenRouter's free models stop at 50 requests a day until you buy $10 in credits, which tells you what these tiers are for. The catch: "permanent" is the list's word, not the providers'. Google and Mistral no longer publish numeric free limits, and some rows were never live-tested. And on October 10 someone pushed a credential-stealing workflow to this repo; it was removed the next day. Check key links before pasting a key.

kimi-k3-in-c10.8k★

kimi-k3-in-c runs a 2.78-trillion-parameter model on a machine with 8 GB of RAM, and the claim holds up. It's a from-scratch inference engine in portable C99, no BLAS, no GPU, no framework. Kimi K3 is a mixture-of-experts model where only a handful of experts fire per token, so the engine streams weights from disk and caches the few it needs. Apache 2.0, free. The headline number is real but it's not the one that matters. You need roughly 1.7 TB of fast NVMe for the checkpoint, Linux on x86-64 with AVX2, and patience: generation runs seconds to tens of seconds per token depending on hardware and memory budget. More RAM buys real speed, but nothing close to interactive. Output is byte-identical at every memory budget, validated against a PyTorch reference, which is the kind of engineering discipline most projects skip. Use this to learn how MoE inference actually works, to benchmark against, or to prove a point. Don't use it to serve anything: Moonshot's hosted API is faster and cheaper for real workloads, and llama.cpp is the right tool for models that fit your hardware. The catch: it's a weeks-old, essentially solo reference implementation, and the model weights carry Moonshot's own license, not Apache. Treat it as a brilliant teaching artifact, not infrastructure.

rikkahub8.7k★

RikkaHub is a native Android app for chatting with any LLM through one interface. Instead of separate apps for ChatGPT, Claude and Gemini, you point it at any OpenAI, Anthropic or Google-compatible endpoint, including your own, and switch providers between conversations. It handles markdown, image, PDF and document input, web search, message branching, memory and MCP. AGPL-3.0, free. Install it from Google Play or the project site, add your API keys and go. There is nothing to self-host: the app talks directly to the providers you configure. Newer builds add a Workspace, a lightweight Linux environment built on proot that runs on the phone without root so agents can gather information and assemble documents, plus a web interface you can open in a browser. For anyone on Android juggling multiple AI providers, this is the clean way to do it. You pay each provider for tokens, or point it at a local model endpoint and pay nothing. It is a personal app, and there is no team tier. The catch is trust in the chain. A multi-provider client is only as private as the endpoints you configure, and the README itself warns that many forks exist, some leaking privacy or asking for excessive permissions. Install the official build and point it at providers you trust.

mlx-lm7.6k★

mlx-lm runs and fine-tunes large language models directly on a Mac. Point it at a model on Hugging Face and one command pulls it down and runs it locally, using Apple's own MLX engine instead of a cloud API or a separate GPU rig. MIT licensed, free, and built by Apple's own ml-explore team, the same group behind MLX itself. It does more than run models. You can quantize them down to 4-bit, fine-tune with LoRA or full-model training, serve with streaming and prompt caching, and even split work across multiple machines. Setup is close to trivial: pip install mlx-lm, then a single command chats with a model. The real constraint is memory. MLX uses the Mac's unified memory, so the model has to roughly fit in RAM, and pushing past that needs macOS 15 or newer plus some system tuning. And it's Apple Silicon only. No M-series chip, no mlx-lm. The honest framing on competition: this is a building block, not a finished app. llama.cpp is the closest peer and runs on more hardware; Ollama and LM Studio are more packaged and app-like, and increasingly use MLX under the hood anyway; vLLM is for datacenter GPUs, a different world. mlx-lm's edge is being the MLX-native option, which means the best raw performance on a Mac and the cleanest fine-tuning story. Solo developers and researchers on Apple Silicon: this is the fast path. Small teams can build on it; larger production serving will want something server-side. The catch is that you're trading convenience and reach for Mac-native speed. It's lower-level than Ollama, locked to Apple hardware, and capped by how much RAM you bought. Within those lines, nothing runs models on a Mac better.

laya-mlx7.1k★

laya-mlx runs Laya decision models natively on Apple Silicon. Laya is an open-weights model from Convai Innovations that answers multiple choice, rating, and yes or no questions with probabilities instead of generated text. It is the open counterpart to TypeSafe AI's closed Jev API. This port drops PyTorch and runs on Apple's MLX, returning a short decision in roughly 7 to 14 milliseconds on an M3 Max by the author's own measurement. Apache 2.0. Setup is pip install laya-mlx on Python 3.11 or later and macOS 14 or later. Weights download from Hugging Face on first load, and peak memory for a short question stays under 1 GB. No cloud API and no account. It covers inference and conversion only. Training and fine-tuning stay in the upstream Laya project. Free at every size. Use it for local classification, routing, or guardrail checks on a Mac where a full LLM call is too slow. TheoLeeCJ/SemIf gets a similar interface out of general open models and also has an MLX backend. The catch: this is an unofficial port from a young pseudonymous account, at version 0.1.0 with five commits. Context tops out at 512 to 1,024 tokens. And an open issue reports that swapping the order of the options flipped the answer 27 to 33 percent of the time on a Chinese benchmark. Test option-order sensitivity before trusting it.

LiteRT-LM6.8k★

LiteRT-LM runs language models directly on a device, no cloud and no internet required. The model lives on the phone, laptop, smartwatch, or even in the browser, so data never leaves the hardware, it works offline, and there's no per-query bill. This is Google's own framework, and Google uses it to power on-device AI in Chrome, Chromebook Plus, and the Pixel Watch. Apache 2.0, completely free. It's cross-platform by design, targeting Android, iOS, desktop, the web via WebGPU, and small boards like Raspberry Pi, and it taps GPU and NPU acceleration instead of grinding on the CPU. It runs open models like Gemma, Llama, Phi, and Qwen. The work isn't running a server, because there is no server. The work is on the build side: you obtain and convert models into the right format, then wire up the native SDK for each platform you ship to, and manage on-device memory per device class. Heavier than calling a cloud API, far lighter than operating an inference cluster. The real competition is other on-device runtimes. llama.cpp has broader model coverage and a bigger community; Meta's ExecuTorch is the closest vendor-backed rival; Apple's MLX wins on Macs but only on Macs. LiteRT-LM's edge is tight, official integration with Android and Google silicon. It doesn't replace a paid product so much as move certain workloads off the paid-API meter: the small and mid-size models you'd otherwise rent from a cloud. Solo and small teams shipping mobile or edge apps: this is the Google-blessed path. Larger teams already on Android get first-party support. The catch is maturity. The core runtime is production-ready and shipping in real Google products, but some bindings, Swift and JavaScript among them, are still early preview, and the project is young. And on-device models are not frontier models. If you need GPT-class quality, this isn't that. It's for when private, offline, free, and good-enough beats cloud-quality.

flash-moe4.5k★

Flash-moe runs Qwen3.5-397B, a 397-billion-parameter model, on a MacBook Pro with 48GB of RAM. It's a Mixture of Experts model, so only a few expert slices fire per token, and flash-moe streams those slices off the SSD instead of holding all 209GB in memory. Pure C, Objective-C and hand-written Metal shaders. Free to read, with a big license caveat. This is a research demo, not a product. The author measured about 4.4 tokens per second at 4-bit on an M3 Max with tool calling working; the faster 2-bit mode breaks JSON output. Apple Silicon only. You download the weights, repack the experts with the included Python scripts and build with make; one user's issue describes getting it working "with a lot of pain." No paid tier. Solo tinkerers with a big Mac: worth an afternoon and the paper. Teams: no. For real local inference, use llama.cpp, Ollama or MLX. The catch: there is no LICENSE file, so legally it's all rights reserved and you can't ship anything built on it. A request to add one has sat open since March. The last commit was March 19, 2026, and community ports, including a CUDA version, sit unmerged.

Rapid-MLX4.1k★

Rapid-MLX runs open models locally on Apple Silicon, and speed is the whole pitch. One command installs it (Homebrew, pip, or a curl script), one command serves an OpenAI-compatible API, and it also speaks the Anthropic messages format, so Claude Code, Codex CLI, and Aider can point at it as a drop-in backend. Apache 2.0, completely free. The engineering focus is on the things that make local agent workflows fail in practice. A prompt cache brings claimed time-to-first-token under a tenth of a second on repeat prompts, and tool calling is handled across 17 parser formats, which matters because local models break agent loops on malformed tool calls far more often than on reasoning. It's in Homebrew core and publishes to PyPI, with releases landing near daily. Mac-only by design. If you run agents against local models on an M-series machine, try it head-to-head with Ollama or LM Studio; the performance claims ship with benchmarks in the repo rather than vibes. On Linux or Windows, look at Ollama or vLLM instead. The catch: it's young and moves extremely fast, the headline numbers are the project's own, and independent benchmarks are still thin. And no inference engine fixes the ceiling: a fast small model is still a small model.

tokenspeed2.9k★

TokenSpeed is an LLM inference engine: the server that loads an open-weight model and answers requests through an OpenAI-compatible API, the job vLLM, SGLang and TensorRT-LLM do. It's tuned for agentic workloads, MIT licensed, free, and run by the non-profit LightSeek Foundation with engineering help from NVIDIA and AMD. This is datacenter software. Its fastest paths target NVIDIA Blackwell (B200, GB200, GB300), with recipes for Hopper, AMD MI350X and MI355X, plus early Radeon RDNA4 and Ascend NPU support. Install pip wheels (CUDA 12.9, CUDA 13, ROCm 7.2) or the Docker image, then run tokenspeed serve. Recipes cover Kimi K3, Qwen3.8, GLM 5.3, DeepSeek V4, GPT-OSS and MiniMax M3, several on launch day. No paid tier; the GPUs are the bill. Solo: free, but on one consumer card Ollama or llama.cpp is the saner pick. Small and mid-size teams serving open models: bake it off against vLLM and SGLang on your model. Large teams serving big MoE models on Blackwell or MI355X: worth a real benchmark. The catch: it's v0.1.1, and the headline chart (beating TensorRT-LLM on Kimi K2.5) is the team's own. The docs call the recipes templates to validate, and the getting-started container runs privileged on the host network.

WARP2.5k★

WARP (formerly WASTE) runs AI models far bigger than your computer's memory. It keeps the always-used core of a mixture-of-experts model in RAM and streams the rest off your SSD per token. The headline: the full 2.78-trillion-parameter Kimi K3 on a 64 GB MacBook Pro. It's a dependency-free C engine from the SQLite Cloud team, free under Apache 2.0. Kimi K3 is the stunt. The useful part is the mid-size models: GLM-5.3-Flash runs at about 3.9 tokens per second on that same MacBook and still works on a 16 GB machine, and DeepSeek-V4.1-Flash manages about 3.8. You get a CLI, an embeddable C library and an optional OpenAI-compatible server. Setup is real work: you download and convert the weights yourself. No paid tier. Solo tinkerers: free, start with the small Kimi-Linear. Small teams: free for offline experiments. Larger teams: this is not a serving stack, use vLLM or llama.cpp on GPUs. The catch is speed and disk. K3 runs under one token per second and needs about 1 TB of NVMe, and prefill is as slow as decode, so long prompts take minutes. The format and API aren't frozen, and the README says LLMs write the code while humans steer.

laya-coreml1.6k★

laya-coreml moves Laya decision models onto Apple's Neural Engine, the low-power AI chip in every Apple Silicon Mac. Laya answers multiple choice, rating, and yes or no questions with probabilities instead of generated text, the open counterpart to TypeSafe's Jev. This is the same author's follow-up to mizorewww/laya-mlx, and it is Apache-2.0. Install is pip install laya-coreml on macOS 15 or later. Weights download from Hugging Face on first load, then everything runs offline, with no PyTorch needed. The author measured about 5 ms per short question on an M3 Max, roughly 1.4x faster than the MLX port and about 2.8x less energy per decision. Free at every size. Use it when a Mac app needs a fast yes or no or a routing decision thousands of times a session and battery matters. For longer prompts, laya-mlx is simpler. The catch is the fine print. The Neural Engine build caps each request at 96 tokens total, question and options included. An experimental 1,024-token Neural Engine build took about 92 ms per request, and the author says the short-prompt speedup does not carry over to long prompts. The weights sit on a pseudonymous Hugging Face account, and this is a port of a port, a few days old.

MoonEP1.2k★

MoonEP is a low-level building block for running Mixture-of-Experts AI models across many GPUs. Its whole job is keeping the token load balanced across GPU ranks, using dynamic redundant experts and fused kernels, so no single GPU becomes the bottleneck during MoE inference or training. It comes from Moonshot AI, MIT-licensed and free. This is infrastructure, not an app. There's nothing to click. You need a datacenter-grade setup: a multi-GPU NVIDIA cluster with NVLink, since it relies on symmetric memory and remote reads between GPUs. It was tested on eight H20s. If that sentence didn't apply to you, this isn't your tool. For the specific teams training or serving large MoE models on GPU clusters, MoonEP is a serious performance primitive that competes with libraries like DeepEP. Everyone else can skip it. There's no team-size question here, it's for infrastructure engineers at labs and companies running big models. The catch is reach and maturity. It's NVIDIA-only today, with other hardware listed as coming soon, and it's very new, benchmarked against DeepEP but not yet widely battle-tested. It's a sharp tool for a narrow, expensive job, not something you casually adopt.

backburner1.2k★

Backburner turns your iPhone into a helper GPU for your Mac. Plug a recent Pro iPhone into an Apple Silicon Mac over a 10 Gb/s USB-C cable and it helps run Qwen3.8-27B locally: the phone runs the last 24 layers while the Mac reads your prompt, 29-44% faster at 16k-48k of context in the maintainer's tests. It also holds the oldest context past 64k. Free, MIT. Setup is real work: a 24 GB Apple Silicon Mac, an iPhone 15 Pro or newer, Xcode, and the app sideloaded through AltStore with a free Apple ID that re-signs every 7 days. One curl | bash pulls about 24 GB of models, then backburner serves an OpenAI-compatible API on localhost. Its llama.cpp fork alone more than doubles writing speed over stock. Solo developers running local coding agents: free, worth an afternoon. Teams: skip it, it handles one request at a time on one desk. The catch: it's pre-release, and the phone must stay unlocked with the app in front. Past 64k, a locked screen stops the server. Below 64k the phone doesn't speed up writing at all. Versions before 0.0.3 answered anyone on Wi-Fi with no auth, so update.

deltafin857★

Deltafin runs Moonshot's Kimi K3, a massive trillion-parameter model, locally on a single machine and puts an OpenAI-compatible API in front of it. That means your local chat or coding agent can talk to a frontier-scale model running entirely on your own hardware, no API keys, no data leaving the box. It's MIT-licensed and free. The honesty starts with the requirements. You need at least 64GB of RAM and about 1.7TB of storage for the full model, or roughly 215GB in streaming mode. It'll use Metal or CUDA if you have it and fall back to CPU if you don't. This is a heavy lift, firmly in enthusiast-with-a-workstation territory. For a hobbyist who wants to run a giant model fully private and doesn't mind the setup, it's a fascinating way to do it for free instead of paying for hosted API calls. It's a proof of concept, not a team server, so treat it as an experiment rather than infrastructure. The catch is speed and maturity, and the README is refreshingly blunt about it: set your client timeouts to hours, not seconds. It does greedy decoding only, handles one request at a time, and re-processes the full context every turn. It's an impressive hack, not a practical replacement for a hosted API when you need answers now.

nativ0★

nativ runs AI models locally on your Mac, built for Apple Silicon specifically using MLX, Apple's own ML framework. Chat with local models, do vision and image tasks, manage what you've downloaded, and expose an OpenAI or Anthropic-compatible endpoint so your coding tools can hit a model running on your own machine. MIT, free, no account. Setup is about as easy as local AI gets: download the DMG, drag it to Applications, launch. It auto-updates. The only real gate is hardware. You need an Apple Silicon Mac and macOS 26, which is a very new OS to require, plus enough unified memory for whatever model you load. For privacy-sensitive work, or just dodging API bills, running models locally is the move, and on a modern Mac nativ makes it painless, solo or small team, at no cost. It competes with LM Studio and Ollama front-ends; the difference is it's MLX-native and Mac-only by design. The catch: that Mac-only, macOS 26 requirement locks out everyone else, and a couple of the headline features (audio models, image generation as a chat tool) are still "coming soon" rather than shipped. Solid today, with asterisks.