flash-moe

Flash-moe runs Qwen3.5-397B, a 397-billion-parameter model, on a MacBook Pro with 48GB of RAM. It's a Mixture of Experts model, so only a few expert slices fire per token, and flash-moe streams those slices off the SSD instead of holding all 209GB in memory. Pure C, Objective-C and hand-written Metal shaders. Free to read, with a big license caveat.

This is a research demo, not a product. The author measured about 4.4 tokens per second at 4-bit on an M3 Max with tool calling working; the faster 2-bit mode breaks JSON output. Apple Silicon only. You download the weights, repack the experts with the included Python scripts and build with make; one user's issue describes getting it working "with a lot of pain." No paid tier. Solo tinkerers with a big Mac: worth an afternoon and the paper. Teams: no. For real local inference, use llama.cpp, Ollama or MLX.

The catch: there is no LICENSE file, so legally it's all rights reserved and you can't ship anything built on it. A request to add one has sat open since March. The last commit was March 19, 2026, and community ports, including a CUDA version, sit unmerged.