DeepInfer
DeepInfer: High-performance, large-scale inference framework for DeepSeek models
The Lens
By Erik Loyd, SaaS CEO and former COO/CFO of an AWS Premier Partner.
Updated Oct 2026
DeepInfer is the inference engine behind DeepSeek's own production API, now open source under Apache-2.0. That's the software that loads a huge model onto GPUs and turns prompts into answers fast and cheap. It sits in DeepSeek's official GitHub org next to FlashMLA and DeepGEMM, so it's the real thing, and it's free.
Free to download is not the same as runnable. It supports one model today, DeepSeek-V4.1-Flash, on two kinds of hardware: NVIDIA SM100 GPUs (Blackwell datacenter parts like the B200) and Huawei Ascend 950 NPUs. No H100s, no gaming cards. You build from source on Linux with CUDA 13, examples assume an 8-GPU node, and DeepSeek's own guide puts the smallest efficient production unit at 128 GPUs.
There's no paid tier. The bill is the hardware. Solo and small teams: skip it and use DeepSeek's API, vLLM or SGLang. Larger teams serving DeepSeek models on Blackwell or Ascend: benchmark it against vLLM.
The catch: this release is offline inference only. The README says a KV cache system, prefill/decode disaggregation and load balancing are not included, which is most of what turns an engine into a service. You get DeepSeek's engine, not DeepSeek's platform.
Free vs Self-Hosted vs Paid
fully freeFree: All of it. DeepInfer is Apache-2.0, and the default checkpoint it downloads (DeepSeek-V4.1-Flash-1017 on Hugging Face) is MIT-licensed. Included: optimized kernels, expert and data parallelism, DSpark speculative decoding, batch-invariant and deterministic inference (NVIDIA only) and structured outputs (regex, JSON schema, EBNF).
Self-hosted: Linux, GCC 13+, Python 3.12, and CUDA 13 (NVIDIA) or CANN 9.2 (Ascend). Supported hardware is NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs only. Build from source with ./build.sh; examples run with --ep 8. DeepSeek's published benchmarks used 4 SM100 GPUs for prefill and 64 for decode, and its deployment Q&A calls 64 prefill plus 64 decode GPUs the smallest efficient unit. Not included: a KV cache system, prefill/decode disaggregation, and load balancing across replicas.
Paid: No paid tier. The real cost is Blackwell-class or Ascend 950 hardware, owned or rented. For most teams DeepSeek's hosted API, or vLLM/SGLang on hardware you already have, is the practical route.
Free software that only runs on NVIDIA SM100 (Blackwell datacenter) GPUs or Huawei Ascend 950s, and DeepSeek sizes an efficient deployment at 128 GPUs. Almost everyone is better off with DeepSeek's API or vLLM.
What to do by team size
- Solo
- skip; needs datacenter Blackwell or Ascend 950 hardware
- Small team
- skip; use DeepSeek's API, vLLM or SGLang
- Medium team
- free, but only if you already run SM100 GPUs and can build the missing serving pieces
- Large team
- free; benchmark it against vLLM and SGLang if you serve DeepSeek models on Blackwell or Ascend
Get tools like this every Wednesday
One featured tool, three on the radar. No fluff.
A low score is not a verdict on quality. Young and niche tools start low by design. How we calculate scores
Trust Signals
About
- Owner
- DeepSeek (Organization)
- Stars
- 20,285
- Forks
- 2,183
Also by DeepSeek
An internal component used by DeepSeek Harness
A high-performance communication library for machine learning training and inference on Huawei Ascen
DeepGEMM-Ascend: clean and efficient matrix multiplication kernel library for Huawei Ascend NPUs
Explore Further
More tools in the directory
openclaw
Your own personal AI assistant. Any OS. Any Platform. The lobster way. 🦞
392.8k ★everything-claude-code
The agent harness performance optimization system. Skills, instincts, memory, security, and research-first development for Claude Code, Codex, Opencode, Cursor and beyond.
286.7k ★DeepSeek Harness
DeepSeek Harness: Everything is a Plugin.
258.6k ★