tokenspeed

TokenSpeed is an LLM inference engine: the server that loads an open-weight model and answers requests through an OpenAI-compatible API, the job vLLM, SGLang and TensorRT-LLM do. It's tuned for agentic workloads, MIT licensed, free, and run by the non-profit LightSeek Foundation with engineering help from NVIDIA and AMD.

This is datacenter software. Its fastest paths target NVIDIA Blackwell (B200, GB200, GB300), with recipes for Hopper, AMD MI350X and MI355X, plus early Radeon RDNA4 and Ascend NPU support. Install pip wheels (CUDA 12.9, CUDA 13, ROCm 7.2) or the Docker image, then run tokenspeed serve. Recipes cover Kimi K3, Qwen3.8, GLM 5.3, DeepSeek V4, GPT-OSS and MiniMax M3, several on launch day.

No paid tier; the GPUs are the bill. Solo: free, but on one consumer card Ollama or llama.cpp is the saner pick. Small and mid-size teams serving open models: bake it off against vLLM and SGLang on your model. Large teams serving big MoE models on Blackwell or MI355X: worth a real benchmark.

The catch: it's v0.1.1, and the headline chart (beating TensorRT-LLM on Kimi K2.5) is the team's own. The docs call the recipes templates to validate, and the getting-started container runs privileged on the host network.