TensorRT-LLM

TensorRT-LLM squeezes maximum inference performance out of NVIDIA GPUs for large language models. It handles quantization (FP8, FP4, INT4), custom attention kernels, paged KV caching, and multi-GPU deployment through a Python API. If you are serving LLMs at scale on NVIDIA hardware, this is the optimization layer that makes the economics work.

Running it yourself means you need NVIDIA GPUs, full stop. No AMD, no Apple Silicon, no CPU fallback. You will also need CUDA installed and compatible driver versions. The setup is not trivial, but NVIDIA provides containers and Docker images that smooth out the worst of it. Once running, the performance gains over naive PyTorch inference are substantial, often 2-4x throughput improvements.

For teams already committed to NVIDIA hardware, TensorRT-LLM is the right call over vLLM when you need every last token per second. vLLM is easier to set up and supports more hardware. llama.cpp is better for local, single-GPU experimentation. TensorRT-LLM is for production serving where GPU cost is a real line item.

The catch: you are locked to NVIDIA forever. The library only works on their GPUs, and if your cloud costs push you toward AMD or custom silicon, you are rewriting your inference stack from scratch.