
GPTQModel
LLM model quantization (compression) toolkit with HW acceleration support for Nvidia CUDA, AMD ROCm, Intel XPU, Tenstorrent, Huawei Ascent NPU, and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.
The Lens
By Erik Loyd, SaaS CEO and former COO/CFO of an AWS Premier Partner.
Updated Nov 2026
GPTQModel shrinks large AI models so they fit on smaller, cheaper hardware. The trick is quantization (storing weights in 4 bits instead of 16, trading a little accuracy for a lot of memory), and this one Python package covers GPTQ, AWQ, GGUF, FP8, EXL3 and newer methods. Apache-2.0, free from pip.
It runs on NVIDIA, AMD ROCm, Intel XPU, Apple Silicon through MLX, Huawei Ascend and Tenstorrent. Kernels compile on first use, so install the matching CUDA or ROCm toolkit, and calibrating a big model takes serious GPU memory. Output loads in Hugging Face Transformers, vllm-project/vllm and sgl-project/sglang: compress here, serve there.
This is the toolkit for quantizing a model yourself instead of downloading someone else's copy. Solo: free, though most people only need the pre-quantized checkpoints already on Hugging Face. Small teams: free; use its built-in evaluation against the full model before shipping. Larger teams: free, and sponsor it if you depend on it.
The catch: several releases a month, and it is not purely Apache. The optional Swordfish kernel it loads at runtime is AGPL-3.0, and a September 2026 out-of-bounds read CVE was fixed in 7.3.0. Stay current and know which backends you ship.
Free vs Self-Hosted vs Paid
fully freeFree: The whole library, from pip or uv, under Apache-2.0. Every quantization method, the evaluation tools and the inference backends are included.
Self-hosted: It runs wherever your Python and GPUs are: Linux with NVIDIA, AMD, Intel XPU, Huawei Ascend or Tenstorrent, macOS on Apple Silicon, and Windows 11 with NVIDIA or CPU. Your cost is compute time for calibration and the GPU memory a large model needs while it is being quantized.
Paid: None. ModelCloud asks for sponsorship (compute, API credits or money) through the maintainer's X account; there is no paid edition or support plan.
Free with no upsell. Your real spend is the GPU time to calibrate and evaluate a model, which is still far cheaper than running it unquantized.
What to do by team size
- Solo
- free, or just download pre-quantized checkpoints
- Small team
- free, evaluate quality against the full model before shipping
- Medium team
- free, pin releases given the fast cadence
- Large team
- free, check the AGPL Swordfish kernel against your distribution policy
Get tools like this every Wednesday
One featured tool, three on the radar. No fluff.
A low score is not a verdict on quality. Young and niche tools start low by design. How we calculate scores
Trust Signals
License: Apache License 2.0
Apache-2.0, but the optional JIT-loaded Swordfish kernel is AGPL-3.0-or-later.
Commercial use: ✓ Yes
About
- Owner
- ModelCloud.ai (Organization)
- Stars
- 1,290
- Forks
- 216
Explore Further
More tools in the directory
llmfit
Hundreds of models & providers. One command to find what runs on your hardware.
39.0k ★sglang
SGLang is a high-performance serving framework for large language models and multimodal models.
38.9k ★filament
Filament is a real-time physically based rendering engine for Android, iOS, Windows, Linux, macOS, and WebGL2
20.8k ★