Tools/ModelCloud/GPTQModel

GPTQModel

LLM model quantization (compression) toolkit with HW acceleration support for Nvidia CUDA, AMD ROCm, Intel XPU, Tenstorrent, Huawei Ascent NPU, and Intel/AMD/Apple CPU via HF, vLLM, and SGLang.

1.3k ★emergingPythonApache License 2.0 →trending

The Lens

By Erik Loyd, SaaS CEO and former COO/CFO of an AWS Premier Partner.

Updated Nov 2026

GPTQModel shrinks large AI models so they fit on smaller, cheaper hardware. The trick is quantization (storing weights in 4 bits instead of 16, trading a little accuracy for a lot of memory), and this one Python package covers GPTQ, AWQ, GGUF, FP8, EXL3 and newer methods. Apache-2.0, free from pip.

It runs on NVIDIA, AMD ROCm, Intel XPU, Apple Silicon through MLX, Huawei Ascend and Tenstorrent. Kernels compile on first use, so install the matching CUDA or ROCm toolkit, and calibrating a big model takes serious GPU memory. Output loads in Hugging Face Transformers, vllm-project/vllm and sgl-project/sglang: compress here, serve there.

This is the toolkit for quantizing a model yourself instead of downloading someone else's copy. Solo: free, though most people only need the pre-quantized checkpoints already on Hugging Face. Small teams: free; use its built-in evaluation against the full model before shipping. Larger teams: free, and sponsor it if you depend on it.

The catch: several releases a month, and it is not purely Apache. The optional Swordfish kernel it loads at runtime is AGPL-3.0, and a September 2026 out-of-bounds read CVE was fixed in 7.3.0. Stay current and know which backends you ship.

Free vs Self-Hosted vs Paid

fully free

Free: The whole library, from pip or uv, under Apache-2.0. Every quantization method, the evaluation tools and the inference backends are included.

Self-hosted: It runs wherever your Python and GPUs are: Linux with NVIDIA, AMD, Intel XPU, Huawei Ascend or Tenstorrent, macOS on Apple Silicon, and Windows 11 with NVIDIA or CPU. Your cost is compute time for calibration and the GPU memory a large model needs while it is being quantized.

Paid: None. ModelCloud asks for sponsorship (compute, API credits or money) through the maintainer's X account; there is no paid edition or support plan.

Free with no upsell. Your real spend is the GPU time to calibrate and evaluate a model, which is still far cheaper than running it unquantized.

What to do by team size

Solo
free, or just download pre-quantized checkpoints
Small team
free, evaluate quality against the full model before shipping
Medium team
free, pin releases given the fast cadence
Large team
free, check the AGPL Swordfish kernel against your distribution policy
Self-hosting ops:moderate

Get tools like this every Wednesday

One featured tool, three on the radar. No fluff.

Score
70/100 · B+
Adoption13/30
Maintenance25/25
Community7/20
License15/15
Analysis10/10

A low score is not a verdict on quality. Young and niche tools start low by design. How we calculate scores

Trust Signals

Organization account (28 public repos)

License: Apache License 2.0

Apache-2.0, but the optional JIT-loaded Swordfish kernel is AGPL-3.0-or-later.

Commercial use: ✓ Yes

About

Owner
ModelCloud.ai (Organization)
Stars
1,290
Forks
216

Explore Further

More tools in the directory