Quantization
Quantization is a way to store a model’s numbers using less space. It can make the model smaller and faster, but the answers may change.
A smaller model is not always faster or good enough. Try the exact version you plan to use, and check both its speed and its answers.
What quantization is
Quantization stores numbers with less detail. For example, you might replace 3.14159 with 3.14. A suitable format can store the shorter number in less space, but it is not exact. Models use more sophisticated versions of this idea to store their weights. A scale helps turn each stored code back into an approximate number.
Most integer formats use groups. A group is a small block of weights that shares a scale. Smaller groups let the scale follow local values more closely, at the cost of more scales. The error depends on the values and the quantizer. The bytes per weight include both the code and the scale:
For example, a 4-bit code with an FP16 scale per 16 weights uses about 4 + 16/16 = 5 bits per weight. The real number also includes alignment, metadata, and any global scale. A 27B model at 4.5 bits per weight needs about 15.2 GB, not 13.5 GB.
Quantization is not one method. It covers weight-only compression, activation quantization, KV cache quantization, and mixed precision. Each one has a different risk and a different kernel path.
Common formats
These names describe different things. GGUF is a container with quantized tensor types; FP8, NVFP4, and MX formats describe numerical representations; EXL3 describes a coded weight representation. AWQ and GPTQ are quantization methods. Storage costs below are approximate and vary with the checkpoint.
| Format | Approx. bits | How it stores values | Typical path |
|---|---|---|---|
| GGUF Q8_0 | 8.5 | 8-bit integer blocks with scales | llama.cpp, CPU and GPU |
| GGUF Q5_K_M | 5.5 | Mixed k-quant blocks | llama.cpp |
| GGUF Q4_K_M | 4.5 | 4-bit k-quant blocks, some tensors wider | llama.cpp |
| FP8 E4M3 | 8 | 8-bit float, per-tensor or per-channel scale | Hopper, Ada, some kernels |
| NVFP4 | 4.5 | 4-bit float values, FP8 block scale, global scale | Blackwell, ModelOpt |
| MXFP8 / MXFP4 | 8.25 / 4.25 | Block scale shared by 32 values | Open standard, new kernels |
| EXL3 | Variable | Trellis-coded weights, a width per tensor | ExLlamaV3-style kernels |
MX storage includes one 8-bit scale per 32 values: 0.25 extra bits per value. See the MX specification and EXL3 conversion documentation for format details.
AWQ and GPTQ are post-training methods often used for 4-bit integer weights; their actual storage and runtime depend on the checkpoint and engine.
These rows are not equivalent. NVFP4 uses a floating-point grid and block scales. Its error differs from a uniform integer grid; neither is best for every set of values. EXL3 uses trellis coding with a configurable bit budget; 3 bits per weight is one possible target. GGUF K-quants mix formats across tensors. Compare checkpoints, not labels.
Quantization-aware distillation (QAD) is a training method, not a storage format. It trains a low-bit student to match a high-precision teacher. That can recover quality at 4 bits, but it needs training compute and a good teacher. It does not remove the need to test the final checkpoint.
The chart shows the original weights as dots and the reconstructed weights as a line. Smaller groups can reduce reconstruction error, although the result depends on the weights and the quantizer. They also add a scale for every group, so the bytes per weight rise. This is the central trade.
Methods
Post-training quantization (PTQ) calibrates on a small sample of text, then chooses scales. GPTQ and AWQ are common weight-only methods. PTQ is cheap, but it can fail on layers with large outliers. Calibration data that does not match your workload makes the failure worse.
Quantization-aware training (QAT) simulates low precision during training. The model learns to tolerate the error. QAT costs more compute, and it can reach better quality than PTQ at the same bit width.
Quantization-aware distillation (QAD) trains a low-bit student to match a teacher's outputs. It can combine low-bit weights with a recovery pass. The result depends on the teacher, the data, and the training budget.
Activation quantization reduces the bytes moved in matrix multiplies. It is harder than weight quantization, because activations change with every input and can have large outliers. W4A4 and W8A8 are common labels: weight bits and activation bits.
KV cache quantization cuts the cache bytes per token. It can extend context or batch size. It can also hurt long-context recall, because the cache feeds every decode step.
Quality checks
Do not accept a quantized checkpoint because its perplexity is close. Perplexity is one number on one dataset. Use a set of checks that match your product:
- Held-out perplexity or KL divergence against the exact baseline.
- Task accuracy on your own evaluation set, with the same prompt template and sampler.
- Long-context recall, if you serve long prompts.
- Code tests, tool-call format, and JSON validity, if you use them.
- Multilingual quality and safety behavior, if they matter.
Run the checks at the same context length, temperature, and top-p as production. A quantized model can pass a short multiple-choice test and fail a long extraction task. Report confidence intervals, not a single run. The measurement page explains how.
Speed is not automatic
Lower precision cuts bytes per weight, so decode can speed up. That only happens if a fast kernel exists for the format. A 4-bit format with a slow dequantization path can be slower than an 8-bit format with a tuned kernel.
Prefill may not benefit. It is often compute-bound, and dequantization adds work. Measure both phases. Check the batch sizes you actually serve.
Converting a checkpoint to a wider representation does not recover information lost during quantization. If conversion also changes scales, activations, or kernels, measure quality and speed for the new path. Use a checkpoint and runtime that have been tested together.