Quantization explained
Every local model you download comes in several sizes. Those Q4_K_M, Q6_K and f16 labels tell you how much VRAM the model needs and how much quality it keeps. This guide explains what they mean in plain English and how to pick the right one for your GPU.
What is quantization?
Quantization compresses a model's weights to fewer bits per parameter. Instead of storing every number in full 16-bit precision, a quantized model rounds each one down to 8, 6, 5, or 4 bits. The result is a much smaller file that uses far less VRAM and RAM, at a small cost to quality.
The trade-off is simple: fewer bits means a smaller model that fits on more modest hardware, but at some point the rounding starts to blur the model's coherence. The art is choosing the lowest quant that still fits comfortably in your GPU while keeping quality high.
Reading GGUF quant names
GGUF is the file format most local models ship in. Its quant names look cryptic but follow a consistent pattern once you break them apart:
- The number = bits per weight.
q4is roughly 4-bit,q5is 5-bit,q6is 6-bit,q8is 8-bit.f16means full 16-bit half precision — no quantization at all. - _K = "k-quant". A modern quantization scheme that gets noticeably better quality per bit than the older methods. Almost every quant worth using today is a k-quant.
- _S / _M / _L = small / medium / large variant. These put more bits into the model's most important tensors. Large keeps more, so it's bigger and better; small trims more, so it's leaner but weaker.
Put it together and the names decode cleanly:
q4_K_M— a 4-bit k-quant, medium variant. The popular default sweet spot.q8_0— 8-bit, near-lossless. Roughly twice the size of a q4.f16— full half precision. The largest file and the reference-quality baseline.q2_K,q3_K_S— aggressive low-bit quants that save the most VRAM but visibly degrade quality and coherence, especially on smaller models.
Quant reference table
| Quant | Bits | Relative size | When to use |
|---|---|---|---|
q2_K / q3_K_S | ~2–3 | Smallest | Last resort — only to squeeze a bigger model in. Expect noticeable quality loss. |
q4_K_M | ~4 | Small | Recommended default. Best quality-per-GB for most people. |
q5_K_M | ~5 | Medium | A step up if it still fits your VRAM and you want more quality. |
q6_K | ~6 | Larger | Very close to lossless. Great when you have VRAM to spare. |
q8_0 | 8 | ~2× a q4 | Near-lossless. Max quality without going to full precision. |
f16 | 16 | Largest | Reference quality. Only if the full model fits and you need the baseline. |
Which quant should you pick?
For most people, q4_K_M is the right default — it delivers the best quality per gigabyte. Go higher (q5_K_M, q6_K, q8_0) if it still fits your VRAM and you want maximum quality. Only drop below q4 if you have to fit a larger model.
One rule to remember: a bigger model at q4 usually beats a smaller model at q8. If you're choosing between a 32B at q4 and a 14B at q8 that take similar VRAM, the bigger model at the lower quant is generally the smarter pick.
Estimating VRAM from parameters
A quick rule of thumb: a model's q4_K_M GGUF is roughly ~0.6 GB per billion parameters. That gives you a fast gut check before you download anything:
| Model size | Approx. q4_K_M file |
|---|---|
| 8B | ~5 GB |
| 32B | ~20 GB |
| 70B | ~40 GB |
The weights aren't the only thing in VRAM. The KV cache grows with your context length, so leave room on top of the model file — a model that just fits at idle can run out of memory once you fill a long context.
KV-cache quantization
The KV cache is the memory the model uses to hold your conversation's context, and it grows as the context gets longer. Just like the weights, it can be quantized. TurboLLM can quantize the KV cache to fit more context into the same VRAM — useful when you want longer conversations or bigger documents without upgrading your GPU.
Let TurboLLM do the guessing
You don't have to memorize any of this. TurboLLM takes the guesswork out of picking a quant:
VRAM-fit verdict
For each model, TurboLLM tells you whether it fits your GPU before you download it — no more trial and error.
Per-GPU quant recommendation
TurboLLM suggests the best quant for your specific hardware, so you land on the sweet spot without reading tables.
Real tokens/sec, measured
After you load a model, TurboLLM measures the actual tokens/sec on your machine — so you can compare quants empirically instead of trusting a spec sheet.
Because the speed numbers are measured on your own GPU (never faked), you can load the same model at q4_K_M and q5_K_M, compare them side by side, and keep whichever balance of speed and quality you prefer. For a fully hands-off approach, let auto-tune find the best settings for you, and browse the models hub to see fit verdicts and quant recommendations for every model.