Every model comes in a dozen quant sizes and it's paralyzing. Here's the simple truth: Q4_K_M is the default for almost everyone, and here's exactly when to go higher.
Every model on Hugging Face comes in a dozen quant sizes — Q2, Q3, Q4_K_S, Q4_K_M, Q5, Q6, Q8 — and it's genuinely paralyzing. So here's the simple truth: Q4_K_M is the right default for almost everyone. It's ~75% smaller than the full FP16 model with only about 3.5% quality loss — the best balance of size and smarts there is. Go Q5_K_M if you have 12GB+ of VRAM to spare and do coding or complex reasoning (the quality jump is noticeable there). And Q8_0 is overkill for most people — nearly identical to full precision but far heavier and slower, for a difference you won't feel in normal use. That's the whole decision, settled. Here are the real numbers behind it.
The real numbers (why Q4_K_M wins)
Quality is measured with perplexity (lower = better; think of it as how 'surprised' the model is by text). The gaps between quant levels are smaller than people fear: Q4_K_M scores 6.61 perplexity vs Q5_K_M's 6.58 — a rounding error. And the delta between Q4_K_M and Q8_0 is about 0.05 perplexity — below the threshold where responses look perceptibly different in normal conversation. In other words, for casual chat, you cannot feel the difference between Q4 and Q8 — so paying the size and speed cost of Q8 buys you nothing you'll notice. Where the jump from Q4 to Q5 does show up is coding and complex reasoning, where small errors compound — there, Q5_K_M (or Q6_K) earns its extra memory. One technical tip that's always worth it: choose the K_M ('medium') variant over K_S ('small'). K_M spends a few more bits on the attention layers that most affect quality, while K_S quantizes everything uniformly — the K_M size increase is tiny and it's almost always worth it. Below Q4, at Q3 and Q2, quality finally degrades in ways you will notice, so only drop that low when it's the only way to fit the model.
Which GGUF quant to pick
8-12GB
Your VRAM
Q4_K_M
Use
Sweet spot — ~3.5% loss, fits easily
12-16GB
Your VRAM
Q5_K_M or Q6_K
Use
Better coding/reasoning, still fits
16-24GB
Your VRAM
Q8_0 (if you want)
Use
Near-FP16, but Q5 is usually plenty
Very tight
Your VRAM
Q3 (last resort)
Use
Noticeable quality loss — avoid if possible
Your VRAM
Use
Why
8-12GB
Q4_K_M
Sweet spot — ~3.5% loss, fits easily
12-16GB
Q5_K_M or Q6_K
Better coding/reasoning, still fits
16-24GB
Q8_0 (if you want)
Near-FP16, but Q5 is usually plenty
Very tight
Q3 (last resort)
Noticeable quality loss — avoid if possible
For casual chat, Q4 and Q8 are perceptually identical — Q4_K_M is the right default for almost everyone. · Unsplash
How to choose in ten seconds
Here's the practical routine. Step one: default to Q4_K_M. Unless you have a specific reason not to, it's the pick — great quality, small footprint, fits comfortably on 8-12GB cards. Step two: if you have VRAM to spare (12GB+) and you code or do heavy reasoning, step up to Q5_K_M or Q6_K — that's where the extra bits actually pay off. Step three: only reach for Q8_0 if you have plenty of memory and want the absolute maximum, knowing you likely won't feel the difference over Q5. Step four: only drop to Q3 or Q2 if it's the only way to fit the model you need — and expect a real quality hit. The overarching rule is simple: pick the highest quant that still leaves comfortable VRAM headroom (you need room for context too, not just the weights), and when in doubt, Q4_K_M. Check the exact fit for any model and quant in the VRAM calculator, and you'll never agonize over that list of quant files again. For the deeper 'what is quantization' explainer, start here.
Quick answers
Which GGUF quantization should I use?
For almost everyone, Q4_K_M is the right default — it's about 75% smaller than the full FP16 model with only ~3.5% quality loss, and it fits comfortably on 8-12GB GPUs. Step up to Q5_K_M or Q6_K if you have 12GB or more of VRAM to spare and you do coding or complex reasoning, where the quality jump is noticeable. Q8_0 is overkill for most people: it's near full-precision quality but much heavier and slower, for a difference you won't perceive in normal use. Only drop to Q3 or Q2 if it's the only way to fit the model. When in doubt, choose Q4_K_M.
Is there a real difference between Q4 and Q8?
On paper yes, in practice usually no. Q4_K_M scores about 6.61 perplexity versus roughly 6.56 for Q8_0 — a gap of around 0.05, which is below the threshold where responses look perceptibly different in normal conversation. So for casual chat and most everyday tasks, you genuinely cannot feel the difference between Q4 and Q8, which means paying Q8's much larger memory and slower speed buys you nothing you'll notice. The one place a higher quant helps is coding and complex reasoning, where small errors compound — but even there, Q5_K_M usually captures the benefit without going all the way to Q8.
What does the K_M in Q4_K_M mean?
K_M stands for the 'medium' K-quant variant. K-quants spend different numbers of bits on different parts of the model, and the K_M version uses more bits for the attention layers that most affect output quality, while the K_S ('small') version quantizes everything uniformly. The result is that K_M preserves noticeably more quality than K_S for only a tiny increase in file size, so it's almost always worth choosing K_M over K_S at the same bit level. That's why Q4_K_M, rather than Q4_K_S or plain Q4_0, is the recommended default for most local models.
