q4_k_m

2 articles · newest first

AI

How to Quantize a Local LLM in 2026 (GGUF Q4 vs Q6 vs Q8 — and What You Lose)

Quantization is the trick that lets a 16GB card run models it has no business running. Here's how GGUF quant levels work, which one I default to, the memory math, and the honest quality trade — so you get the most model your VRAM can hold.

Aliteq Team · Sep 22 · 9 min

AI

which quantization should you use? Q4 vs Q5 vs Q8, settled with real numbers

Every model comes in a dozen quant sizes and it's paralyzing. Here's the simple truth: Q4_K_M is the default for almost everyone, and here's exactly when to go higher.

Aliteq Team · Aug 4 · 10 min