quantization

15 articles · newest first

Hand-drawn editorial illustration of a hand pressing down the lid of a suitcase packed so full of paper cards that a fan of coral cards spills out, on a near-black background in indigo-violet and off-white

Out of memory at long context? Shrink the KV cache with q8_0 (llama.cpp and Ollama)

The model fits. Then you paste in a long document and the GPU runs out of memory. The culprit is the KV cache, it has a formula you can do on a napkin, and two settings cut it by roughly half or three quarters.

Aliteq Team · Oct 16 · 9 min

Hand-drawn illustration of a small bonsai tree in a cream pot with coral pruning shears, in front of the faint violet outline of a giant tree on a near-black background

Bonsai 2 27B: How PrismML Shrinks a 27B Model to 5.9 GB

A 27B model that fits in about 6 GB sounds like a trick. It is ternary weights, a shared scale and a rotation. Here is how it works, what PrismML's own tests show, and where the loss hides.

Aliteq Team · Oct 10 · 8 min

Macro photo of rows of identical blank black memory chips on a circuit board, their pins lit in indigo-violet against a near-black background

Qwen3.8 27B Hardware Requirements: VRAM by Quant, and Why Context Costs So Little

A 24 GB card runs Qwen3.8 27B at Q4 or Q5. Q8 needs 32 GB. The surprise is the context cache: 128K tokens costs about 8 GB, a quarter of what an older 32B model charges.

Aliteq Team · Oct 7 · 9 min

Illustration of compressing a large block of AI experts into a smaller box

Llama 4 Quantized: GGUF Sizes & Memory, Explained Honestly

Quantization is the only reason Scout runs at home — but MoE means you shrink all 109B, not the 17B active. The honest GGUF memory ladder, the Q4_K_M sweet spot, and how low you can safely go.

Aliteq Team · Sep 22 · 8 min

Flat illustration of a person compressing a large cube into a small box, on a teal background

How to Quantize a Local LLM in 2026 (GGUF Q4 vs Q6 vs Q8 — and What You Lose)

Quantization is the trick that lets a 16GB card run models it has no business running. Here's how GGUF quant levels work, which one I default to, the memory math, and the honest quality trade — so you get the most model your VRAM can hold.

Aliteq Team · Sep 22 · 9 min

A consumer graphics card installed in a desktop PC, the kind of hardware llama.cpp's MoE offloading flags are designed to stretch further

your GPU isn't too small for that 235B model — you're just using it wrong

A handful of llama.cpp flags let a 16GB card run models that shouldn't fit, and most people running local AI have never touched them.

Aliteq Team · Aug 25 · 7 min

DDR5 RAM memory modules installed on a desktop motherboard

everyone's obsessing over GPU VRAM for local AI. your RAM and SSD are what actually wreck the experience

A new hardware breakdown puts real numbers on the two components most local-AI buying guides skip entirely — and skimping on either turns a snappy agent into a 10-minute wait.

Aliteq Team · Aug 19 · 7 min

Rows of GPUs mounted in an open server rack used for large-scale AI inference

DeepSeek's 'cheap' AI model costs 3 cents a query in the cloud. running it yourself needs $10k of GPUs

the model that made headlines for being nearly free to rent turns out to be one of the most expensive things you could try to self-host

Aliteq Team · Aug 4 · 6 min

A laptop screen showing a local AI language model running in a terminal window

three free AI models all claim a 128K memory — only one actually holds onto it

Phi-4 Mini, Gemma 3, and Llama 3.2 all advertise roughly the same 128K context window. Benchmark data says that number means something very different for each one.

Aliteq Team · Aug 4 · 7 min

A desktop PC with a mid-range graphics card installed, the kind of 8GB hardware this guide is written for

your 8GB graphics card can run real local AI — you're probably picking the wrong model

Qwen3.5 9B, Gemma 4 and the older Gemma 3 and Phi-4 Mini all claim to fit. We ran each through our VRAM calculator at 8K and 32K context to see which ones actually leave room to breathe.

Aliteq Team · Aug 4 · 6 min

A small humanoid robot sitting on a red wooden bench, reading from a tablet

I priced out every way to run a 70B AI model at home — the cheapest one isn't what you'd guess

Four real paths to running a 70B-parameter model on your own hardware or by the hour — priced out with actual 2026 numbers.

Aliteq Team · Aug 4 · 8 min

A graphics card showing VRAM memory chips

how much VRAM do you need to run AI models? The simple rule, by model size

VRAM is the single number that decides which AI models you can run. Here's the simple rule of thumb, a size-by-size table, and a calculator to check any model against your card.

Aliteq Team · Jul 30 · 9 min

only got 8GB of VRAM? you can still run real local AI — here's exactly what fits

8GB is the entry floor for local AI, and plenty of capable models run on it if you pick the right size and quantization. Here's what actually works, and the trap to avoid.

Aliteq Team · Jul 28 · 9 min

Mathematical equations on a dark background, representing quantization math

NVFP4 is Nvidia's 4-bit trick that gets FP8 accuracy at 3x the speed — but only on Blackwell

The new 4-bit format baked into RTX 50-series and Blackwell chips runs AI models at up to 3x the inference throughput of FP8, at nearly the same accuracy. Here's how it works and why it's exclusive to your newest GPU.

Aliteq Team · Jul 28 · 8 min

Close-up of computer hardware used for local AI inference

The MoE offload trick that makes your model 5× faster — and why it probably won't work for you

Adding CPU offload should make inference slower. Sometimes it makes it five times faster. Both are true, and the reason matters more than the trick.

Aliteq Team · Jul 21 · 9 min