quantization

8 articles · newest first

Rows of GPUs mounted in an open server rack used for large-scale AI inference

DeepSeek's 'cheap' AI model costs 3 cents a query in the cloud. running it yourself needs $10k of GPUs

the model that made headlines for being nearly free to rent turns out to be one of the most expensive things you could try to self-host

Lena Fischer · 6d ago · 6 min

A laptop screen showing a local AI language model running in a terminal window

three free AI models all claim a 128K memory — only one actually holds onto it

Phi-4 Mini, Gemma 3, and Llama 3.2 all advertise roughly the same 128K context window. Benchmark data says that number means something very different for each one.

Lena Fischer · 6d ago · 7 min

A desktop PC with a mid-range graphics card installed, the kind of 8GB hardware this guide is written for

your 8GB graphics card can run real local AI — you're probably picking the wrong model

Phi-4 Mini, Gemma 3, and Llama 3.2 all claim to fit — I checked which one actually leaves room to breathe.

Lena Fischer · 6d ago · 6 min

Nvidia GeForce RTX 3090 graphics card, popular for its 24GB of VRAM

I priced out every way to run a 70B AI model at home — the cheapest one isn't what you'd guess

Four real paths to running a 70B-parameter model on your own hardware or by the hour — priced out with actual 2026 numbers.

Lena Fischer · 6d ago · 8 min

A graphics card showing VRAM memory chips

how much VRAM do you need to run AI models? The simple rule, by model size

VRAM is the single number that decides which AI models you can run. Here's the simple rule of thumb, a size-by-size table, and a calculator to check any model against your card.

Ravi Malhotra · Jul 30 · 9 min

only got 8GB of VRAM? you can still run real local AI — here's exactly what fits

8GB is the entry floor for local AI, and plenty of capable models run on it if you pick the right size and quantization. Here's what actually works, and the trap to avoid.

Lena Fischer · Jul 28 · 9 min

Mathematical equations on a dark background, representing quantization math

NVFP4 is Nvidia's 4-bit trick that gets FP8 accuracy at 3x the speed — but only on Blackwell

The new 4-bit format baked into RTX 50-series and Blackwell chips runs AI models at up to 3x the inference throughput of FP8, at nearly the same accuracy. Here's how it works and why it's exclusive to your newest GPU.

Lena Fischer · Jul 28 · 8 min

Close-up of computer hardware used for local AI inference

The MoE offload trick that makes your model 5× faster — and why it probably won't work for you

Adding CPU offload should make inference slower. Sometimes it makes it five times faster. Both are true, and the reason matters more than the trick.

Lena Fischer · Jul 21 · 9 min