quantization

6 articles · newest first

A consumer graphics card installed in a desktop PC, the kind of hardware llama.cpp's MoE offloading flags are designed to stretch further

your GPU isn't too small for that 235B model — you're just using it wrong

A handful of llama.cpp flags let a 16GB card run models that shouldn't fit, and most people running local AI have never touched them.

Lena Fischer · 5d ago · 7 min

DDR5 RAM memory modules installed on a desktop motherboard

everyone's obsessing over GPU VRAM for local AI. your RAM and SSD are what actually wreck the experience

A new hardware breakdown puts real numbers on the two components most local-AI buying guides skip entirely — and skimping on either turns a snappy agent into a 10-minute wait.

Lena Fischer · Aug 19 · 7 min

Rows of GPUs mounted in an open server rack used for large-scale AI inference

DeepSeek's 'cheap' AI model costs 3 cents a query in the cloud. running it yourself needs $10k of GPUs

the model that made headlines for being nearly free to rent turns out to be one of the most expensive things you could try to self-host

Lena Fischer · Aug 4 · 6 min

A laptop screen showing a local AI language model running in a terminal window

three free AI models all claim a 128K memory — only one actually holds onto it

Phi-4 Mini, Gemma 3, and Llama 3.2 all advertise roughly the same 128K context window. Benchmark data says that number means something very different for each one.

Lena Fischer · Aug 4 · 7 min

A desktop PC with a mid-range graphics card installed, the kind of 8GB hardware this guide is written for

your 8GB graphics card can run real local AI — you're probably picking the wrong model

Phi-4 Mini, Gemma 3, and Llama 3.2 all claim to fit — I checked which one actually leaves room to breathe.

Lena Fischer · Aug 4 · 6 min

Nvidia GeForce RTX 3090 graphics card, popular for its 24GB of VRAM

I priced out every way to run a 70B AI model at home — the cheapest one isn't what you'd guess

Four real paths to running a 70B-parameter model on your own hardware or by the hour — priced out with actual 2026 numbers.

Lena Fischer · Aug 4 · 8 min