llama.cpp

11 articles · newest first

Hardware

someone bolted a $266 Nvidia server GPU to their gaming PC. it's loud as a lawnmower — and it works

A used Tesla V100 and a $50 adapter got one builder to 32GB of VRAM for $266 total — here's the honest math on whether you should copy them.

Ravi Malhotra · Jul 31 · 7 min

AI

Ollama vs LM Studio: which one should you actually use to run AI locally?

They're the two most popular ways to run AI models on your own computer, and both are free. The short answer: LM Studio if you want a friendly app, Ollama if you want to build. Here's the full comparison.

Ravi Malhotra · Jul 30 · 9 min

AI

vLLM, Ollama or llama.cpp? the three ways to run local AI, and which one is actually for you

They all run local models, but they're built for completely different people. One's for easy desktop use, one's for flexibility, one's for serving at scale. Here's how to pick without wasting a weekend.

Lena Fischer · Jul 28 · 9 min

AI

is ExLlama free? yes — and it's the fastest way to run a local model, with one big condition

ExLlamaV2 is open-source and free (MIT), and on a GPU it's roughly twice as fast as llama.cpp. The catch: your model has to fit entirely in VRAM. Here's exactly when to use each one.

Lena Fischer · Jul 27 · 8 min

AI

how to run a 30B model on a 12GB GPU: the one llama.cpp flag that makes it possible

--n-cpu-moe is the setting that lets a modest graphics card punch far above its VRAM. It offloads the parts of a Mixture-of-Experts model you use least to system RAM — and here's exactly how to tune it for your card.

Lena Fischer · Jul 27 · 9 min

AI

the RTX 5080 is newer, faster on paper, and the wrong pick for local AI. here's the 8GB reason why

Blackwell versus the old flagship, and for once the newer card loses. The 5080's 16GB caps what you can run; the 4090's 24GB doesn't. For local models, capacity beats the spec sheet — with the measured numbers to prove it.

Ravi Malhotra · Jul 22 · 9 min

AI

everyone says AMD means fighting ROCm for local AI. on llama.cpp, the thing you're told to install is the slower one

The RX 7900 XTX gives you 24GB cheaper than a 3090. The catch isn't what you've heard — a llama.cpp scoreboard and an open bug report suggest its Vulkan backend beats ROCm for inference, no painful install required.

Lena Fischer · Jul 22 · 9 min

AI

The 4090 is 34% faster for local AI. Your electricity meter can't tell them apart

Measured llama.cpp numbers say the 4090 wins every speed test and costs twice as much — while energy per generated token is a dead heat. Here's who each card is actually for.

Ravi Malhotra · Jul 22 · 9 min

AI

The cheapest way to run Llama 70B isn't the one you're about to build

"Cheapest" flips completely on how often you run it and how many people hit it at once — and in Europe, the electricity bill alone can cost most of what renting does.

Lena Fischer · Jul 21 · 10 min

AI

The MoE offload trick that makes your model 5× faster — and why it probably won't work for you

Adding CPU offload should make inference slower. Sometimes it makes it five times faster. Both are true, and the reason matters more than the trick.

Lena Fischer · Jul 21 · 9 min

Hardware

Two 3090s or one 5090? The answer isn't the one everyone gives

The dual-3090 build doesn't give you double the bandwidth for chat — and the reason has nothing to do with the cards. What the measurements actually say.

Ravi Malhotra · Jul 21 · 11 min