llama.cpp

6 articles · newest first

Hardware

Meta's new AI model runs at home now — one $1,299 AMD card almost keeps up with a $4,700 Nvidia one

AMD's own numbers put its workstation GPU at 53 tokens a second on Meta's new 30B model. Nvidia's flagship does more — for over three times the price.

Ravi Malhotra · 4d ago · 6 min

Hardware

someone bolted a $266 Nvidia server GPU to their gaming PC. it's loud as a lawnmower — and it works

A used Tesla V100 and a $50 adapter got one builder to 32GB of VRAM for $266 total — here's the honest math on whether you should copy them.

Ravi Malhotra · Jul 31 · 7 min

AI

Ollama vs LM Studio: which one should you actually use to run AI locally?

They're the two most popular ways to run AI models on your own computer, and both are free. The short answer: LM Studio if you want a friendly app, Ollama if you want to build. Here's the full comparison.

Ravi Malhotra · Jul 30 · 9 min

AI

vLLM, Ollama or llama.cpp? the three ways to run local AI, and which one is actually for you

They all run local models, but they're built for completely different people. One's for easy desktop use, one's for flexibility, one's for serving at scale. Here's how to pick without wasting a weekend.

Lena Fischer · Jul 28 · 9 min

AI

is ExLlama free? yes — and it's the fastest way to run a local model, with one big condition

ExLlamaV2 is open-source and free (MIT), and on a GPU it's roughly twice as fast as llama.cpp. The catch: your model has to fit entirely in VRAM. Here's exactly when to use each one.

Lena Fischer · Jul 27 · 8 min

AI

how to run a 30B model on a 12GB GPU: the one llama.cpp flag that makes it possible

--n-cpu-moe is the setting that lets a modest graphics card punch far above its VRAM. It offloads the parts of a Mixture-of-Experts model you use least to system RAM — and here's exactly how to tune it for your card.

Lena Fischer · Jul 27 · 9 min