19 articles · newest first
One person on one machine, or a team hitting one server? That single question decides it. Then licence, GPU and Mac support, and the API your app talks to, all read from each project's own docs.
Tensor · 6d ago · 9 min
Seven mixture-of-experts models, from gpt-oss 20B to Qwen3 235B, split into what stays on your card and what moves to system RAM. The card barely changes. The RAM you need goes from 16 GB to 96 GB.
Voltage · 6d ago · 10 min
llama.cpp's own guide runs it on a 16GB Mac with one flag. But on a Mac the CPU and GPU share one pool of memory, so the whole model still has to fit, and 16GB leaves almost nothing for macOS. Here's the maths.
Tensor · Sep 25 · 6 min
Everyone says 16GB minimum. That's only true if the whole model has to live on the GPU. llama.cpp's own guide runs it on an 8GB RTX 2060. Here are the commands, and the starting values for your card.
Tensor · Sep 25 · 7 min
Quantization is the trick that lets a 16GB card run models it has no business running. Here's how GGUF quant levels work, which one I default to, the memory math, and the honest quality trade — so you get the most model your VRAM can hold.
Tensor · Sep 22 · 9 min
A handful of llama.cpp flags let a 16GB card run models that shouldn't fit, and most people running local AI have never touched them.
Tensor · Aug 25 · 7 min
AMD's own numbers put its workstation GPU at 53 tokens a second on Meta's new 30B model. Nvidia's flagship does more — for over three times the price.
Voltage · Aug 11 · 6 min
One builder added a used V100 SXM2 and a bare adapter to an RTX 4080 PC and got 32GB of VRAM. The speed is real. So are the noise, the legacy driver and NVIDIA's dropped Volta support in CUDA 13.
Voltage · Jul 31 · 8 min
Both are free, both run the same open models, and since mid-2025 both have a desktop app. The real split now: LM Studio if you want to browse, tweak and chat; Ollama if you want a model that other apps can call. Here's the full comparison, including speed.
Voltage · Jul 30 · 10 min
They all run local models, but they're built for completely different people. One's for easy desktop use, one's for flexibility, one's for serving at scale. Here's how to pick without wasting a weekend.
Tensor · Jul 28 · 9 min
ExLlamaV2 is open-source and free (MIT), and on a GPU it's roughly twice as fast as llama.cpp. The catch: your model has to fit entirely in VRAM. Here's exactly when to use each one.
Tensor · Jul 27 · 8 min
--n-cpu-moe is the setting that lets a modest graphics card punch far above its VRAM. It offloads the parts of a Mixture-of-Experts model you use least to system RAM — and here's exactly how to tune it for your card.
Tensor · Jul 27 · 9 min
Blackwell versus the old flagship, and for once the newer card loses. The 5080's 16GB caps what you can run; the 4090's 24GB doesn't. For local models, capacity beats the spec sheet — with the measured numbers to prove it.
Voltage · Jul 22 · 9 min
The RX 7900 XTX gives you 24GB cheaper than a 3090. The catch isn't what you've heard — a llama.cpp scoreboard and an open bug report suggest its Vulkan backend beats ROCm for inference, no painful install required.
Tensor · Jul 22 · 9 min
Measured llama.cpp numbers say the 4090 wins every speed test and costs twice as much — while energy per generated token is a dead heat. Here's who each card is actually for.
Voltage · Jul 22 · 9 min
"Cheapest" flips completely on how often you run it and how many people hit it at once — and in Europe, the electricity bill alone can cost most of what renting does.
Tensor · Jul 21 · 10 min
Adding CPU offload should make inference slower. Sometimes it makes it five times faster. Both are true, and the reason matters more than the trick.
Tensor · Jul 21 · 9 min
Two 3090s give you 48GB, but not double the speed for chat, because llama.cpp's default multi-GPU mode makes the cards take turns. What the benchmarks say, rechecked in October 2026.
Voltage · Jul 21 · 11 min
Llama 4 Scout, Qwen3.6-27B, GPT-OSS-20B, Phi-4-14B — four totally different 'best' models, and which one is yours comes down to a single number: how much VRAM your GPU actually has.
Tensor · Jul 18 · 6 min