What it actually costs to run a model locally
Not “best GPU” lists. For each model: the memory it really needs at each quantisation, which cards fit it, and what the same job costs to rent by the hour — computed from the model's own config and live provider pricing.
Interactive
Watch context length eat your GPU
Drag a slider and see the KV cache — not the weights — push a model off the card.
On a 24 GB card at 16k context
| Model | Params | Best fit | Uses |
|---|---|---|---|
| gpt-oss 120B (MoE)MoE | 116.8B | doesn't fit | — |
| Llama 3.3 70B Instruct | 70.6B | doesn't fit | — |
| Mixtral 8x7B (MoE)MoE | 46.7B | Q2_K | 21 GB |
| Qwen2.5 Coder 32B | 32.8B | Q4_K_M | 23 GB |
| DeepSeek R1 Distill 32B | 32.8B | Q4_K_M | 23 GB |
| Qwen3 32B | 32.8B | Q4_K_M | 23 GB |
| Qwen3 30B-A3B (MoE)MoE | 30.5B | Q5_K_M | 22 GB |
| Gemma 3 27B | 27.4B | doesn't fit | — |
| Gemma 2 27B | 27.2B | doesn't fit | — |
| Mistral Small 24B | 23.6B | Q6_K | 21 GB |
| gpt-oss 20B (MoE)MoE | 21.5B | Q8_0 | 23 GB |
| Qwen3 14B | 14.8B | Q8_0 | 18 GB |
| Phi-4 14B | 14.7B | Q8_0 | 18 GB |
| Gemma 2 9B | 9.2B | doesn't fit | — |
| Qwen3 8B | 8.2B | F16 | 18 GB |
| Llama 3.1 8B Instruct | 8.0B | doesn't fit | — |
| Qwen2.5 7B Instruct | 7.6B | F16 | 16 GB |
| Mistral 7B Instruct v0.3 | 7.2B | F16 | 16 GB |
Hardware guides
The longer decisions — which card, rent or buy, what the marketing leaves out.
RTX 5090 vs two RTX 3090s
Two 3090s give you 48 GB but not double the speed for chat — llama.cpp's default multi-GPU mode makes the cards take turns. What the benchmarks actually show.
Cheapest way to serve Llama 70B
"Cheapest" flips on duty cycle and concurrency — and in Europe the electricity bill alone can approach the cost of just renting. Plus the config defaults that silently break a 16k deployment.
What --n-cpu-moe actually does
The trick that makes MoE models 5× faster — except it usually makes them slower, and the famous speedup only happens when the model didn't fit in the first place.
Is the DGX Spark worth it?
NVIDIA spent three months optimising it and token generation got slower — because 128 GB of memory on a 273 GB/s bus holds huge models and reads them slowly. Who should actually buy one.
Memory figures are computed from each model's own config.json, using bits-per-weight constants checked against real published quantised file sizes (median error 0.7%). Rental prices are pulled hourly from provider APIs. Full methodology is on each model's page.