What it actually costs to run a model locally
Not “best GPU” lists. For each model: the memory it really needs at each quantisation, which cards fit it, and what the same job costs to rent by the hour — computed from the model's own config and live provider pricing.
18 models52 GPUs99 live prices tracked hourly
On a 24 GB card at 16k context
| Model | Params | Best fit | Uses |
|---|---|---|---|
| gpt-oss 120B (MoE)MoE | 120.4B | doesn't fit | — |
| Llama 3.3 70B Instruct | 70.6B | doesn't fit | — |
| Mixtral 8x7B (MoE)MoE | 46.7B | Q2_K | 21 GB |
| Qwen2.5 Coder 32B | 32.8B | Q4_K_M | 23 GB |
| DeepSeek R1 Distill 32B | 32.8B | Q4_K_M | 23 GB |
| Qwen3 32B | 32.8B | Q4_K_M | 22 GB |
| Qwen3 30B-A3B (MoE)MoE | 30.5B | Q5_K_M | 22 GB |
| Gemma 3 27B | 27.4B | doesn't fit | — |
| Gemma 2 27B | 27.2B | doesn't fit | — |
| Mistral Small 24B | 23.6B | Q6_K | 22 GB |
| gpt-oss 20B (MoE)MoE | 21.5B | Q8_0 | 23 GB |
| Qwen3 14B | 14.8B | Q8_0 | 18 GB |
| Phi-4 14B | 14.7B | Q8_0 | 18 GB |
| Gemma 2 9B | 9.2B | doesn't fit | — |
| Qwen3 8B | 8.2B | F16 | 18 GB |
| Llama 3.1 8B Instruct | 8.0B | doesn't fit | — |
| Qwen2.5 7B Instruct | 7.6B | F16 | 16 GB |
| Mistral 7B Instruct v0.3 | 7.2B | F16 | 16 GB |
Memory figures are computed from each model's own config.json, using bits-per-weight constants checked against real published quantised file sizes (median error 0.7%). Rental prices are pulled hourly from provider APIs. Full methodology is on each model's page.