What it costs to run gpt-oss 120B (MoE)
Trying to decide what to buy? The best GPU for gpt-oss 120B (MoE) →
Memory needed, by quantisation
At 16k context. Weights + KV cache + ~0.8 GB overhead.
| Quant | Weights | KV cache | Total | Quality |
|---|---|---|---|---|
| F16 | 224.3 GB | 0.8 GB | 225.9 GB | Full precision. Reference quality, twice the size of Q8 for no practical gain in most chat use. |
| Q8_0 | 119.2 GB | 0.8 GB | 120.7 GB | Effectively lossless. Use when VRAM is not the constraint. |
| Q6_K | 92.0 GB | 0.8 GB | 93.5 GB | Very close to Q8 at meaningfully less memory. A safe high-quality pick. |
| Q5_K_M | 79.6 GB | 0.8 GB | 81.2 GB | Small, hard-to-notice quality loss. Good balance. |
| Q5_0 | 77.7 GB | 0.8 GB | 79.2 GB | Older-style 5-bit. Q5_K_M is usually the better pick at the same size. |
| Q4_K_M | 68.0 GB | 0.8 GB | 69.6 GB | The community default. Best quality-per-gigabyte for most people. |
| Q4_0 | 63.8 GB | 0.8 GB | 65.4 GB | Older-style 4-bit, measurably worse than Q4_K_M at a similar size. Avoid unless required. |
| IQ4_XS | 59.6 GB | 0.8 GB | 61.1 GB | Newer 4-bit, smaller than Q4_K_M with comparable quality. Needs a recent llama.cpp. |
| Q3_K_M | 54.8 GB | 0.8 GB | 56.4 GB | Noticeable degradation. Use to fit a larger model that would otherwise not run. |
| Q2_K | 47.0 GB | 0.8 GB | 48.5 GB | Heavy degradation. Almost always better to run a smaller model at Q4_K_M instead. |
| IQ3_XS | 46.3 GB | 0.8 GB | 47.8 GB | Aggressive. Usually better than a smaller model at Q4, but test before trusting it. |
Cards that can run it
Best quantisation each card fits at 16k context, with live rental price where we track one.
| GPU | VRAM | Best fit | Uses | ~tok/s | Rent from |
|---|---|---|---|---|---|
| Apple M3 Ultra | 512 GB | F16 | 226 GB | — | — |
| Apple M2 Ultra | 192 GB | Q8_0 | 121 GB | — | — |
| Apple M4 Max (16-core CPU / 40-core GPU) | 128 GB | Q8_0tight | 121 GB | — | — |
| AMD Ryzen AI Max+ 395 (Strix Halo, 128 GB) | 128 GB | Q8_0tight | 121 GB | — | — |
| NVIDIA DGX Spark (GB10 Grace Blackwell) | 128 GB | Q8_0tight | 121 GB | — | — |
| NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition | 96 GB | Q6_Ktight | 94 GB | — | $1.690/hr |
| NVIDIA RTX PRO 6000 Blackwell Workstation Edition | 96 GB | Q6_Ktight | 94 GB | — | — |
| Apple M2 Max | 96 GB | Q6_Ktight | 94 GB | — | — |
| Apple M4 Pro | 64 GB | IQ4_XStight | 61 GB | — | — |
| NVIDIA RTX 6000 Ada Generation | 48 GB | IQ3_XStight | 48 GB | — | $0.356/hr |
| NVIDIA RTX A6000 | 48 GB | IQ3_XStight | 48 GB | — | $0.267/hr |
| AMD Radeon PRO W7900 | 48 GB | IQ3_XStight | 48 GB | — | — |
How these numbers are produced
- Model shape comes from the model's own config.json — 36 layers, 64 attention heads, 8 KV heads.
- Weight size is parameters × effective bits-per-weight. Those constants are checked against real published quantised file sizes — median error 0.7% across 40 measurements.
- KV cache is 2 × layers × KV-heads × head-dim × context × 2 bytes. Using KV-heads rather than attention heads is what makes this correct for grouped-query attention; treating a GQA model as multi-head overstates the cache by up to 8×.
- Tokens/sec is an estimate, not a benchmark. Generation is memory-bandwidth-bound, so this is bandwidth ÷ weight-bytes derated to 75%. Real throughput depends on your runtime, batch size and kernels. We don't run our own hardware tests — see our editorial policy.
- This is a mixture-of-experts model. Generation reads only the experts active for each token (4 of 128), so throughput is scored on active parameters, not the full 120B. Memory is the opposite: every expert must still be resident in VRAM, so the figures above use the full weight set.Derived from architecture; runs ~15% above the vendor-published active count for this family (distinct GptOssForCausalLM architecture). Treat derived tok/s as optimistic.
- Rental prices are pulled hourly from provider APIs (last updated Wed, 22 Jul 2026 22:20:08 GMT). Spot/interruptible pricing can change or vanish without notice.
Try it interactively
Drag the context slider and watch the KV cache fill the card
Worth reading before you buy
What --n-cpu-moe actually does
The trick that makes MoE models 5× faster — except it usually makes them slower, and the famous speedup only happens when the model didn't fit in the first place.
Is the DGX Spark worth it?
NVIDIA spent three months optimising it and token generation got slower — because 128 GB of memory on a 273 GB/s bus holds huge models and reads them slowly. Who should actually buy one.
RTX 5090 vs two RTX 3090s
Two 3090s give you 48 GB but not double the speed for chat — llama.cpp's default multi-GPU mode makes the cards take turns. What the benchmarks actually show.