DGX Spark for local LLMs: fast at reading prompts, slow at writing replies

Between November 2025 and February 2026, llama.cpp got three months of CUDA optimization work. Georgi Gerganov benchmarked a DGX Spark on both builds: same machine, same harness, same models. Prompt processing got up to 27% faster.

Token generation got slower, by between 2.8% and 4.1% on every model tested.

That is not a bug anyone needs to fix. It is a memory bus behaving like a memory bus, and it tells you more about whether to buy this machine than any spec sheet does.

How many tokens per second does the DGX Spark get?

Between 29 and 83 tokens per second on the small and mid-size models in llama.cpp's own benchmark, at short context. Mixture-of-experts (MoE) models are fast because only part of the model is read per token. Dense models are slow, because every weight is read for every token. Speed drops as the conversation gets longer.

For scale, llama.cpp's CUDA scoreboard lists the DGX Spark at 56.74 tokens/s on its standard Llama 2 7B Q4_0 test, against 300.40 for an RTX 5090. Our gpt-oss-120b hardware guide compares the Spark with other ways to run that model.

Two numbers, and the one that gets advertised

Generating text is two jobs wearing one name. Prefill is reading your prompt. It handles many tokens at once, so it scales with compute. Decode is writing the reply, one token at a time. Each token means reading the model's active weights from memory, so it scales with memory bandwidth and little else.

NVIDIA's product page leads with up to 1 petaFLOP of FP4 compute. That is a prefill number. The same page lists memory bandwidth at 273 GB/s, and that is the decode number. For context, an RTX 5090 has 1,792 GB/s and Apple's M2 Ultra has 800 GB/s.

The accidental experiment

Gerganov keeps DGX Spark benchmark results in the llama.cpp repository. The file has two versions: build 6989 in November 2025 and build 7941 in February 2026. Comparing them gives a controlled before-and-after on identical hardware, with three months of software work in between.

Three months of kernel work bought a fifth to a quarter more prefill speed. It bought nothing for generation, because generation was never waiting on the kernels. It was waiting on the bus, and the bus doesn't get software updates. The other three models in both files tell the same story: generation fell 2.8% on Qwen3-Coder 30B-A3B, 4.0% on Qwen2.5-Coder 7B and 4.1% on Gemma 3 4B.

DGX Spark vs Mac: it depends on the model

The obvious rival is a Mac with lots of unified memory. llama.cpp keeps a matching M2 Ultra benchmark file with the same models, so the comparison is fair. The winner flips depending on what you run.

On a dense model it isn't close. Qwen2.5-Coder 7B at Q8_0: the M2 Ultra does 79.68 tokens/s, the Spark 29.43. That is a 2.71x gap against a bandwidth ratio of 2.93x (800 / 273). Both machines run at roughly 80 to 90% of what their memory bus allows. The ranking is about which one has the wider pipe.

On MoE models the gap shrinks, because only a fraction of the weights is read per token. On gpt-oss-120b the Mac's lead falls from 2.71x to 1.46x (85.60 vs 58.72 tokens/s).

And on prefill the Spark wins outright, more so as context grows:

That last row is the strongest case for this machine. At 32K of context on a 30B MoE model, the Spark processes prompts nearly four times faster than the Mac, and wins on generation too (32.65 vs 21.81 tokens/s). Agent work, with long tool-call chains and big retrieved contexts, spends most of its time in prefill. That is the workload this machine is built for. For a newer Mac comparison, see Mac Studio M4 Max vs GB10 and Strix Halo.

Can the DGX Spark run a 70B or 200B model?

It can hold one. NVIDIA says one 128 GB Spark handles inference on models up to 200 billion parameters, two clustered units up to 400 billion, and four up to 700 billion. Holding a model and running it at a usable speed are different claims.

LMSYS measured Llama 3.1 70B at FP8 in its DGX Spark review: 803 tokens/s prefill, 2.7 tokens/s decode. Roughly 70 GB of weights read at 273 GB/s gives a ceiling of about 3.9 tokens/s, so 2.7 is about 69% of what the hardware can physically do. Nothing is broken. That is the machine.

Our cost-to-run pages separate the memory requirement from the speed estimate for exactly this reason. Fitting and running fast are different questions, and the answer is often yes to one and no to the other.

How much does the DGX Spark cost in 2026?

$4,699 for NVIDIA's Founders Edition, as of NVIDIA's last announced price. NVIDIA raised it from $3,999 in late February 2026 and said on its developer forum that the change was "due to memory supply constraints". We could not load NVIDIA's store page on 4 October 2026 to confirm stock, so check availability before you plan around it.

A cheaper route is coming. NVIDIA announced on 2 October 2026 that a 64GB DGX Spark will be sold by Acer, ASUS, Dell, Gigabyte, HP and MSI from 23 October. NVIDIA says it supports models up to 100 billion parameters, and that two 64GB units can be clustered to pool 128GB. NVIDIA did not give a price, so we don't print one.

The 64GB model has the same chip, so expect the same 273 GB/s bandwidth story. It lowers the entry price. It doesn't change the generation speed. If you are weighing it against an AMD box, our DGX Spark vs Strix Halo comparison covers that trade.

Is the DGX Spark worth it?

Related guides

• Interactive VRAM calculator: drag the context slider and watch the KV cache fill the memory.
• RTX 5090 vs two RTX 3090s: one fast card or two big ones.
• Cheapest way to serve Llama 70B: rent or buy, and on what.
• What --n-cpu-moe actually does: how to run a MoE model that won't fit.

This is one machine in a bigger category. For the full rundown of unified-memory computers for local AI, from Strix Halo to Mac Studio to DGX Spark, see our best mini-PC for local AI guide.