moe

3 articles · newest first

Rows of GPUs mounted in an open server rack used for large-scale AI inference

DeepSeek's 'cheap' AI model costs 3 cents a query in the cloud. running it yourself needs $10k of GPUs

the model that made headlines for being nearly free to rent turns out to be one of the most expensive things you could try to self-host

Lena Fischer · 6d ago · 6 min

NVIDIA spent three months optimising the DGX Spark. Token generation got slower.

Same machine, same benchmarks, three months apart. Prefill improved 27%. Generation regressed on every model — and that tells you exactly who should buy one.

Lena Fischer · Jul 21 · 10 min

Close-up of computer hardware used for local AI inference

The MoE offload trick that makes your model 5× faster — and why it probably won't work for you

Adding CPU offload should make inference slower. Sometimes it makes it five times faster. Both are true, and the reason matters more than the trick.

Lena Fischer · Jul 21 · 9 min