mixture-of-experts

8 articles · newest first

An Nvidia RTX graphics card, the kind of single consumer GPU Nemotron 3.5 Lightning is designed to run on

jensen huang's first-ever X post begged for open AI. three weeks later, nvidia delivered

Nemotron 3.5 Lightning is a 30-billion-parameter model with only about 3 billion active per token — Nvidia says that's why it runs up to 4x faster than comparable open models on a single consumer GPU.

Ravi Malhotra · 2d ago · 7 min

rows of server infrastructure representing large-scale cloud computing

Hardware

Alibaba just open-sourced a 2.4 trillion parameter AI model — good luck ever running it yourself

it beats Claude on some benchmarks and rivals GPT on others, and it's free to download next week. here's the hardware math that ruins the fun.

Ravi Malhotra · Aug 3 · 7 min

Branching tree limbs forming a network pattern

MoE vs dense AI models explained — why some huge models run faster than small ones

A '30B' model that runs as fast as a 3B one? That's a Mixture-of-Experts model, and it changes the local-AI math. Here's what MoE means and why it matters for what you can run.

Lena Fischer · Aug 3 · 10 min

A data center representing large-scale AI inference

what it really costs to run DeepSeek R1 locally: the full model, the distills, and the honest answer

The full DeepSeek R1 is a 671-billion-parameter monster that needs a small cluster. But its distilled versions run on a single GPU — and knowing the difference saves you a fortune.

Lena Fischer · Jul 28 · 9 min

A server GPU representing large model inference

gpt-oss-120b is a 120-billion-parameter model you can actually run at home — if you know the MoE trick

OpenAI's open-weight 120B model sounds impossible for consumer hardware. But it's a Mixture-of-Experts model, and that changes the math completely. Here's what it really takes to run it.

Lena Fischer · Jul 28 · 9 min

An AMD Ryzen CPU seated in a motherboard next to DDR4 memory slots

how to run a 30B model on a 12GB GPU: the one llama.cpp flag that makes it possible

--n-cpu-moe is the setting that lets a modest graphics card punch far above its VRAM. It offloads the parts of a Mixture-of-Experts model you use least to system RAM — and here's exactly how to tune it for your card.

Lena Fischer · Jul 27 · 9 min

Kimi K3 just topped the coding leaderboard. here's the model you can actually run on one GPU that gets you most of the way

The best open coding model in the world needs a datacenter to run. The best coding model you can run on a single 24GB card is a different name entirely — and it's shockingly close for real work.

Lena Fischer · Jul 24 · 9 min

Kimi K3 'Open Frontier Intelligence' announcement page from Moonshot AI

the biggest open AI model ever drops this week. you can download all 1.4 terabytes — you just can't run it

Kimi K3's open weights land July 27. It's genuinely great and genuinely free. It also needs about 1.4TB of memory to load, which means the download is the easy part. Here's the real hardware math — and the far smaller model that gets you most of the way.

Lena Fischer · Jul 23 · 9 min