mixture of experts

4 articles · newest first

A consumer graphics card installed in a desktop PC, the kind of hardware llama.cpp's MoE offloading flags are designed to stretch further

your GPU isn't too small for that 235B model — you're just using it wrong

A handful of llama.cpp flags let a 16GB card run models that shouldn't fit, and most people running local AI have never touched them.

Lena Fischer · Aug 25 · 7 min

An Nvidia RTX graphics card, the kind of single consumer GPU Nemotron 3.5 Lightning is designed to run on

Hardware

jensen huang's first-ever X post begged for open AI. three weeks later, nvidia delivered

Nemotron 3.5 Lightning is a 30-billion-parameter model with only about 3 billion active per token — Nvidia says that's why it runs up to 4x faster than comparable open models on a single consumer GPU.

Ravi Malhotra · Aug 13 · 7 min

rows of server infrastructure representing large-scale cloud computing

Hardware

Alibaba just open-sourced a 2.4 trillion parameter AI model — good luck ever running it yourself

it beats Claude on some benchmarks and rivals GPT on others, and it's free to download next week. here's the hardware math that ruins the fun.

Ravi Malhotra · Aug 3 · 7 min

Branching tree limbs forming a network pattern

MoE vs dense AI models explained — why some huge models run faster than small ones

A '30B' model that runs as fast as a 3B one? That's a Mixture-of-Experts model, and it changes the local-AI math. Here's what MoE means and why it matters for what you can run.

Lena Fischer · Aug 3 · 10 min