moe

7 articles · newest first

Isometric diorama of a huge grid of expert blocks overflowing off a single graphics card into several system-RAM sticks, near-black background, indigo-violet with a coral accent

Qwen3-Next 80B-A3B locally: an 80B at home, if you can hold it

Qwen3-Next 80B-A3B activates only 3B parameters per token — so it's fast. But 80B of weights still have to live somewhere. The honest VRAM math, the offload trick, and what it really takes.

Tensor · Sep 26 · 7 min

Editorial infographic of a mixture-of-experts model: a few active expert blocks on a GPU and the rest resting in system memory, near-black background, indigo-violet with a coral accent

Gemma 4 26B-A4B locally: the Gemma that runs fast on 16GB

Gemma 4's mixture-of-experts model holds about 26B parameters but activates only 3.8B — so it's quick, and it fits a 16GB card at Q4. The VRAM math, the GPU, and the run command.

Tensor · Sep 26 · 7 min

Isometric diorama of a grid of many small expert blocks with only a few lit up feeding a token, near-black background, indigo-violet with a coral accent

Qwen3.6 35B-A3B locally: a 35B that thinks like a 3B

Qwen3.6's default is a mixture-of-experts model: 35B on disk, 3B active per token. That makes it fast — but the VRAM math has a twist worth understanding before you buy a card.

Tensor · Sep 26 · 7 min

An Nvidia RTX graphics card, the kind of single consumer GPU Nemotron 3.5 Lightning is designed to run on

Hardware

jensen huang's first-ever X post begged for open AI. three weeks later, nvidia delivered

Nemotron 3.5 Lightning is a 30-billion-parameter model with only about 3 billion active per token — Nvidia says that's why it runs up to 4x faster than comparable open models on a single consumer GPU.

Voltage · Aug 13 · 7 min

Rows of GPUs mounted in an open server rack used for large-scale AI inference

DeepSeek's 'cheap' AI model costs 3 cents a query in the cloud. running it yourself needs $10k of GPUs

the model that made headlines for being nearly free to rent turns out to be one of the most expensive things you could try to self-host

Tensor · Aug 4 · 6 min

DGX Spark for local LLMs: fast at reading prompts, slow at writing replies

Same machine, same benchmarks, three months apart: prefill improved 27%, generation got slower on every model. Tokens per second for seven models, the $4,699 price, and who should actually buy one.

Tensor · Jul 21 · 11 min

Close-up of computer hardware used for local AI inference

The MoE offload trick that makes your model 5× faster — and why it probably won't work for you

Adding CPU offload should make inference slower. Sometimes it makes it five times faster. Both are true, and the reason matters more than the trick.

Tensor · Jul 21 · 9 min