mixture of experts

14 articles · newest first

Dark close-up photo of a row of blank hardware modules inside a computer, lit warm coral against deep indigo shadow

Ollama has no --n-cpu-moe switch. Here's what it does with MoE experts instead

The GitHub request to keep MoE experts in RAM is still open after 14 months. But since Ollama 0.30, llama.cpp's own fit step can park them there for you, as long as you leave num_gpu alone.

Voltage · 16h ago · 8 min

Low-angle photo of a row of blank matte memory modules on a motherboard in the foreground, with a plain graphics card blurred behind them under indigo and coral light

Best MoE Models for a 12GB or 16GB GPU: The --n-cpu-moe Number and the RAM Each Needs

Seven mixture-of-experts models, from gpt-oss 20B to Qwen3 235B, split into what stays on your card and what moves to system RAM. The card barely changes. The RAM you need goes from 16 GB to 96 GB.

Voltage · Oct 4 · 10 min

Illustration weighing a small and a very large AI model on a balance

Llama 4 Scout vs Maverick: Which to Actually Run Locally

They look like a small/large ladder, but they're built for different machines. Why Scout is the home model, why the smaller one has the bigger context, and the rare case for Maverick.

Tensor · Sep 22 · 7 min

Illustration of a person running a local Llama 4 model built of mixture-of-experts blocks

How to Run Llama 4 Locally in 2026 (The Honest Hardware Guide)

Llama 4 didn't dethrone Qwen3 or DeepSeek — but its MoE design and huge context make it worth understanding. What Scout and Maverick actually are, the memory trap that catches everyone, and what you really need to run it.

Tensor · Sep 22 · 10 min

A consumer graphics card installed in a desktop PC, the kind of hardware llama.cpp's MoE offloading flags are designed to stretch further

your GPU isn't too small for that 235B model — you're just using it wrong

A handful of llama.cpp flags let a 16GB card run models that shouldn't fit, and most people running local AI have never touched them.

Tensor · Aug 25 · 7 min

An Nvidia RTX graphics card, the kind of single consumer GPU Nemotron 3.5 Lightning is designed to run on

Hardware

jensen huang's first-ever X post begged for open AI. three weeks later, nvidia delivered

Nemotron 3.5 Lightning is a 30-billion-parameter model with only about 3 billion active per token — Nvidia says that's why it runs up to 4x faster than comparable open models on a single consumer GPU.

Voltage · Aug 13 · 7 min

rows of server infrastructure representing large-scale cloud computing

Hardware

Qwen3.8-Max open weights (2.4T-A95B): what it takes to run, and the 27B you can actually run

Alibaba put its Max-class model on Hugging Face as Qwen3.8-2.4T-A95B. The download is 2.5 TB in FP8 and the smallest community quant is still 397 GB. Here is the file-size math, the license, and the sibling that fits a 24 GB card.

Voltage · Aug 3 · 8 min

Branching tree limbs forming a network pattern

MoE vs dense AI models explained — why some huge models run faster than small ones

A '30B' model that runs as fast as a 3B one? That's a Mixture-of-Experts model, and it changes the local-AI math. Here's what MoE means and why it matters for what you can run.

Tensor · Aug 3 · 10 min

A data center representing large-scale AI inference

what it really costs to run DeepSeek R1 locally: the full model, the distills, and the honest answer

The full DeepSeek R1 is a 671-billion-parameter monster that needs a small cluster. But its distilled versions run on a single GPU — and knowing the difference saves you a fortune.

Tensor · Jul 28 · 9 min

A server GPU representing large model inference

gpt-oss-120b hardware requirements: about 64 GB of memory, and why a smaller quant won't save you

OpenAI's 117B open model ships its experts in 4-bit MXFP4, so the files are about 63 GB whichever quant you pick. Here is how much VRAM and RAM it needs at 8K to 128K context, which GPUs and machines hold it, and when a 24 GB card plus RAM is enough.

Tensor · Jul 28 · 9 min

An AMD Ryzen CPU seated in a motherboard next to DDR4 memory slots

how to run a 30B model on a 12GB GPU: the one llama.cpp flag that makes it possible

--n-cpu-moe is the setting that lets a modest graphics card punch far above its VRAM. It offloads the parts of a Mixture-of-Experts model you use least to system RAM — and here's exactly how to tune it for your card.

Tensor · Jul 27 · 9 min

Kimi K3 just topped the coding leaderboard. here's the model you can actually run on one GPU that gets you most of the way

The best open coding model in the world needs a datacenter to run. The best coding model you can run on a single 24GB card is a different name entirely — and it's shockingly close for real work.

Tensor · Jul 24 · 9 min

Kimi K3 'Open Frontier Intelligence' announcement page from Moonshot AI

Kimi K3 hardware requirements: the open weights are 1.56 TB, and here's what that means for your GPU

Moonshot's Kimi K3 is open to download. It is also 2.8 trillion parameters in 1.56 TB of MXFP4 files, and the smallest community quant is 509 GB. The real memory math, what vLLM says you need, and what to run at home instead.

Tensor · Jul 23 · 8 min

China just dropped an AI model that beats Claude. good luck running it — 8 GPUs isn't even close

Moonshot's new open-weight model is the biggest ever made, and it's genuinely good. it's also flat-out impossible to run at home — and the reason why is a lesson in how GPUs actually work.

Tensor · Jul 18 · 5 min