cpu offload

4 articles · newest first

Dark photo of an open compact computer case on a wooden desk, two dark blank modules and two coral modules standing inside, with a loose circuit board beside it

LM Studio's 'force MoE weights onto CPU' setting: fit a 30B model on an 8GB GPU

It started as a toggle and became a layer slider in LM Studio 0.4.0. Where it lives, what it moves, the number to set on an 8, 12 or 16GB card, and why Mac MLX models don't have it.

Aliteq Team · 11h ago · 8 min

Dark close-up photo of a row of blank hardware modules inside a computer, lit warm coral against deep indigo shadow

Ollama has no --n-cpu-moe switch. Here's what it does with MoE experts instead

The GitHub request to keep MoE experts in RAM is still open after 14 months. But since Ollama 0.30, llama.cpp's own fit step can park them there for you, as long as you leave num_gpu alone.

Aliteq Team · 4d ago · 8 min

how much system RAM do you actually need for local AI? more than you think, if you offload

Everyone obsesses over VRAM, but system RAM matters too — especially the moment you try to run a model bigger than your GPU. Here's the real number for every kind of local-AI setup.

Aliteq Team · Jul 28 · 9 min

An AMD Ryzen CPU seated in a motherboard next to DDR4 memory slots

how to run a 30B model on a 12GB GPU: the one llama.cpp flag that makes it possible

--n-cpu-moe is the setting that lets a modest graphics card punch far above its VRAM. It offloads the parts of a Mixture-of-Experts model you use least to system RAM — and here's exactly how to tune it for your card.

Aliteq Team · Jul 27 · 9 min