gpu offloading

1 article · newest first

A consumer graphics card installed in a desktop PC, the kind of hardware llama.cpp's MoE offloading flags are designed to stretch further

your GPU isn't too small for that 235B model — you're just using it wrong

A handful of llama.cpp flags let a 16GB card run models that shouldn't fit, and most people running local AI have never touched them.

Aliteq Team · Aug 25 · 7 min