llama.cpp

22 articles · newest first

Hand-drawn editorial illustration of a hand pressing down the lid of a suitcase packed so full of paper cards that a fan of coral cards spills out, on a near-black background in indigo-violet and off-white

Out of memory at long context? Shrink the KV cache with q8_0 (llama.cpp and Ollama)

The model fits. Then you paste in a long document and the GPU runs out of memory. The culprit is the KV cache, it has a formula you can do on a napkin, and two settings cut it by roughly half or three quarters.

Aliteq Team · Oct 16 · 9 min

Hand-drawn editorial illustration of a laptop with a tall stack of off-white memory blocks on its keyboard, part of the stack lifted by a violet hand pulling a lever along a coral dashed line

MLX vs llama.cpp on a Mac: Which Runtime to Pick, and the Memory Limit That Matters

Four ways to run a local model on Apple silicon: mlx-lm, llama.cpp, Ollama and LM Studio. I read each project's docs to sort out who each is for, which model files they take, and the one sysctl the MLX README tells you to try when a model runs slow.

Aliteq Team · Oct 16 · 9 min

Macro photo of two identical blank black chips side by side on a dark circuit board with gold contact pads, lit in indigo-violet light

Slow Local LLM? Speculative Decoding Explained for llama.cpp and vLLM

A small model guesses the next few words and the big one checks them all at once. Same answers, fewer slow steps. What the original paper measured, what llama.cpp and vLLM actually let you switch on, and when it won't help.

Aliteq Team · Oct 16 · 9 min

Four small black computer towers with warm-lit vents in a row on a dark desk, a taller server unit with a red status light blurred behind them

Ollama vs llama.cpp vs LM Studio vs vLLM: Pick by Who's Using It

One person on one machine, or a team hitting one server? That single question decides it. Then licence, GPU and Mac support, and the API your app talks to, all read from each project's own docs.

Aliteq Team · Oct 4 · 9 min

Low-angle photo of a row of blank matte memory modules on a motherboard in the foreground, with a plain graphics card blurred behind them under indigo and coral light

Best MoE Models for a 12GB or 16GB GPU: The --n-cpu-moe Number and the RAM Each Needs

Seven mixture-of-experts models, from gpt-oss 20B to Qwen3 235B, split into what stays on your card and what moves to system RAM. The card barely changes. The RAM you need goes from 16 GB to 96 GB.

Aliteq Team · Oct 4 · 10 min

Flat illustration of a person at a desk with a small plain mini desktop computer and an AI brain shape split into two pieces above it

gpt-oss-20b on a Mac mini (16GB): yes, but get 24GB

llama.cpp's own guide runs it on a 16GB Mac with one flag. But on a Mac the CPU and GPU share one pool of memory, so the whole model still has to fit, and 16GB leaves almost nothing for macOS. Here's the maths.

Aliteq Team · Sep 25 · 6 min

Flat illustration of a person splitting an AI brain puzzle piece between a small graphics card and a row of memory sticks

gpt-oss-20b on an 8GB or 12GB GPU: yes, with --n-cpu-moe

Everyone says 16GB minimum. That's only true if the whole model has to live on the GPU. llama.cpp's own guide runs it on an 8GB RTX 2060. Here are the commands, and the starting values for your card.

Aliteq Team · Sep 25 · 7 min

Flat illustration of a person compressing a large cube into a small box, on a teal background

How to Quantize a Local LLM in 2026 (GGUF Q4 vs Q6 vs Q8 — and What You Lose)

Quantization is the trick that lets a 16GB card run models it has no business running. Here's how GGUF quant levels work, which one I default to, the memory math, and the honest quality trade — so you get the most model your VRAM can hold.

Aliteq Team · Sep 22 · 9 min

A consumer graphics card installed in a desktop PC, the kind of hardware llama.cpp's MoE offloading flags are designed to stretch further

your GPU isn't too small for that 235B model — you're just using it wrong

A handful of llama.cpp flags let a 16GB card run models that shouldn't fit, and most people running local AI have never touched them.

Aliteq Team · Aug 25 · 7 min

An AMD Radeon AI PRO workstation graphics card

Hardware

Meta's new AI model runs at home now — one $1,299 AMD card almost keeps up with a $4,700 Nvidia one

AMD's own numbers put its workstation GPU at 53 tokens a second on Meta's new 30B model. Nvidia's flagship does more — for over three times the price.

Aliteq Team · Aug 11 · 6 min

Nvidia Tesla V100 data-center GPU accelerator card

Hardware

Tesla V100 for local AI: a £200 server GPU in a gaming PC, and whether it's worth copying

One builder added a used V100 SXM2 and a bare adapter to an RTX 4080 PC and got 32GB of VRAM. The speed is real. So are the noise, the legacy driver and NVIDIA's dropped Volta support in CUDA 13.

Aliteq Team · Jul 31 · 8 min

Ollama vs LM Studio: which one should you actually use to run AI locally?

Both are free, both run the same open models, and since mid-2025 both have a desktop app. The real split now: LM Studio if you want to browse, tweak and chat; Ollama if you want a model that other apps can call. Here's the full comparison, including speed.

Aliteq Team · Jul 30 · 10 min

Code representing a local AI inference engine

vLLM, Ollama or llama.cpp? the three ways to run local AI, and which one is actually for you

They all run local models, but they're built for completely different people. One's for easy desktop use, one's for flexibility, one's for serving at scale. Here's how to pick without wasting a weekend.

Aliteq Team · Jul 28 · 9 min

The ExLlamaV2 GitHub repository page showing its MIT license and description

is ExLlama free? yes — and it's the fastest way to run a local model, with one big condition

ExLlamaV2 is open-source and free (MIT), and on a GPU it's roughly twice as fast as llama.cpp. The catch: your model has to fit entirely in VRAM. Here's exactly when to use each one.

Aliteq Team · Jul 27 · 8 min

An AMD Ryzen CPU seated in a motherboard next to DDR4 memory slots

how to run a 30B model on a 12GB GPU: the one llama.cpp flag that makes it possible

--n-cpu-moe is the setting that lets a modest graphics card punch far above its VRAM. It offloads the parts of a Mixture-of-Experts model you use least to system RAM — and here's exactly how to tune it for your card.

Aliteq Team · Jul 27 · 9 min

the RTX 5080 is newer, faster on paper, and the wrong pick for local AI. here's the 8GB reason why

Blackwell versus the old flagship, and for once the newer card loses. The 5080's 16GB caps what you can run; the 4090's 24GB doesn't. For local models, capacity beats the spec sheet, and you can rent either card by the hour to check.

Aliteq Team · Jul 22 · 9 min

everyone says AMD means fighting ROCm for local AI. on llama.cpp, the thing you're told to install is the slower one

The RX 7900 XTX gives you 24GB cheaper than a 3090. The catch isn't what you've heard — a llama.cpp scoreboard and an open bug report suggest its Vulkan backend beats ROCm for inference, no painful install required.

Aliteq Team · Jul 22 · 9 min

NVIDIA GeForce RTX 4090 Founders Edition graphics card

The 4090 is 34% faster for local AI. Your electricity meter can't tell them apart

Published llama.cpp numbers say the 4090 wins every speed test and costs far more, while energy per generated token is a dead heat. Here's who each card is for, what 24GB runs today, and how to try both by the hour.

Aliteq Team · Jul 22 · 9 min

A multi-GPU workstation built for running large language models locally

The cheapest way to run Llama 70B isn't the one you're about to build

"Cheapest" flips completely on how often you run it and how many people hit it at once — and in Europe, the electricity bill alone can cost most of what renting does.

Aliteq Team · Jul 21 · 10 min

Close-up of computer hardware used for local AI inference

The MoE offload trick that makes your model 5× faster — and why it probably won't work for you

Adding CPU offload should make inference slower. Sometimes it makes it five times faster. Both are true, and the reason matters more than the trick.

Aliteq Team · Jul 21 · 9 min

Two graphics cards installed in a desktop PC

Hardware

Two RTX 3090s or one RTX 5090 for local LLMs? The answer isn't the one everyone gives

Two 3090s give you 48GB, but not double the speed for chat, because llama.cpp's default multi-GPU mode makes the cards take turns. What the benchmarks say, rechecked in October 2026.

Aliteq Team · Jul 21 · 11 min

Two graphics cards side by side, illustrating GPU VRAM tier comparison

everyone asks which local AI model is 'best.' wrong question — your VRAM already picked it

Llama 4 Scout, Qwen3.6-27B, GPT-OSS-20B, Phi-4-14B — four totally different 'best' models, and which one is yours comes down to a single number: how much VRAM your GPU actually has.

Aliteq Team · Jul 18 · 6 min