local llm

40 articles · newest first

Hand-drawn editorial illustration of a laptop with a tall stack of off-white memory blocks on its keyboard, part of the stack lifted by a violet hand pulling a lever along a coral dashed line

MLX vs llama.cpp on a Mac: Which Runtime to Pick, and the Memory Limit That Matters

Four ways to run a local model on Apple silicon: mlx-lm, llama.cpp, Ollama and LM Studio. I read each project's docs to sort out who each is for, which model files they take, and the one sysctl the MLX README tells you to try when a model runs slow.

Aliteq Team · Oct 16 · 9 min

Photo of a small plain black cube computer on a dark wooden desk at night, a thin cable trailing off, indigo-violet light behind it and a coral desk lamp in soft focus

Got a Strix Halo or a 128GB Mac? The Best Local LLMs for 128GB of Unified Memory

128GB does not mean 128GB for the model. Plan on 96GB, and the box becomes a home for 100B-plus Mixture-of-Experts models. Five picks by job, with the math shown.

Aliteq Team · Oct 16 · 11 min

Macro photo of two unbranded graphics boards side by side on a dark desk, one bare with exposed memory chips and one under a large pin heatsink, violet and coral light

Hardware

RTX 5080 vs RTX 5090 for Local AI: The 16GB Gap Decides Which Models You Can Run

At NVIDIA's own list prices, a gigabyte of VRAM costs the same $62 on both cards. So the real question is which models you want. We ran 13 popular ones through our VRAM math, then priced renting each card by the hour.

Aliteq Team · Oct 16 · 11 min

A laptop with a blank dark screen on a rolling cart in a long, dim data-center aisle lined with black server racks and a few green status lights

NVIDIA's Free Hosted Models at build.nvidia.com: What the Free API Key Really Allows

Voltage read NVIDIA's FAQ, its API Trial Terms and the forum replies. The key is free, the credits are gone, the rate limit isn't printed, and customers are off limits.

Aliteq Team · Oct 16 · 8 min

Macro photo of two identical blank black chips side by side on a dark circuit board with gold contact pads, lit in indigo-violet light

Slow Local LLM? Speculative Decoding Explained for llama.cpp and vLLM

A small model guesses the next few words and the big one checks them all at once. Same answers, fewer slow steps. What the original paper measured, what llama.cpp and vLLM actually let you switch on, and when it won't help.

Aliteq Team · Oct 16 · 9 min

Hand-drawn editorial illustration of a hand pressing down the lid of a suitcase packed so full of paper cards that a fan of coral cards spills out, on a near-black background in indigo-violet and off-white

Out of memory at long context? Shrink the KV cache with q8_0 (llama.cpp and Ollama)

The model fits. Then you paste in a long document and the GPU runs out of memory. The culprit is the KV cache, it has a formula you can do on a napkin, and two settings cut it by roughly half or three quarters.

Aliteq Team · Oct 16 · 9 min

Dark photo of an open compact computer case on a wooden desk, two dark blank modules and two coral modules standing inside, with a loose circuit board beside it

LM Studio's 'force MoE weights onto CPU' setting: fit a 30B model on an 8GB GPU

It started as a toggle and became a layer slider in LM Studio 0.4.0. Where it lives, what it moves, the number to set on an 8, 12 or 16GB card, and why Mac MLX models don't have it.

Aliteq Team · Oct 16 · 8 min

Flat vector illustration of a woman dropping a paper document onto a large map of dots, with a coral arrow pointing it to the cluster of similar coral dots, on an indigo-violet background

Embedding Models Explained: Dimensions, Storage Cost and the Re-Embedding Bill (2026)

Your RAG search lives or dies on one small model most people never pick on purpose. What an embedding is, why 3,072 dimensions costs eight times the storage of 384, how to read the MTEB leaderboard without getting fooled, and what switching models really costs.

Aliteq Team · Oct 16 · 11 min

Macro photo of the silver aluminum heatsink fins of a graphics card above a black fan, lit in violet and warm coral light

Hardware

Got an Intel Arc Card? Here's Every Local AI Model It Fits (2026)

Intel sells six Arc desktop cards today, from the 10GB B570 to the 32GB Arc Pro B70. I matched each one against 12 popular local models with our memory math, and found the cheapest 16GB Arc hiding in the workstation aisle.

Aliteq Team · Oct 11 · 12 min

Dark close-up photo of a row of blank hardware modules inside a computer, lit warm coral against deep indigo shadow

Ollama has no --n-cpu-moe switch. Here's what it does with MoE experts instead

The GitHub request to keep MoE experts in RAM is still open after 14 months. But since Ollama 0.30, llama.cpp's own fit step can park them there for you, as long as you leave num_gpu alone.

Aliteq Team · Oct 11 · 8 min

Hand-drawn illustration of a small bonsai tree in a cream pot with coral pruning shears, in front of the faint violet outline of a giant tree on a near-black background

Bonsai 2 27B: How PrismML Shrinks a 27B Model to 5.9 GB

A 27B model that fits in about 6 GB sounds like a trick. It is ternary weights, a shared scale and a rotation. Here is how it works, what PrismML's own tests show, and where the loss hides.

Aliteq Team · Oct 10 · 8 min

Macro photo of rows of identical blank black memory chips on a circuit board, their pins lit in indigo-violet against a near-black background

Qwen3.8 27B Hardware Requirements: VRAM by Quant, and Why Context Costs So Little

A 24 GB card runs Qwen3.8 27B at Q4 or Q5. Q8 needs 32 GB. The surprise is the context cache: 128K tokens costs about 8 GB, a quarter of what an older 32B model charges.

Aliteq Team · Oct 7 · 9 min

Photo of two small unbranded desktop computers side by side on a dark wooden desk, a black boxy mini PC on the left and a low silver aluminum box on the right, lit violet and coral from behind

Hardware

Ryzen AI Max+ 395 vs Mac Mini and Mac Studio for Local AI: Same 96GB, $1,650 Apart

A 128GB Strix Halo mini PC and a 128GB Mac Studio give an AI model about the same 96GB to work with. One costs $3,449 at Framework's US store, the other $5,099 at Apple's. Here is what the extra $1,650 buys, and what it doesn't.

Aliteq Team · Oct 7 · 13 min

Macro photo of a row of plain black memory chips on a circuit board, a streak of indigo-violet and coral light on the traces, near-black background

Got an RTX 5090? The Best Local LLMs for 32GB of VRAM in 2026

32GB does not unlock 70B models. It buys better quants and a much longer memory for the 27B to 35B class. Five picks that fit a 5090, with the math shown.

Aliteq Team · Oct 5 · 10 min

Four small black computer towers with warm-lit vents in a row on a dark desk, a taller server unit with a red status light blurred behind them

Ollama vs llama.cpp vs LM Studio vs vLLM: Pick by Who's Using It

One person on one machine, or a team hitting one server? That single question decides it. Then licence, GPU and Mac support, and the API your app talks to, all read from each project's own docs.

Aliteq Team · Oct 4 · 9 min

Low-angle photo of a row of blank matte memory modules on a motherboard in the foreground, with a plain graphics card blurred behind them under indigo and coral light

Best MoE Models for a 12GB or 16GB GPU: The --n-cpu-moe Number and the RAM Each Needs

Seven mixture-of-experts models, from gpt-oss 20B to Qwen3 235B, split into what stays on your card and what moves to system RAM. The card barely changes. The RAM you need goes from 16 GB to 96 GB.

Aliteq Team · Oct 4 · 10 min

Low-angle macro photo of the front edge of a small silver aluminum desktop computer on a dark desk, a tiny green status light glowing, violet and coral light behind it

Hardware

Before You Buy a Mac Mini for Local AI, Read This (It’ll Save You $1,800)

I mapped every Mac mini and Mac Studio Apple sells today, all 17 configs, against 13 popular local AI models. If your biggest model is 27B-class, the math says a $1,299 Mac mini holds what a $3,099 Mac Studio holds.

Aliteq Team · Oct 4 · 16 min

A gaming laptop and a desktop PC tower side by side on a dark desk with RGB glow

Hardware

Gaming Laptop vs Desktop for Local AI: The Honest Answer (2026)

For local AI the decider is VRAM — and that's exactly where laptops are capped. Why a desktop (or a unified-memory mini-PC) usually wins, when a gaming laptop is still right, and how to choose.

Aliteq Team · Sep 22 · 8 min

Illustration of choosing between two AI model badges over a laptop

Llama 4 vs Qwen3 for Local Use: The Honest Comparison

The quiet truth: for most home users, Qwen3 is the better default than Llama 4 Scout. Where each actually wins, the LMArena controversy, and the specific cases where Scout is worth it.

Aliteq Team · Sep 22 · 7 min

Illustration of compressing a large block of AI experts into a smaller box

Llama 4 Quantized: GGUF Sizes & Memory, Explained Honestly

Quantization is the only reason Scout runs at home — but MoE means you shrink all 109B, not the 17B active. The honest GGUF memory ladder, the Q4_K_M sweet spot, and how low you can safely go.

Aliteq Team · Sep 22 · 8 min

Illustration weighing a small and a very large AI model on a balance

Llama 4 Scout vs Maverick: Which to Actually Run Locally

They look like a small/large ladder, but they're built for different machines. Why Scout is the home model, why the smaller one has the bigger context, and the rare case for Maverick.

Aliteq Team · Sep 22 · 7 min

A high-end consumer graphics card on a dark desk with soft RGB glow

Hardware

The Best GPU for Llama 4 Scout in 2026 (VRAM-First, Honest)

Scout is mixture-of-experts, so you're buying for 109B in memory, not 17B active. The honest GPU ladder — why a 32GB card is the sweet spot, when 24GB works, and when unified memory beats them all.

Aliteq Team · Sep 22 · 8 min

Illustration of a person running a local Llama 4 model built of mixture-of-experts blocks

How to Run Llama 4 Locally in 2026 (The Honest Hardware Guide)

Llama 4 didn't dethrone Qwen3 or DeepSeek — but its MoE design and huge context make it worth understanding. What Scout and Maverick actually are, the memory trap that catches everyone, and what you really need to run it.

Aliteq Team · Sep 22 · 10 min

Flat illustration of a person compressing a large cube into a small box, on a teal background

How to Quantize a Local LLM in 2026 (GGUF Q4 vs Q6 vs Q8 — and What You Lose)

Quantization is the trick that lets a 16GB card run models it has no business running. Here's how GGUF quant levels work, which one I default to, the memory math, and the honest quality trade — so you get the most model your VRAM can hold.

Aliteq Team · Sep 22 · 9 min

An Intel Arc B580 Limited Edition graphics card

Hardware

Is the Intel Arc B580 a Smart Budget GPU for Local AI in 2026?

12GB of VRAM at ~$249, and the one card the 2026 price surge barely touched. Superb budget hardware for local LLMs — if you can live with Intel's software tax. My honest take on who should buy one.

Aliteq Team · Sep 22 · 8 min

Flat illustration of a person at a PC running an AI assistant with a graphics card in the case, on a teal background

The Best GPU for Running Qwen3 Locally in 2026 (by Model Size)

Qwen3 isn't one model — it's a family from 4B to 235B, and the right GPU depends entirely on which you run. My VRAM-first map of every variant, the MoE trap that trips people up, and the used card I'd hand most people.

Aliteq Team · Sep 22 · 8 min

A developer's laptop screen showing a terminal window running a local AI model

I compared Ollama, vLLM and LM Studio — only one of them survives a second user

at one person typing, all three are basically tied. add a second and the gap turns into a cliff

Aliteq Team · Aug 22 · 7 min

AMD's Radeon AI PRO R9700 workstation graphics card

AMD just proved you don't need a graphics card to run a serious AI model at home

Day-zero support for the new Qwen3.8 27B just landed on two AMD boxes — one needs a graphics card, one doesn't — and the tokens-per-second numbers are real.

Aliteq Team · Aug 18 · 6 min

A person using an AI chatbot app on an Android phone

before you drop $800 on a GPU for AI, know what your phone can already do for free

Android Police ran a real local model on a real phone as a daily driver. it's not a GPU replacement — but it covers more than the GPU-shopping conversation usually admits.

Aliteq Team · Aug 16 · 6 min

A smartphone screen displaying a command-line terminal interface

Software

someone let their phone's AI install its own software. it actually worked — with a catch

An Android app called RikkaHub Agent turns a phone into a local-LLM-powered agent that can write code, run Linux commands, and compile software on its own — as long as you keep saying yes.

Aliteq Team · Aug 9 · 8 min

A small ESP32-S3 microcontroller development board on a workbench, the $8 chip now capable of running a 28.9-million-parameter AI language model.

Hardware

someone got a real ai talking on a chip that costs less than your coffee

A hobbyist just proved a genuine language model can run entirely on an $8 microcontroller — no cloud, no GPU, no internet connection required.

Aliteq Team · Aug 6 · 6 min

A laptop screen showing a local AI language model running in a terminal window

three free AI models all claim a 128K memory — only one actually holds onto it

Phi-4 Mini, Gemma 3, and Llama 3.2 all advertise roughly the same 128K context window. Benchmark data says that number means something very different for each one.

Aliteq Team · Aug 4 · 7 min

A small humanoid robot sitting on a red wooden bench, reading from a tablet

I priced out every way to run a 70B AI model at home — the cheapest one isn't what you'd guess

Four real paths to running a 70B-parameter model on your own hardware or by the hour — priced out with actual 2026 numbers.

Aliteq Team · Aug 4 · 8 min

Illustration of a person holding a mini-PC whose large memory pool fits a big AI model

Hardware

The Best Mini-PC for Local AI in 2026 (Unified Memory, Honestly)

A book-sized box with 128GB of unified memory can run models no single consumer GPU can hold. The honest hub for the whole category — Strix Halo, Mac Studio and DGX Spark — and when a GPU still wins.

Aliteq Team · Aug 4 · 10 min

How to run uncensored AI models locally, and what "abliterated" really means

Uncensored local models answer without the refusals and lectures of cloud chatbots. How the two kinds are made, how to run one in Ollama on an 8 GB card, and the rules that still apply after the filter is gone.

Aliteq Team · Aug 3 · 8 min

how to run Llama locally — Meta's famous open model, on your own PC in one command

Llama is the open model that kicked off the whole local-AI movement, and running it yourself is genuinely one command. Here's how, and which size fits your GPU.

Aliteq Team · Aug 3 · 10 min

how to run Mistral locally — the lean European open model, on your own PC

Mistral's models are famously efficient — they punch above their size, so you get a lot on modest hardware. Here's how to run Mistral locally in one command, by size.

Aliteq Team · Aug 3 · 10 min

A friendly 3D AI robot working at a laptop

how to run Qwen3 locally in 2026 — the best open model, on your own PC, in one command

Qwen3 is arguably the best open model you can run at home right now. Getting it running is genuinely one command. Here's exactly how, and which size fits your GPU.

Aliteq Team · Aug 1 · 10 min

A small ESP32-S3 microcontroller development board wired up on a workbench

someone crammed a working AI language model onto a $10 chip. it writes actual stories

No cloud, no Wi-Fi, no GPU — just a cheap microcontroller quietly telling itself bedtime stories, one token at a time.

Aliteq Team · Jul 31 · 6 min

A graphics card showing VRAM memory chips

how much VRAM do you need to run AI models? The simple rule, by model size

VRAM is the single number that decides which AI models you can run. Here's the simple rule of thumb, a size-by-size table, and a calculator to check any model against your card.

Aliteq Team · Jul 30 · 9 min