inference engine

2 articles · newest first

AI

vLLM, Ollama or llama.cpp? the three ways to run local AI, and which one is actually for you

They all run local models, but they're built for completely different people. One's for easy desktop use, one's for flexibility, one's for serving at scale. Here's how to pick without wasting a weekend.

Lena Fischer · Jul 28 · 9 min

AI

is ExLlama free? yes — and it's the fastest way to run a local model, with one big condition

ExLlamaV2 is open-source and free (MIT), and on a GPU it's roughly twice as fast as llama.cpp. The catch: your model has to fit entirely in VRAM. Here's exactly when to use each one.

Lena Fischer · Jul 27 · 8 min