gguf

3 articles · newest first

Rows of GPUs mounted in an open server rack used for large-scale AI inference

DeepSeek's 'cheap' AI model costs 3 cents a query in the cloud. running it yourself needs $10k of GPUs

the model that made headlines for being nearly free to rent turns out to be one of the most expensive things you could try to self-host

Lena Fischer · Aug 4 · 6 min

Code representing a local AI inference engine

vLLM, Ollama or llama.cpp? the three ways to run local AI, and which one is actually for you

They all run local models, but they're built for completely different people. One's for easy desktop use, one's for flexibility, one's for serving at scale. Here's how to pick without wasting a weekend.

Lena Fischer · Jul 28 · 9 min

The ExLlamaV2 GitHub repository page showing its MIT license and description

is ExLlama free? yes — and it's the fastest way to run a local model, with one big condition

ExLlamaV2 is open-source and free (MIT), and on a GPU it's roughly twice as fast as llama.cpp. The catch: your model has to fit entirely in VRAM. Here's exactly when to use each one.

Lena Fischer · Jul 27 · 8 min