vllm

3 articles · newest first

A server rack with network cabling in green light

how to turn your local AI into an API — a private, free drop-in for the OpenAI API

Point your apps, scripts, and coding tools at a model on your own machine instead of a paid cloud API. It's an OpenAI-compatible endpoint, it's free, and it's one command to start.

Lena Fischer · Aug 1 · 10 min

Code representing a local AI inference engine

vLLM, Ollama or llama.cpp? the three ways to run local AI, and which one is actually for you

They all run local models, but they're built for completely different people. One's for easy desktop use, one's for flexibility, one's for serving at scale. Here's how to pick without wasting a weekend.

Lena Fischer · Jul 28 · 9 min

A multi-GPU workstation built for running large language models locally

The cheapest way to run Llama 70B isn't the one you're about to build

"Cheapest" flips completely on how often you run it and how many people hit it at once — and in Europe, the electricity bill alone can cost most of what renting does.

Lena Fischer · Jul 21 · 10 min