how to run local AI in Docker — a clean, portable, reproducible setup in minutes

Why run local AI in Docker?

Because it keeps everything clean, portable, and reproducible. Running your local AI stack — Ollama plus a web UI like Open WebUI — in Docker containers means it's isolated from your system, easy to start and stop, trivial to reset if something breaks, and simple to move to another machine or a server. Instead of installing a pile of dependencies directly on your OS, you run a couple of containers with GPU access and you're done. It's the tidy, developer-friendly way to run local AI, and with NVIDIA GPU support in Docker, you get full performance. Here's how.

The setup

The classic local-AI-in-Docker stack is Ollama (serving models) plus Open WebUI (a browser chat interface), which together give you a private, self-hosted ChatGPT. You need Docker installed, and for GPU acceleration, the NVIDIA Container Toolkit (a quick install that lets containers access your NVIDIA GPU). Then a small docker-compose.yml brings up both services. Here's a minimal starting point:

# docker-compose.yml — Ollama + Open WebUI with GPU
services:
  ollama:
    image: ollama/ollama
    deploy: { resources: { reservations: { devices: [{driver: nvidia, count: all, capabilities: [gpu]}] } } }
    volumes: ["ollama:/root/.ollama"]
  webui:
    image: ghcr.io/open-webui/open-webui:main
    ports: ["3000:8080"]
    environment: ["OLLAMA_BASE_URL=http://ollama:11434"]
    volumes: ["webui:/app/backend/data"]
volumes: { ollama: {}, webui: {} }

# then run:  docker compose up -d   →  open http://localhost:3000

Run docker compose up -d, open http://localhost:3000, and you have a private local-AI chat with a nice UI — pull models from the interface, and everything (models, chats, document RAG) stays in Docker volumes on your machine.

When Docker is (and isn't) the right choice

Docker is a great fit if you're a developer, want a reproducible setup you can version and share, plan to run local AI on a server or headless box, or like keeping your host OS clean. It's also ideal for running a whole stack — model server, web UI, vector database, and app — together in one reproducible bundle, which is exactly how you'd deploy a RAG or agent system. It's overkill if you just want to chat with a model on your desktop — in that case, installing Ollama or LM Studio directly is simpler and just as fast. So: use Docker when you value isolation, portability, and reproducibility (developers, servers, multi-service stacks); skip it for a simple personal chatbot. Either way, GPU performance inside Docker is full-speed with the NVIDIA toolkit, so you lose nothing but the small setup effort. Match the model to your VRAM and you're running a clean, containerized local-AI stack.

Docker gives you a clean, portable local-AI stack — Ollama plus a web UI in minutes, full GPU speed. It's ideal for RAG and agent stacks; for simple desktop chat, Ollama or LM Studio direct is easier. Size the model to your GPU. Source: Ollama and Open WebUI.