Running Local AI Models with Docker Model Runner

Introduction

Docker offers two ways to run AI applications locally: Docker Model Runner (DMR) and traditional containers. DMR is the preferred approach, since it runs models directly on host hardware rather than inside a container, giving it far better access to acceleration hardware such as GPUs.

Why Models Run Outside Containers

Most AI acceleration hardware — GPUs, NPUs, and TPUs — relies on proprietary drivers and SDKs that are extremely difficult for container runtimes to support consistently. While NVIDIA GPUs can be exposed to containers using the NVIDIA Container Toolkit, this setup is complex and limited to CUDA-capable devices. By running models on the host directly, DMR sidesteps this limitation entirely while still integrating with the broader Docker ecosystem.

Architecture

DMR runs as a host process, separate from the Docker Engine, wrapping a pluggable runtime layer — currently llama.cpp — that loads and executes models. It dynamically loads and unloads models based on demand and exposes them through OpenAI-compatible endpoints. Containers can reach these endpoints via the special hostname model-runner.docker.internal, while local or remote applications can reach them over the network on port 12434.

Installing Docker Model Runner

DMR requires Docker Desktop v4.41 or newer. It can be enabled from Docker Desktop's Settings → Features in development page by checking Enable Docker Model Runner. Once enabled, its status can be confirmed with:

docker model status

Pulling and Testing Models

Docker Hub hosts a curated catalog of verified models under the ai namespace. A model can be downloaded just like an image:

docker model pull ai/gemma3:4B-Q4_K_M

Local models can be listed and inspected with docker model ls and docker model inspect. Under the hood, models are stored and distributed as a new type of OCI artifact, meaning they can leverage existing container registries alongside images, SBOMs, and Helm charts.

Models can be tested quickly through an interactive CLI session:

docker model run ai/gemma3:4B-Q4_K_M

or through Docker Desktop's built-in chat interface, which additionally preserves conversational context between questions.

Calling the API Directly

DMR exposes both native model-management endpoints and OpenAI-compatible inference endpoints. A chat request can be sent directly with curl:

curl -s http://localhost:12434/engines/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "ai/gemma3:4B-Q4_K_M", "messages": [{"role": "user", "content": "How long is a day on Mars?"}]}'

Using Docker Model Runner with Compose

DMR integrates with Docker Compose through a dedicated provider extension, allowing a model to be declared as a first-class service dependency:

dmr:
  provider:
    type: model
    options:
      model: ${LLM_MODEL_NAME}

This makes it possible to build a full chatbot stack — a frontend, a backend API, and a model server — where the backend communicates with DMR internally via http://model-runner.docker.internal/engines/v1, all deployed together with a single docker compose up command.

Using Docker Model Runner with Third-Party Apps

Since DMR's inference endpoints are OpenAI-compatible, third-party tools such as Open WebUI can connect to it directly, offering a polished, ChatGPT-like local interface, complete with model switching, conversation history, and customizable system prompts.

Conclusion

Docker Model Runner brings local AI inference into the same workflow developers already use for containers, Compose, and registries, while solving the hardware access problem that limits AI workloads inside containers. For anyone already invested in the Docker ecosystem, it offers a natural, integrated path to running and experimenting with local LLMs.