← All guides AI Release
RU Subscribe to @ai_release1

How to fix Docker errors when running a local LLM: GPU, memory and port forwarding

Updated: 23.10.2026 · AI Release · @ai_release1
How to fix Docker errors when running a local LLM: GPU, memory and port forwarding

TL;DR

- Docker Engine on Linux, or Docker Desktop with the WSL2 backend on Windows.

- An NVIDIA GPU with a working driver (nvidia-smi runs on the host).

- NVIDIA Container Toolkit on Linux (the package that lets Docker pass the GPU into a container).

- The official Ollama image ollama/ollama (confirm the current tag in the registry before you start).

- ~10 GB of free disk space for the image plus model weights; exact model sizes are in the RAM/VRAM sizing table.

Requirements and setup

Illustration: Requirements and setup
Illustration: Requirements and setup

Docker runs your LLM in an isolated box with no direct access to the GPU, the host filesystem or the host network unless you grant it explicitly. Almost every "it doesn't start" report comes from one of those three grants missing.

Run this checklist before touching the LLM image:

nvidia-smi
docker version
docker info | grep -i -E "runtimes|default runtime"
df -h

Example: on a Linux workstation docker info | grep -i runtimes returns a line containing nvidia; on a machine without the toolkit the output contains only runc, and every --gpus command will fail later with the error described in Troubleshooting.

Why this works: the toolkit writes a runtime entry into /etc/docker/daemon.json, so Docker knows *how* to mount the GPU device nodes when you ask for them.

If you hit unrelated Docker Desktop/WSL2 breakage after a Windows feature update, see Windows 11 26H2: common issues after update.

Step 1. Prove that a container can actually see the GPU

What to do: run a throwaway container that only calls nvidia-smi.

docker run --rm --gpus=all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

--gpus=all asks Docker for every GPU on the host. --rm deletes the container after it exits. If that tag is not published for your driver, pick another -base-ubuntu22.04 tag — check the documentation/tag list rather than guessing.

Why: this separates two problems that look identical to a beginner — "the driver is broken" and "Docker isn't allowed to use the driver". If this command works, your LLM container has no excuse.

How to check: you must see the same GPU table you saw on the host, rendered inside the container. Anything else — a driver error, an empty table, or the could not select device driver message — means stop here and fix the runtime before pulling the model image.

Example: expected screen contents include a header row like GPU Name Persistence-M Bus-Id Disp.A Memory-Usage followed by one line per GPU. If the output shows NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver, the container started but the driver handshake failed — reboot the host after a driver install, then repeat.

Why this works: with --gpus, Docker mounts /dev/nvidia* and the driver libraries from the host into the container, so the container uses the host kernel module.

Step 2. Map storage, memory and ports when starting the LLM container

Illustration: Step 2. Map storage, memory and ports when starting the LLM container
Illustration: Step 2. Map storage, memory and ports when starting the LLM container

What to do: start Ollama with three explicit decisions — where models live, which port is published, and how much memory the container may take.

docker run -d \
  --gpus=all \
  --name ollama \
  -p 127.0.0.1:11434:11434 \
  -v ollama:/root/.ollama \
  --memory=16g \
  --restart unless-stopped \
  ollama/ollama

If you want the API reachable from another machine, do not widen the port mapping blindly — put a reverse proxy or tunnel in front of it, as described in How to Expose a Local LLM to the Internet Without Port Forwarding.

Why: a named volume survives docker rm, so you download weights once; a loopback-bound port keeps a model with no authentication off the network; the memory cap turns a silent host freeze into a clean container exit you can debug.

How to check:

docker ps --filter name=ollama
docker inspect -f '{{.State.Status}} mem={{.HostConfig.Memory}}' ollama

Expected: status running, and mem= showing the byte value of the limit you set (not 0, which means "unlimited").

Example: mem=17179869184 is 16g in bytes. If you see mem=0, the --memory flag was dropped — recreate the container with the flag.

Step 3. Verify the model answers and confirm it runs on the GPU

Illustration: Step 3. Verify the model answers and confirm it runs on the GPU
Illustration: Step 3. Verify the model answers and confirm it runs on the GPU

What to do: pull a small model, send one request over the API, and watch GPU utilization while it answers.

docker exec -it ollama ollama pull llama3.2
curl http://localhost:11434/api/generate -d '{"model":"llama3.2","prompt":"Say OK","stream":false}'

In a second terminal, while the request runs:

nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv
docker stats --no-stream ollama

Why: a container that starts is not a container that computes on the GPU. Ollama silently falls back to CPU when it cannot allocate VRAM, and the only visible symptom is "it answers, but slowly".

How to check:

Example: request sent, then nvidia-smi --query-gpu prints 35 %, 5120 MiB mid-generation and returns to 0 %, 5120 MiB after — the memory stays allocated because the model is still loaded. docker ps showing Up 2 minutes plus a populated "response" field is your success state.

Why this works: --gpus=all gave the container the device, and the model loader puts the weights into VRAM on first request, so the utilization counter moves only when inference starts.

Troubleshooting

Problem. The container refuses to start at all; nothing is created.

Problem. The container starts and immediately is not reachable on localhost:11434, or it dies on start with a port message.

docker ps -a --filter publish=11434
ss -ltnp | grep 11434
docker rm -f ollama
docker run -d --gpus=all --name ollama -p 127.0.0.1:11435:11434 -v ollama:/root/.ollama ollama/ollama
curl http://localhost:11435/api/tags

If instead the container is up but curl gets connection refused, the API is bound to loopback inside the container — restart with -e OLLAMA_HOST=0.0.0.0:11434 so it listens on all container interfaces.

Problem. The model loads, then the container vanishes mid-answer.

docker rm -f ollama
docker run -d --gpus=all --name ollama -p 127.0.0.1:11434:11434 \
  -v ollama:/root/.ollama --memory=24g ollama/ollama

Summary and next steps

You now have a container that sees the GPU, stores models in a persistent volume, publishes its API on loopback only, and has a memory ceiling you can explain in one sentence. Next, read the sizing table to pick a model that fits your card, and only then decide whether to publish the API beyond localhost.

FAQ

Why does my container start fine but the model answers very slowly?

Ollama fell back to CPU because it could not allocate VRAM. Check nvidia-smi during a request — idle GPU utilization plus high CPU in docker stats confirms it. Reduce the model size or quantization, or free VRAM on the host.

Do I have to re-download the model every time I recreate the container?

No, as long as the model directory is a volume. -v ollama:/root/.ollama keeps the weights outside the container filesystem; docker rm ollama and re-running the same docker run line reuses them. Without that -v, every recreation re-downloads.

Which port does the local LLM API listen on and how do I change it?

Ollama's default is 11434 inside the container. Use -p 127.0.0.1:<host-port>:11434 to change the host side, then call http://localhost:<host-port>/api/.... Changing the container side requires setting OLLAMA_HOST — verify the exact syntax in the documentation for your image version.

Can I use my GPU with Docker Desktop on macOS?

Not currently — Docker Desktop on macOS does not pass an NVIDIA GPU into containers, so the model runs on CPU. On a Mac, expect slower inference and check the documentation for any change in GPU support before planning the setup.

🤖 AI summary

Related guides

How to Expose a Local LLM to the Internet Without Port Forwarding: Tunnels and Reverse Proxy

Пошаговая инструкция: как дать удалённый доступ к OpenAI-совместимому API своей модели за NAT через Cloudflare Tunnel, T

How to build an AI shopping assistant on a local model

Практический гайд по созданию локального ИИ-ассистента для магазина на 9B-модели без облачных API.

Windows 11 26H2: common issues after update and how to fix them

Собрали типичные ошибки и баги Windows 11 26H2, а также пошаговые решения для каждой из них.

How to Give Your Coding Agent Persistent Project Memory: devmemory, Graphicle and MCP

Настройка локальной памяти для Claude Code и Cursor: MCP-сервер devmemory, граф кода Graphicle и сохранение контекста ме