# How to fix Docker errors when running a local LLM: GPU, memory and port forwarding

> AI Release · @ai_release1 · https://ai-release.org/guides/kak-ispravit-oshibki-docker-pri-zapuske-lokalnoy-llm-gp_en.html

## TL;DR
- **Time to complete:** ~30 minutes (first launch takes longer while the image and model download).
- **Difficulty:** Intermediate.
- **You will need:**
  - Docker Engine on Linux, or Docker Desktop with the WSL2 backend on Windows.
  - An NVIDIA GPU with a working driver (`nvidia-smi` runs on the host).
  - NVIDIA Container Toolkit on Linux (the package that lets Docker pass the GPU into a container).
  - The official Ollama image `ollama/ollama` (confirm the current tag in the registry before you start).
  - ~10 GB of free disk space for the image plus model weights; exact model sizes are in the [RAM/VRAM sizing table](https://ai-release.org/guides/localnye-ii-modeli-skolko-ram-i-vram.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide).

## Requirements and setup
![Illustration: Requirements and setup](https://ai-release.org/guides/img/kak-ispravit-oshibki-docker-pri-zapuske-lokalnoy-llm-gp-en-1.jpg?v=1792789996)


Docker runs your LLM in an isolated box with no direct access to the GPU, the host filesystem or the host network unless you grant it explicitly. Almost every "it doesn't start" report comes from one of those three grants missing.

- **OS:** Linux (any current distro) or Windows 11 with WSL2. macOS has no NVIDIA GPU passthrough in Docker Desktop — check the documentation for the current state of GPU support before starting.
- **Permissions:** your user must be able to talk to the Docker daemon. On Linux either add yourself to the `docker` group (`sudo usermod -aG docker $USER`, then log out and back in) or prefix commands with `sudo`.
- **Driver:** install the NVIDIA driver **on the host** first. The container never installs a driver; it borrows the host one.
- **Toolkit (Linux):** install `nvidia-container-toolkit`, then restart Docker so the `nvidia` runtime is registered.

Run this checklist before touching the LLM image:

```bash
nvidia-smi
docker version
docker info | grep -i -E "runtimes|default runtime"
df -h
```

- [ ] `nvidia-smi` prints a table with driver version and GPU memory.
- [ ] `docker version` shows both Client and Server sections (no "Cannot connect to the Docker daemon").
- [ ] `docker info` lists `nvidia` among the runtimes (Linux).
- [ ] `df -h` shows enough free space for image + weights.

**Example:** on a Linux workstation `docker info | grep -i runtimes` returns a line containing `nvidia`; on a machine without the toolkit the output contains only `runc`, and every `--gpus` command will fail later with the error described in Troubleshooting.

**Why this works:** the toolkit writes a runtime entry into `/etc/docker/daemon.json`, so Docker knows *how* to mount the GPU device nodes when you ask for them.

If you hit unrelated Docker Desktop/WSL2 breakage after a Windows feature update, see [Windows 11 26H2: common issues after update](https://ai-release.org/guides/windows-11-26h2-chastye-problemy-posle-obnovleniya-i-ka.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide).

## Step 1. Prove that a container can actually see the GPU

**What to do:** run a throwaway container that only calls `nvidia-smi`.

```bash
docker run --rm --gpus=all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi
```

`--gpus=all` asks Docker for every GPU on the host. `--rm` deletes the container after it exits. If that tag is not published for your driver, pick another `-base-ubuntu22.04` tag — check the documentation/tag list rather than guessing.

**Why:** this separates two problems that look identical to a beginner — "the driver is broken" and "Docker isn't allowed to use the driver". If this command works, your LLM container has no excuse.

**How to check:** you must see the same GPU table you saw on the host, rendered inside the container. Anything else — a driver error, an empty table, or the `could not select device driver` message — means stop here and fix the runtime before pulling the model image.

**Example:** expected screen contents include a header row like `GPU  Name  Persistence-M  Bus-Id  Disp.A  Memory-Usage` followed by one line per GPU. If the output shows `NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver`, the container started but the driver handshake failed — reboot the host after a driver install, then repeat.

**Why this works:** with `--gpus`, Docker mounts `/dev/nvidia*` and the driver libraries from the host into the container, so the container uses the host kernel module.

## Step 2. Map storage, memory and ports when starting the LLM container
![Illustration: Step 2. Map storage, memory and ports when starting the LLM container](https://ai-release.org/guides/img/kak-ispravit-oshibki-docker-pri-zapuske-lokalnoy-llm-gp-en-2.jpg?v=1792790003)


**What to do:** start Ollama with three explicit decisions — where models live, which port is published, and how much memory the container may take.

```bash
docker run -d \
  --gpus=all \
  --name ollama \
  -p 127.0.0.1:11434:11434 \
  -v ollama:/root/.ollama \
  --memory=16g \
  --restart unless-stopped \
  ollama/ollama
```

- `-p 127.0.0.1:11434:11434` — publish container port `11434` (Ollama's default API port) **only on the host loopback**. `-p host:container`; the leading `127.0.0.1` is what keeps the API off your LAN.
- `-v ollama:/root/.ollama` — a named volume for the model store. `/root/.ollama` is the container path where Ollama keeps downloaded models.
- `--memory=16g` — a hard ceiling. Replace `16g` with the value you computed from the [size and quantization table](https://ai-release.org/guides/localnye-ii-modeli-skolko-ram-i-vram.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide); leave headroom for the host.
- `--restart unless-stopped` — the container comes back after a reboot, where Docker Desktop often is not running yet.

If you want the API reachable from another machine, do **not** widen the port mapping blindly — put a reverse proxy or tunnel in front of it, as described in [How to Expose a Local LLM to the Internet Without Port Forwarding](https://ai-release.org/guides/kak-otkryt-dostup-k-lokalnoy-llm-iz-interneta-bez-probr.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide).

**Why:** a named volume survives `docker rm`, so you download weights once; a loopback-bound port keeps a model with no authentication off the network; the memory cap turns a silent host freeze into a clean container exit you can debug.

**How to check:**

```bash
docker ps --filter name=ollama
docker inspect -f '{{.State.Status}} mem={{.HostConfig.Memory}}' ollama
```

Expected: status `running`, and `mem=` showing the byte value of the limit you set (not `0`, which means "unlimited").

**Example:** `mem=17179869184` is `16g` in bytes. If you see `mem=0`, the `--memory` flag was dropped — recreate the container with the flag.

## Step 3. Verify the model answers and confirm it runs on the GPU
![Illustration: Step 3. Verify the model answers and confirm it runs on the GPU](https://ai-release.org/guides/img/kak-ispravit-oshibki-docker-pri-zapuske-lokalnoy-llm-gp-en-3.jpg?v=1792790007)


**What to do:** pull a small model, send one request over the API, and watch GPU utilization while it answers.

```bash
docker exec -it ollama ollama pull llama3.2
curl http://localhost:11434/api/generate -d '{"model":"llama3.2","prompt":"Say OK","stream":false}'
```

In a second terminal, while the request runs:

```bash
nvidia-smi --query-gpu=utilization.gpu,memory.used --format=csv
docker stats --no-stream ollama
```

**Why:** a container that starts is not a container that computes on the GPU. Ollama silently falls back to CPU when it cannot allocate VRAM, and the only visible symptom is "it answers, but slowly".

**How to check:**
- `curl` returns JSON containing a `"response"` field with text.
- `nvidia-smi` shows non-zero `utilization.gpu` and a `memory.used` jump during the request.
- `docker stats` shows CPU well below 100% for a single stream; near-100% CPU with an idle GPU means CPU fallback.

**Example:** request sent, then `nvidia-smi --query-gpu` prints `35 %, 5120 MiB` mid-generation and returns to `0 %, 5120 MiB` after — the memory stays allocated because the model is still loaded. `docker ps` showing `Up 2 minutes` plus a populated `"response"` field is your success state.

**Why this works:** `--gpus=all` gave the container the device, and the model loader puts the weights into VRAM on first request, so the utilization counter moves only when inference starts.

## Troubleshooting

**Problem. The container refuses to start at all; nothing is created.**
- Error: `docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]].`
- Cause: the NVIDIA Container Toolkit is not installed, or Docker was not restarted after installing it, so no `nvidia` runtime is registered. On Windows it usually means Docker Desktop is not using the WSL2 backend.
- Solution: install `nvidia-container-toolkit` on the host, restart Docker (`sudo systemctl restart docker`), confirm `docker info | grep -i runtimes` lists `nvidia`, then re-run the Step 1 test command.

**Problem. The container starts and immediately is not reachable on `localhost:11434`, or it dies on start with a port message.**
- Error: `Error response from daemon: driver failed programming external connectivity on endpoint ollama: Bind for 0.0.0.0:11434 failed: port is already allocated.`
- Cause: another process (often a natively installed Ollama, or a previous container) already holds port `11434`.
- Solution: find the owner and either stop it or move the host port.

```bash
docker ps -a --filter publish=11434
ss -ltnp | grep 11434
docker rm -f ollama
docker run -d --gpus=all --name ollama -p 127.0.0.1:11435:11434 -v ollama:/root/.ollama ollama/ollama
curl http://localhost:11435/api/tags
```

If instead the container is up but `curl` gets connection refused, the API is bound to loopback inside the container — restart with `-e OLLAMA_HOST=0.0.0.0:11434` so it listens on all container interfaces.

**Problem. The model loads, then the container vanishes mid-answer.**
- Error: `docker ps -a` shows `Exited (137)`, and `docker inspect -f '{{.State.OOMKilled}}' ollama` returns `true`.
- Cause: the kernel OOM killer hit the container limit. The weights plus the context window did not fit in the `--memory` ceiling, or VRAM ran out and the model spilled into host RAM.
- Solution: lower the model size or quantization (use the [sizing table](https://ai-release.org/guides/localnye-ii-modeli-skolko-ram-i-vram.html?utm_source=tg&utm_medium=channel&utm_campaign=guide_inline&utm_content=guide_to_guide)), or raise the ceiling if the host has room:

```bash
docker rm -f ollama
docker run -d --gpus=all --name ollama -p 127.0.0.1:11434:11434 \
  -v ollama:/root/.ollama --memory=24g ollama/ollama
```

## Summary and next steps
You now have a container that sees the GPU, stores models in a persistent volume, publishes its API on loopback only, and has a memory ceiling you can explain in one sentence. Next, read the sizing table to pick a model that fits your card, and only then decide whether to publish the API beyond localhost.

## FAQ

**Why does my container start fine but the model answers very slowly?**
Ollama fell back to CPU because it could not allocate VRAM. Check `nvidia-smi` during a request — idle GPU utilization plus high CPU in `docker stats` confirms it. Reduce the model size or quantization, or free VRAM on the host.

**Do I have to re-download the model every time I recreate the container?**
No, as long as the model directory is a volume. `-v ollama:/root/.ollama` keeps the weights outside the container filesystem; `docker rm ollama` and re-running the same `docker run` line reuses them. Without that `-v`, every recreation re-downloads.

**Which port does the local LLM API listen on and how do I change it?**
Ollama's default is `11434` inside the container. Use `-p 127.0.0.1:<host-port>:11434` to change the host side, then call `http://localhost:<host-port>/api/...`. Changing the container side requires setting `OLLAMA_HOST` — verify the exact syntax in the documentation for your image version.

**Can I use my GPU with Docker Desktop on macOS?**
Not currently — Docker Desktop on macOS does not pass an NVIDIA GPU into containers, so the model runs on CPU. On a Mac, expect slower inference and check the documentation for any change in GPU support before planning the setup.