How to Diagnose a Slow Local AI Model on Windows: GPU, VRAM and Logs
TL;DR
- Time to complete: 20–30 minutes
- Difficulty: Intermediate
- You will need:
- Windows 10 or Windows 11 (confirm with winver)
- The local runtime you already use (Ollama, LM Studio, llama.cpp — any one of them)
- A vendor GPU tool for live counters (nvidia-smi ships with NVIDIA drivers; AMD/Intel users — check the documentation for the equivalent tool for your driver)
- Administrator rights, for Event Viewer and Resource Monitor
- The exact model tag/file you are debugging, already downloaded once, plus one fixed test prompt
Requirements and setup
Prepare the environment before measuring anything — a download or a background game during the test invalidates the numbers.
- OS: Windows 10 or 11. Run
winverand note the build number. - Driver: the GPU driver installed by the vendor, not the generic Microsoft Basic Display Adapter.
- Runtime: installed, and the model pulled at least once, so the test is not measuring disk download speed.
- Free RAM: enough to hold the model file if it spills out of VRAM.
- Quiet machine: close browsers with hardware acceleration, games, video editors and any second model server.
- One fixed prompt, reused for every run. Example:
Summarize this text in 3 bullets: <paste ~200 words>.
If you have never started a model locally, do that first with How to run AI models locally on your PC: a beginner's guide, then come back here.
What to do: list the adapters Windows sees and their driver versions.
Get-CimInstance Win32_VideoController | Select-Object Name, DriverVersion, AdapterRAMWhy: this confirms which GPU the runtime will pick on a laptop with both integrated and discrete graphics. AdapterRAM is often reported incorrectly for cards with large VRAM, so treat it as a rough hint only — Step 2 uses the vendor tool for real numbers.
How to check: you get one row per adapter. If you see Microsoft Basic Display Adapter, the vendor driver is missing and every GPU measurement below will be wrong.
Step 1. Baseline: record what "slow" actually means
What to do: split the run into two phases — load (model into memory) and generation (tokens per second) — and record both.
Why: load time is bound by disk and RAM; generation is bound by GPU, VRAM and compute. They have different fixes, so measuring them together hides the cause.
How to check: with Ollama, run with verbose output and read the timing block:
ollama run <your-model> --verboseExpected output ends with a timing block containing fields such as load duration, eval count, eval duration and eval rate. The two numbers that matter: load duration (how long the weights took to load) and eval rate (tokens per second during generation). Field names differ between versions — check the documentation for your build.
Example: run the same prompt twice. The first run is cold (weights not cached), the second is warm. If the cold run takes far longer than the warm one but eval rate is identical, the problem is loading, not the GPU.
| Observation | Likely bottleneck | Where to look next |
|---|---|---|
| Warm `eval rate` is low, GPU idle | Model is on CPU | Step 2, then Step 3 offload line |
| `load duration` dominates | Disk or RAM too small | Task Manager → Disk / Memory |
| First answer slow, later answers fast | Cold load, expected behaviour | Nothing to fix |
| `eval rate` drops as the chat grows | Context length eating VRAM | Step 2, reduce context |
Step 2. Watch GPU and VRAM while the model generates
What to do: start the monitor *before* you send the prompt and keep it running until generation finishes.
Why: a single snapshot after the answer is useless — VRAM is released when generation ends. You must catch the peak.
How to check: with an NVIDIA card, sample every second in a second terminal:
nvidia-smi --query-gpu=name,memory.total,memory.used,utilization.gpu --format=csv -l 1Expected: a repeated CSV line, e.g. NVIDIA GeForce RTX ..., 8192 MiB, 7340 MiB, 97 %. Your values will differ; what matters is the peak memory.used and whether utilization.gpu rises above idle during generation. Stop it with Ctrl+C.
Example: open Task Manager → Performance → GPU and watch three counters while the model answers: Dedicated GPU memory, Shared GPU memory, GPU utilization.
| What you see | What it means | Action |
|---|---|---|
| GPU utilization stays near 0 %, dedicated memory flat | The runtime runs on CPU — the GPU is not being used at all | Go to Step 3, find the offload line |
| Dedicated memory pinned at maximum, shared memory climbing | VRAM overflow: layers are spilling into system RAM over PCIe, which is far slower | Lower context size, use a smaller quantization, or reduce offloaded layers |
| GPU utilization high, memory not full, still slow | Compute-bound: thermals, power limit, or a model too large for the chip | Re-run `nvidia-smi` and compare clocks under load vs idle |
| GPU idle, disk at 100 % | Weights are being streamed from disk because RAM is too small | Free RAM or use a smaller model file |
VRAM is consumed by four things: model weights, the KV cache (the per-conversation memory of the model), the runtime's own CUDA context, and anything else using the GPU. Reducing context length is the fastest way to test the KV-cache share — halve it and repeat Step 1.
- [ ] Monitors closed, only the runtime running
- [ ] Peak
memory.usedrecorded - [ ] Peak GPU utilization recorded
- [ ] Shared GPU memory checked
- [ ] Test repeated with halved context length
Step 3. Read the logs and confirm the bottleneck
What to do: turn on verbose logging, reproduce the slow run, and read the model-loading block plus the Windows System log.
Why: logs state explicitly how many layers were placed on the GPU and whether the driver crashed mid-run — the two facts Task Manager cannot show.
How to check: in llama.cpp-based runtimes the loader prints a line per buffer. Look for the pattern:
llm_load_tensors: offloaded 0/33 layers to GPU0/<N> means nothing went to the GPU — that is your answer. 33/33 (or any number equal to the total) means full offload. The total depends on the model, so read the ratio, not the number.
Example: for an Ollama-style server log, tail the file while you send the prompt. The path depends on the version — check the documentation for yours; the commonly used location is:
Get-Content "$env:USERPROFILE\.ollama\logs\server.log" -Tail 50 -WaitThen check whether Windows recorded a driver-level event during the run:
Get-WinEvent -FilterHashtable @{LogName='System'; Level=2; StartTime=(Get-Date).AddHours(-1)} |
Select-Object TimeCreated, ProviderName, Id, Message | Format-ListExpected: a list of errors from the last hour. An empty result means Windows saw no hardware fault, which points to configuration rather than the driver.
For the wider picture of how Windows schedules GPU work and background AI features, see Windows with AI: Copilot and neural networks in the system.
Troubleshooting
Problem. Generation is slow and Task Manager shows the GPU idle
- Error:
llm_load_tensors: offloaded 0/33 layers to GPU - Cause: the offload/GPU-layers setting is
0, or the installed runtime build has no GPU backend. - Solution: set GPU layers to the model's layer count (start with a partial value if VRAM is tight), restart the runtime, and confirm the ratio changes in the log. If it stays
0/<N>at any setting, the build lacks GPU support — check the documentation for a GPU-enabled build.
Problem. The run dies or stalls right after the prompt
- Error:
CUDA out of memory - Cause: weights plus KV cache exceed available VRAM; often triggered by a long context or another app holding memory.
- Solution: close GPU consumers, halve the context size, and retry. Then lower the number of offloaded layers until the run completes:
nvidia-smi --query-gpu=memory.used,memory.total --format=csvRun it right before starting the model — if memory.used is already high with nothing running, a background process is holding VRAM.
Problem. The screen flickers black and the answer is cut off
- Error:
Display driver nvlddmkm stopped responding and has successfully recovered. - Cause: Windows TDR (Timeout Detection and Recovery) reset the GPU because a kernel took too long — common when a large model runs on the same GPU that drives the display.
- Solution: reduce the load per run (smaller model or fewer offloaded layers), update the vendor driver, and prefer running the model on a discrete GPU that is not driving the monitor. Re-check with the
Get-WinEventcommand from Step 3 — the event should stop appearing.
Summary and next steps
Measure load time and eval rate separately, then watch dedicated versus shared GPU memory during generation to learn whether the model is on the GPU at all. Confirm the verdict in the runtime log's offload line and the Windows System log, and fix the layer that is actually saturated instead of reinstalling drivers blindly.
FAQ
Why is the first answer slow and the next ones fast?
The first run loads the weights from disk into memory; later runs reuse them. Compare load duration against eval rate in the verbose output — that difference is expected, not a fault.
Does the model use shared GPU memory?
Yes, when it does not fit into dedicated VRAM, and it is much slower. Watch the Shared GPU memory counter in Task Manager during generation; if it climbs, reduce context size, quantization or offloaded layers.
How do I know if the GPU is being used at all?
Two checks: utilization.gpu rises above idle in nvidia-smi during generation, and the log shows offloaded <N>/<N> layers to GPU rather than 0/<N>.
Can I fix slow generation by raising the context size?
No — a larger context needs a larger KV cache and more VRAM, which usually makes things slower or triggers an out-of-memory error. Test with a smaller context first to isolate the cause.
🤖 AI summary
- Updated: 2026-10-21
- Topics: локальная модель тормозит, медленная генерация токенов, проверить загрузку GPU при инференсе, nvidia-smi windows, почему llm работает медленно
- Summary: Разбор причин низкой скорости генерации: проверка загрузки GPU и VRAM в диспетчере задач, GPU-Z и nvidia-smi, чтение логов llama.cpp и подбор квантизации под железо.
- Key facts:
- TL;DR Time to complete: 20–30 minutes Difficulty: Intermediate You will need: - Windows 10 or Windows 11 (confirm with winver ) - The local runtime you already use (Ollama, LM Studio, llama.cpp — any one of them) - A vendor GPU tool for live counters ( nvidia-smi ships with NVIDIA drivers; AMD/Intel users — check the documentation for the equivalent tool for your driver) - Administrator rights, for Event Viewer and Resource Monitor - The exact model tag/file you are debugging, already downloaded once, plus one fixed test prompt Requirements and setup Illustration: Requirements and setup Prepare the environment before measuring anything — a download or a background game during the test invalidates the numbers.
- OS: Windows 10 or 11.
- Run winver and note the build number.
- Driver: the GPU driver installed by the vendor, not the generic Microsoft Basic Display Adapter.
- Runtime: installed, and the model pulled at least once, so the test is not measuring disk download speed.
- Free RAM: enough to hold the model file if it spills out of VRAM.