← All guides AI Release
RU Subscribe to @ai_releaseorg

How to speed up a local LLM: speculative decoding, batching and KV cache

Updated: 17.11.2026 · AI Release · @ai_releaseorg
How to speed up a local LLM: speculative decoding, batching and KV cache

TL;DR

Most "speed up your local LLM" posts tell you to switch on speculative decoding and stop there. That advice breaks in two situations they never mention: when the server handles several requests at once, and when your prompt is long but the answer is short. Before changing a single flag, decide which number you are optimising — tokens per second on one stream (what you feel while chatting) or total throughput across concurrent requests (what matters when you expose an API). Speculative decoding — a small "draft" model guesses the next few tokens and the big model verifies them in one pass — targets the first number. Continuous batching targets the second. KV cache quantization makes nothing faster on its own; it buys VRAM so you can afford a longer context or more parallel slots.

The second thing typical guides skip: each of these three knobs has a failure mode where it makes things worse. A draft model that guesses badly adds work instead of removing it. More parallel slots raise throughput while making every individual answer slower. A quantized KV cache can shift output quality on long contexts. That is why every step below ends with a measurement, so you compare against your own baseline instead of trusting a number from someone else's machine.

Requirements and setup

Illustration: Requirements and setup
Illustration: Requirements and setup
  git clone https://github.com/ggml-org/llama.cpp
  cd llama.cpp
  cmake -B build -DGGML_CUDA=ON
  cmake --build build --config Release -j

Expected result: build/bin/llama-server exists.

What to do: confirm the flags your build supports before writing any config, because flag names in llama.cpp change between releases.

Why: every command later in this guide uses these names; if your build renamed one, you will debug the wrong problem for an hour.

How to check:

llama-server --help | grep -i -E 'draft|spec|parallel|cache-type|flash'

You should see one line per supported flag (for example a draft-model flag and a --cache-type-k flag). If a flag you intend to use is missing, update the build — do not invent an alias.

Step 1. Record a baseline before touching any flag

Illustration: Step 1. Record a baseline before touching any flag
Illustration: Step 1. Record a baseline before touching any flag

What to do: start the server with one slot and no speculation, then record three numbers: single-stream tokens per second, wall time for four concurrent requests, and peak VRAM.

Why: without a baseline you cannot tell whether a knob helped or hurt. A number like "2x faster" is meaningless until you know your own starting point.

How to check: start the server:

llama-server \
  -m /models/<your-model>.gguf \
  -c 4096 \
  -np 1 \
  --port 8080

-c is total context length, -np is the number of parallel slots (one slot means one request at a time). Then, in a second terminal, send one request and read the timings object the server returns:

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"local","messages":[{"role":"user","content":"Summarize HTTP caching in 200 words."}],"max_tokens":200,"temperature":0}' \
  | jq '.timings'

Expected output shape — your numbers will differ, that is the point:

{
  "prompt_n": <prompt tokens>,
  "prompt_per_second": <prompt processing speed>,
  "predicted_n": <generated tokens>,
  "predicted_per_second": <your baseline tokens/s>
}

If your build does not return timings, read the eval time lines in the server log instead — check with the documentation for your build. Now measure concurrency. With -np 1 the four requests queue behind each other:

for i in 1 2 3 4; do
  curl -s -o /dev/null -w "req $i: %{time_total}s\n" \
    http://127.0.0.1:8080/v1/chat/completions \
    -H 'Content-Type: application/json' \
    -d '{"model":"local","messages":[{"role":"user","content":"List five uses of a KV cache."}],"max_tokens":150}' &
done
wait

Write the results down in this checklist before continuing:

The reason this works as a baseline: -np 1 removes batching from the equation, and temperature: 0 makes the two runs comparable instead of sampling different tokens each time.

Step 2. Turn on speculative decoding, batching and KV cache quantization

What to do: restart the server with all three knobs, changing nothing else.

Why: each flag attacks a different bottleneck, so they are safe to combine but impossible to evaluate as a bundle — you will separate them in Step 3.

How to check: run:

llama-server \
  -m /models/<your-model>.gguf \
  -md /models/<your-draft-model>.gguf \
  --draft-max 8 \
  -c 16384 \
  -np 4 \
  --flash-attn \
  --cache-type-k q8_0 \
  --cache-type-v q8_0 \
  --port 8080

What each part does:

KnobFlagWhat it improvesWhat it costs
Speculative decoding`-md`, `--draft-max`tokens/s on a single streamextra VRAM for the draft model
Continuous batching`-np` (with the build's batching default)total throughput with many clientsper-request latency
KV cache quantization`--cache-type-k`, `--cache-type-v`more context/slots in the same VRAMsmall quality risk on long contexts
Flash attention`--flash-attn`memory traffic at long contextmust be compiled in

--draft-max caps how many tokens the draft model proposes per step — a higher value means more guessing per pass, which only pays off if the draft is usually right. Note that in current builds quantizing the V cache requires flash attention, and continuous batching is already the default in the server, so -np alone may be enough — confirm both points against your build's documentation.

Example of the arithmetic that trips people up: -c 16384 with -np 4 means each of the four slots has roughly 4096 tokens of context, not 16384. If your prompts are longer than that, raise -c.

One hard requirement: the draft model must use the same tokenizer and vocabulary as the target model. A draft from a different family will load, but its guesses will be rejected constantly, and you will pay the overhead for nothing. Check the model cards, not the filenames.

Step 3. Re-measure and decide which knobs to keep

Illustration: Step 3. Re-measure and decide which knobs to keep
Illustration: Step 3. Re-measure and decide which knobs to keep

What to do: repeat the exact two tests from Step 1 against the new server and compare. Then remove any knob that did not move the number you care about.

Why: speculative decoding is not free — it spends VRAM and per-step compute on guesses. If you serve many concurrent clients, the win can shrink or disappear, and the honest answer is to keep batching and drop -md.

How to check: send the identical single-stream request:

curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"local","messages":[{"role":"user","content":"Summarize HTTP caching in 200 words."}],"max_tokens":200,"temperature":0}' \
  | jq '.timings.predicted_per_second'

Then the same four-request loop from Step 1, plus a VRAM reading while it runs:

nvidia-smi --query-gpu=memory.used,memory.total --format=csv

Success looks like this: predicted_per_second is higher than your Step 1 value, the four-request wall time is shorter (requests now overlap instead of queuing), and memory.used still leaves headroom — if the card is at its limit, the next long prompt will kill the process.

Decision rules, applied one knob at a time:

1. Single-stream gain but concurrency gain flat → keep -md, keep -np 4.

2. Single-stream flat or slower → remove -md and re-measure; the draft model is guessing wrong.

3. Concurrency gain but answers feel slower one-to-one → lower -np to 2.

4. Quality regression on long prompts → revert --cache-type-v to the default type first.

If your target is an OpenAI-compatible endpoint that other tools can call, the server is already compatible; the wiring for a coding agent is described in How to connect a local LLM to Claude Code: llama-server and an OpenAI-compatible API.

Troubleshooting

Problem. The server refuses to start as soon as you add the draft model

  llama-server --help | grep -i -E 'draft|spec'

Problem. CUDA out of memory after raising -np and -c

  llama-server -m /models/<your-model>.gguf -c 8192 -np 2 --n-gpu-layers 24 --port 8080

Problem. Clients hang and the log repeats a slot message

  llama-server -m /models/<your-model>.gguf -c 32768 -np 4 --port 8080

Summary and next steps

You now have measured baselines for single-stream speed, concurrency and VRAM, and a flag set you can defend with your own numbers. Next, pin the winning command in a systemd unit or container and re-run the same two tests after every model swap — if you plan to serve several users, continue with How to Deploy a Local LLM on Kubernetes: Step-by-Step Guide.

FAQ

Does speculative decoding still help when several users hit the server at once?

It helps less, and sometimes not at all — with a full batch the GPU is already busy and the draft model's extra steps compete for the same compute. Measure the four-request loop with and without -md and keep whichever wins.

Which draft model should I pick?

A much smaller model from the same family as your target, with the same tokenizer and vocabulary. Same family matters more than size: mismatched vocabularies cause constant rejections and you pay overhead with no gain.

Does quantizing the KV cache change answer quality?

It can, mostly on long contexts where errors accumulate. Test with a long prompt at temperature: 0 and compare answers before and after switching --cache-type-v back to the default.

Do I have to restart the server for every flag change?

Yes — these are startup flags, not runtime settings. Keep the previous working command in a shell script so you can roll back in one line.

🤖 AI summary

AI Release Knowledge Layer
Machine-readable description · Version 1.0

Status: practical guide. Some statements come from vendors; verify in your environment.

1. Summary

Time to complete: 45–60 minutes, including model download and two measurement rounds. Difficulty: Intermediate (you need a working llama-server and a GPU with free VRAM).

Inside: direct answers, key facts, entities, context, reliability, unknowns and sources.

Machine data: Citation Map JSON · JSON-LD · Semantic Embeddings — separate stage.

Full Knowledge Layer: sections 2–8

2. Direct answers

What is required?
OS and driver: Linux x86_64, or Windows 11 with WSL2. Run nvidia-smi and confirm the driver loads and reports your GPU. If it prints NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver, fix the driver first — nothing below will work.; llama.cpp with CUDA: build it yourself so GPU offload is… → details (AR-GUIDE-KAKUSKORIT-003)
How to do it?
Step 1. Record a baseline before touching any flag → Step 2. Turn on speculative decoding, batching and KV cache quantization → Step 3. Re-measure and decide which knobs to keep → details (AR-GUIDE-KAKUSKORIT-004)
Which topics does it cover?
как ускорить локальную LLM, спекулятивное декодирование настройка, ускорить генерацию токенов llama.cpp vLLM → details (AR-GUIDE-KAKUSKORIT-002)
What are the key points?
Time to complete: 45–60 minutes, including model download and two measurement rounds. Difficulty: Intermediate (you need a working llama-server and a GPU with free VRAM). → details (AR-GUIDE-KAKUSKORIT-007)

3. Key facts

  1. Time to complete: 45–60 minutes, including model download and two measurement rounds. — evidence; status: заявление источника.
  2. Difficulty: Intermediate (you need a working llama-server and a GPU with free VRAM). — evidence; status: заявление источника.
  3. You will need: a llama.cpp build that contains llama-server (verify with llama-server --version), a GGUF target model, a smaller GGUF draft model from the same model family, an NVIDIA GPU, curl, and optionally jq. GGUF — the single-file model format used by llama.cpp; VRAM — memory on your graphics card. — evidence; status: заявление источника.
  4. OS and driver: Linux x86_64, or Windows 11 with WSL2. Run nvidia-smi and confirm the driver loads and reports your GPU. If it prints NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver, fix the driver first — nothing below will work. — evidence; status: заявление источника.
  5. llama.cpp with CUDA: build it yourself so GPU offload is compiled in: — evidence; status: заявление источника.
  6. Models: two GGUF files — <your-model>.gguf (the model you actually serve) and <your-draft-model>.gguf (a much smaller model from the same family, so both share a tokenizer). File sizes depend on quantization; read them off the model card, do not guess. — evidence; status: заявление источника.

4. Entities

EntityTypeRole
как ускорить локальную LLMтема/инструментупоминается в заголовке, TL;DR или разделах гайда
спекулятивное декодирование настройкатема/инструментупоминается в заголовке, TL;DR или разделах гайда
ускорить генерацию токенов llama.cpp vLLMтема/инструментупоминается в заголовке, TL;DR или разделах гайда
Windowsплатформа/технологияупоминается в заголовке, TL;DR или разделах гайда
Linuxплатформа/технологияупоминается в заголовке, TL;DR или разделах гайда
GitHubплатформа/технологияупоминается в заголовке, TL;DR или разделах гайда
CUDAплатформа/технологияупоминается в заголовке, TL;DR или разделах гайда
Gitплатформа/технологияупоминается в заголовке, TL;DR или разделах гайда
NVIDIAкомпанияупоминается в заголовке, TL;DR или разделах гайда
llama.cppтема/инструментупоминается в заголовке, TL;DR или разделах гайда

5. Context

  • Guide — describes → How to speed up a local LLM: speculative decoding, batching and KV cache (AR-GUIDE-KAKUSKORIT-001).
  • Guide — includes step → Step 1. Record a baseline before touching any flag (AR-GUIDE-KAKUSKORIT-001).

6. Reliability

Confirmed

No independent confirmation is provided; source claims and editorial conclusions are separated below.

Source claims

  • Product features, versions, prices and limits come from vendors and the guide.
  • Commands and steps are recommendations, not guaranteed results.

AI Release analysis

  • Structure, comparison and selection criteria are editorial.
  • The guide does not replace an independent audit or legal advice.

7. Unknowns

  • Current versions, prices and limits.
  • Independent benchmarks and quality checks.
  • Terms of use and data policy.

8. Sources

SourceTypeRole
github.comвнешний источникссылка из тела гайда

Cite as: AI Release. Как ускорить локальную LLM: спекулятивное декодирование, батчинг и кэш KV. ai-release.org.