llama-server and a GPU with free VRAM).llama-server (verify with llama-server --version), a GGUF target model, a smaller GGUF draft model from the same model family, an NVIDIA GPU, curl, and optionally jq. GGUF — the single-file model format used by llama.cpp; VRAM — memory on your graphics card.Most "speed up your local LLM" posts tell you to switch on speculative decoding and stop there. That advice breaks in two situations they never mention: when the server handles several requests at once, and when your prompt is long but the answer is short. Before changing a single flag, decide which number you are optimising — tokens per second on one stream (what you feel while chatting) or total throughput across concurrent requests (what matters when you expose an API). Speculative decoding — a small "draft" model guesses the next few tokens and the big model verifies them in one pass — targets the first number. Continuous batching targets the second. KV cache quantization makes nothing faster on its own; it buys VRAM so you can afford a longer context or more parallel slots.
The second thing typical guides skip: each of these three knobs has a failure mode where it makes things worse. A draft model that guesses badly adds work instead of removing it. More parallel slots raise throughput while making every individual answer slower. A quantized KV cache can shift output quality on long contexts. That is why every step below ends with a measurement, so you compare against your own baseline instead of trusting a number from someone else's machine.

nvidia-smi and confirm the driver loads and reports your GPU. If it prints NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver, fix the driver first — nothing below will work. git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build -DGGML_CUDA=ON
cmake --build build --config Release -j
Expected result: build/bin/llama-server exists.
<your-model>.gguf (the model you actually serve) and <your-draft-model>.gguf (a much smaller model from the same family, so both share a tokenizer). File sizes depend on quantization; read them off the model card, do not guess.curl for requests, jq for reading JSON, nvidia-smi for VRAM.What to do: confirm the flags your build supports before writing any config, because flag names in llama.cpp change between releases.
Why: every command later in this guide uses these names; if your build renamed one, you will debug the wrong problem for an hour.
How to check:
llama-server --help | grep -i -E 'draft|spec|parallel|cache-type|flash'
You should see one line per supported flag (for example a draft-model flag and a --cache-type-k flag). If a flag you intend to use is missing, update the build — do not invent an alias.

What to do: start the server with one slot and no speculation, then record three numbers: single-stream tokens per second, wall time for four concurrent requests, and peak VRAM.
Why: without a baseline you cannot tell whether a knob helped or hurt. A number like "2x faster" is meaningless until you know your own starting point.
How to check: start the server:
llama-server \
-m /models/<your-model>.gguf \
-c 4096 \
-np 1 \
--port 8080
-c is total context length, -np is the number of parallel slots (one slot means one request at a time). Then, in a second terminal, send one request and read the timings object the server returns:
curl -s http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"local","messages":[{"role":"user","content":"Summarize HTTP caching in 200 words."}],"max_tokens":200,"temperature":0}' \
| jq '.timings'
Expected output shape — your numbers will differ, that is the point:
{
"prompt_n": <prompt tokens>,
"prompt_per_second": <prompt processing speed>,
"predicted_n": <generated tokens>,
"predicted_per_second": <your baseline tokens/s>
}
If your build does not return timings, read the eval time lines in the server log instead — check with the documentation for your build. Now measure concurrency. With -np 1 the four requests queue behind each other:
for i in 1 2 3 4; do
curl -s -o /dev/null -w "req $i: %{time_total}s\n" \
http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"local","messages":[{"role":"user","content":"List five uses of a KV cache."}],"max_tokens":150}' &
done
wait
Write the results down in this checklist before continuing:
predicted_per_second recordednvidia-smi --query-gpu=memory.used --format=csv recorded during the run-c value written next to the numbersThe reason this works as a baseline: -np 1 removes batching from the equation, and temperature: 0 makes the two runs comparable instead of sampling different tokens each time.
What to do: restart the server with all three knobs, changing nothing else.
Why: each flag attacks a different bottleneck, so they are safe to combine but impossible to evaluate as a bundle — you will separate them in Step 3.
How to check: run:
llama-server \
-m /models/<your-model>.gguf \
-md /models/<your-draft-model>.gguf \
--draft-max 8 \
-c 16384 \
-np 4 \
--flash-attn \
--cache-type-k q8_0 \
--cache-type-v q8_0 \
--port 8080
What each part does:
| Knob | Flag | What it improves | What it costs |
|---|---|---|---|
| Speculative decoding | `-md`, `--draft-max` | tokens/s on a single stream | extra VRAM for the draft model |
| Continuous batching | `-np` (with the build's batching default) | total throughput with many clients | per-request latency |
| KV cache quantization | `--cache-type-k`, `--cache-type-v` | more context/slots in the same VRAM | small quality risk on long contexts |
| Flash attention | `--flash-attn` | memory traffic at long context | must be compiled in |
--draft-max caps how many tokens the draft model proposes per step — a higher value means more guessing per pass, which only pays off if the draft is usually right. Note that in current builds quantizing the V cache requires flash attention, and continuous batching is already the default in the server, so -np alone may be enough — confirm both points against your build's documentation.
Example of the arithmetic that trips people up: -c 16384 with -np 4 means each of the four slots has roughly 4096 tokens of context, not 16384. If your prompts are longer than that, raise -c.
One hard requirement: the draft model must use the same tokenizer and vocabulary as the target model. A draft from a different family will load, but its guesses will be rejected constantly, and you will pay the overhead for nothing. Check the model cards, not the filenames.

What to do: repeat the exact two tests from Step 1 against the new server and compare. Then remove any knob that did not move the number you care about.
Why: speculative decoding is not free — it spends VRAM and per-step compute on guesses. If you serve many concurrent clients, the win can shrink or disappear, and the honest answer is to keep batching and drop -md.
How to check: send the identical single-stream request:
curl -s http://127.0.0.1:8080/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"local","messages":[{"role":"user","content":"Summarize HTTP caching in 200 words."}],"max_tokens":200,"temperature":0}' \
| jq '.timings.predicted_per_second'
Then the same four-request loop from Step 1, plus a VRAM reading while it runs:
nvidia-smi --query-gpu=memory.used,memory.total --format=csv
Success looks like this: predicted_per_second is higher than your Step 1 value, the four-request wall time is shorter (requests now overlap instead of queuing), and memory.used still leaves headroom — if the card is at its limit, the next long prompt will kill the process.
Decision rules, applied one knob at a time:
1. Single-stream gain but concurrency gain flat → keep -md, keep -np 4.
2. Single-stream flat or slower → remove -md and re-measure; the draft model is guessing wrong.
3. Concurrency gain but answers feel slower one-to-one → lower -np to 2.
4. Quality regression on long prompts → revert --cache-type-v to the default type first.
If your target is an OpenAI-compatible endpoint that other tools can call, the server is already compatible; the wiring for a coding agent is described in How to connect a local LLM to Claude Code: llama-server and an OpenAI-compatible API.
Problem. The server refuses to start as soon as you add the draft model
error: invalid argument: -md (the wording of this message varies between builds) llama-server --help | grep -i -E 'draft|spec'
Problem. CUDA out of memory after raising -np and -c
ggml_backend_cuda_buffer_type_alloc_buffer: allocating 1024.00 MiB on device 0: cudaMalloc failed: out of memory--n-gpu-layers: llama-server -m /models/<your-model>.gguf -c 8192 -np 2 --n-gpu-layers 24 --port 8080
Problem. Clients hang and the log repeats a slot message
srv update_slots: all slots are busy (log line, wording varies)-np 1 means three of them wait.-np and raise -c with it, otherwise each slot gets an unusably small context: llama-server -m /models/<your-model>.gguf -c 32768 -np 4 --port 8080
You now have measured baselines for single-stream speed, concurrency and VRAM, and a flag set you can defend with your own numbers. Next, pin the winning command in a systemd unit or container and re-run the same two tests after every model swap — if you plan to serve several users, continue with How to Deploy a Local LLM on Kubernetes: Step-by-Step Guide.
Does speculative decoding still help when several users hit the server at once?
It helps less, and sometimes not at all — with a full batch the GPU is already busy and the draft model's extra steps compete for the same compute. Measure the four-request loop with and without -md and keep whichever wins.
Which draft model should I pick?
A much smaller model from the same family as your target, with the same tokenizer and vocabulary. Same family matters more than size: mismatched vocabularies cause constant rejections and you pay overhead with no gain.
Does quantizing the KV cache change answer quality?
It can, mostly on long contexts where errors accumulate. Test with a long prompt at temperature: 0 and compare answers before and after switching --cache-type-v back to the default.
Do I have to restart the server for every flag change?
Yes — these are startup flags, not runtime settings. Keep the previous working command in a shell script so you can roll back in one line.
Status: practical guide. Some statements come from vendors; verify in your environment.
Time to complete: 45–60 minutes, including model download and two measurement rounds. Difficulty: Intermediate (you need a working llama-server and a GPU with free VRAM).
Machine data: Citation Map JSON · JSON-LD · Semantic Embeddings — separate stage.
| Entity | Type | Role |
|---|---|---|
| как ускорить локальную LLM | тема/инструмент | упоминается в заголовке, TL;DR или разделах гайда |
| спекулятивное декодирование настройка | тема/инструмент | упоминается в заголовке, TL;DR или разделах гайда |
| ускорить генерацию токенов llama.cpp vLLM | тема/инструмент | упоминается в заголовке, TL;DR или разделах гайда |
| Windows | платформа/технология | упоминается в заголовке, TL;DR или разделах гайда |
| Linux | платформа/технология | упоминается в заголовке, TL;DR или разделах гайда |
| GitHub | платформа/технология | упоминается в заголовке, TL;DR или разделах гайда |
| CUDA | платформа/технология | упоминается в заголовке, TL;DR или разделах гайда |
| Git | платформа/технология | упоминается в заголовке, TL;DR или разделах гайда |
| NVIDIA | компания | упоминается в заголовке, TL;DR или разделах гайда |
| llama.cpp | тема/инструмент | упоминается в заголовке, TL;DR или разделах гайда |
No independent confirmation is provided; source claims and editorial conclusions are separated below.
| Source | Type | Role |
|---|---|---|
| github.com | внешний источник | ссылка из тела гайда |
Cite as: AI Release. Как ускорить локальную LLM: спекулятивное декодирование, батчинг и кэш KV. ai-release.org.