{
  "schema_version": "1.0",
  "publisher": "AI Release",
  "article": {
    "title": "How to speed up a local LLM: speculative decoding, batching and KV cache",
    "canonical_url": "https://ai-release.org/guides/kak-uskorit-lokalnuyu-llm-spekulyativnoe-dekodirovanie_en.html",
    "language": "en",
    "article_type": "guide",
    "published_at": null,
    "updated_at": "2026-11-17T00:00:00+00:00"
  },
  "summary": "Time to complete: 45–60 minutes, including model download and two measurement rounds. Difficulty: Intermediate (you need a working llama-server and a GPU with free VRAM).",
  "topics": [
    "как ускорить локальную LLM",
    "спекулятивное декодирование настройка",
    "ускорить генерацию токенов llama.cpp vLLM"
  ],
  "entities": [
    {
      "id": "E1",
      "name": "как ускорить локальную LLM",
      "type": "тема/инструмент",
      "description": "упоминается в заголовке, TL;DR или разделах гайда",
      "url": null
    },
    {
      "id": "E2",
      "name": "спекулятивное декодирование настройка",
      "type": "тема/инструмент",
      "description": "упоминается в заголовке, TL;DR или разделах гайда",
      "url": null
    },
    {
      "id": "E3",
      "name": "ускорить генерацию токенов llama.cpp vLLM",
      "type": "тема/инструмент",
      "description": "упоминается в заголовке, TL;DR или разделах гайда",
      "url": null
    },
    {
      "id": "E4",
      "name": "Windows",
      "type": "платформа/технология",
      "description": "упоминается в заголовке, TL;DR или разделах гайда",
      "url": null
    },
    {
      "id": "E5",
      "name": "Linux",
      "type": "платформа/технология",
      "description": "упоминается в заголовке, TL;DR или разделах гайда",
      "url": null
    },
    {
      "id": "E6",
      "name": "GitHub",
      "type": "платформа/технология",
      "description": "упоминается в заголовке, TL;DR или разделах гайда",
      "url": null
    },
    {
      "id": "E7",
      "name": "CUDA",
      "type": "платформа/технология",
      "description": "упоминается в заголовке, TL;DR или разделах гайда",
      "url": null
    },
    {
      "id": "E8",
      "name": "Git",
      "type": "платформа/технология",
      "description": "упоминается в заголовке, TL;DR или разделах гайда",
      "url": null
    },
    {
      "id": "E9",
      "name": "NVIDIA",
      "type": "компания",
      "description": "упоминается в заголовке, TL;DR или разделах гайда",
      "url": null
    },
    {
      "id": "E10",
      "name": "llama.cpp",
      "type": "тема/инструмент",
      "description": "упоминается в заголовке, TL;DR или разделах гайда",
      "url": null
    }
  ],
  "claims": [
    {
      "id": "AR-GUIDE-KAKUSKORIT-001",
      "statement": "Time to complete: 45–60 minutes, including model download and two measurement rounds.",
      "claim_type": "guide_fact",
      "confidence": "not_assessed",
      "source_type": "заявление источника",
      "verification_status": "reported_unverified",
      "source_url": "https://ai-release.org/guides/kak-uskorit-lokalnuyu-llm-spekulyativnoe-dekodirovanie_en.html",
      "article_anchor": "#tldr",
      "evidence_excerpt": "Time to complete: 45–60 minutes, including model download and two measurement rounds."
    },
    {
      "id": "AR-GUIDE-KAKUSKORIT-002",
      "statement": "Difficulty: Intermediate (you need a working llama-server and a GPU with free VRAM).",
      "claim_type": "guide_fact",
      "confidence": "not_assessed",
      "source_type": "заявление источника",
      "verification_status": "reported_unverified",
      "source_url": "https://ai-release.org/guides/kak-uskorit-lokalnuyu-llm-spekulyativnoe-dekodirovanie_en.html",
      "article_anchor": "#tldr",
      "evidence_excerpt": "Difficulty: Intermediate (you need a working llama-server and a GPU with free VRAM)."
    },
    {
      "id": "AR-GUIDE-KAKUSKORIT-003",
      "statement": "You will need: a llama.cpp build that contains llama-server (verify with llama-server --version), a GGUF target model, a smaller GGUF draft model from the same model family, an NVIDIA GPU, curl, and optionally jq. GGUF — the single-file model format used by llama.cpp; VRAM — memory on your graphics card.",
      "claim_type": "guide_fact",
      "confidence": "not_assessed",
      "source_type": "заявление источника",
      "verification_status": "reported_unverified",
      "source_url": "https://ai-release.org/guides/kak-uskorit-lokalnuyu-llm-spekulyativnoe-dekodirovanie_en.html",
      "article_anchor": "#tldr",
      "evidence_excerpt": "You will need: a llama.cpp build that contains llama-server (verify with llama-server --version), a GGUF target model, a smaller GGUF draft model from the same model family, an NVIDIA GPU, curl, and optionally jq. GGUF — the single-file model format used by llama.cpp; VRAM — memory on your graphics card."
    },
    {
      "id": "AR-GUIDE-KAKUSKORIT-004",
      "statement": "OS and driver: Linux x86_64, or Windows 11 with WSL2. Run nvidia-smi and confirm the driver loads and reports your GPU. If it prints NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver, fix the driver first — nothing below will work.",
      "claim_type": "guide_fact",
      "confidence": "not_assessed",
      "source_type": "заявление источника",
      "verification_status": "reported_unverified",
      "source_url": "https://ai-release.org/guides/kak-uskorit-lokalnuyu-llm-spekulyativnoe-dekodirovanie_en.html",
      "article_anchor": "#requirements",
      "evidence_excerpt": "OS and driver: Linux x86_64, or Windows 11 with WSL2. Run nvidia-smi and confirm the driver loads and reports your GPU. If it prints NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver, fix the driver first — nothing below will work."
    },
    {
      "id": "AR-GUIDE-KAKUSKORIT-005",
      "statement": "llama.cpp with CUDA: build it yourself so GPU offload is compiled in:",
      "claim_type": "guide_fact",
      "confidence": "not_assessed",
      "source_type": "заявление источника",
      "verification_status": "reported_unverified",
      "source_url": "https://ai-release.org/guides/kak-uskorit-lokalnuyu-llm-spekulyativnoe-dekodirovanie_en.html",
      "article_anchor": "#requirements",
      "evidence_excerpt": "llama.cpp with CUDA: build it yourself so GPU offload is compiled in:"
    },
    {
      "id": "AR-GUIDE-KAKUSKORIT-006",
      "statement": "Models: two GGUF files — <your-model>.gguf (the model you actually serve) and <your-draft-model>.gguf (a much smaller model from the same family, so both share a tokenizer). File sizes depend on quantization; read them off the model card, do not guess.",
      "claim_type": "guide_fact",
      "confidence": "not_assessed",
      "source_type": "заявление источника",
      "verification_status": "reported_unverified",
      "source_url": "https://ai-release.org/guides/kak-uskorit-lokalnuyu-llm-spekulyativnoe-dekodirovanie_en.html",
      "article_anchor": "#requirements",
      "evidence_excerpt": "Models: two GGUF files — <your-model>.gguf (the model you actually serve) and <your-draft-model>.gguf (a much smaller model from the same family, so both share a tokenizer). File sizes depend on quantization; read them off the model card, do not guess."
    }
  ],
  "relationships": [
    {
      "subject": "Guide",
      "predicate": "describes",
      "object": "How to speed up a local LLM: speculative decoding, batching and KV cache",
      "claim_id": "AR-GUIDE-KAKUSKORIT-001"
    },
    {
      "subject": "Guide",
      "predicate": "includes step",
      "object": "Step 1. Record a baseline before touching any flag",
      "claim_id": "AR-GUIDE-KAKUSKORIT-001"
    }
  ],
  "questions_and_answers": [
    {
      "question": "What is required?",
      "answer": "OS and driver: Linux x86_64, or Windows 11 with WSL2. Run nvidia-smi and confirm the driver loads and reports your GPU. If it prints NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver, fix the driver first — nothing below will work.; llama.cpp with CUDA: build it yourself so GPU offload is…",
      "claim_ids": [
        "AR-GUIDE-KAKUSKORIT-003"
      ],
      "article_anchor": "#requirements"
    },
    {
      "question": "How to do it?",
      "answer": "Step 1. Record a baseline before touching any flag → Step 2. Turn on speculative decoding, batching and KV cache quantization → Step 3. Re-measure and decide which knobs to keep",
      "claim_ids": [
        "AR-GUIDE-KAKUSKORIT-004"
      ],
      "article_anchor": "#step-1"
    },
    {
      "question": "Which topics does it cover?",
      "answer": "как ускорить локальную LLM, спекулятивное декодирование настройка, ускорить генерацию токенов llama.cpp vLLM",
      "claim_ids": [
        "AR-GUIDE-KAKUSKORIT-002"
      ],
      "article_anchor": "#entities"
    },
    {
      "question": "What are the key points?",
      "answer": "Time to complete: 45–60 minutes, including model download and two measurement rounds. Difficulty: Intermediate (you need a working llama-server and a GPU with free VRAM).",
      "claim_ids": [
        "AR-GUIDE-KAKUSKORIT-007"
      ],
      "article_anchor": "#tldr"
    }
  ],
  "limitations": [
    "Current versions, prices and limits.",
    "Independent benchmarks and quality checks.",
    "Terms of use and data policy."
  ],
  "citation": {
    "preferred_citation": "AI Release. Как ускорить локальную LLM: спекулятивное декодирование, батчинг и кэш KV. https://ai-release.org/guides/kak-uskorit-lokalnuyu-llm-spekulyativnoe-dekodirovanie_en.html",
    "canonical_url": "https://ai-release.org/guides/kak-uskorit-lokalnuyu-llm-spekulyativnoe-dekodirovanie_en.html",
    "publisher": "AI Release"
  }
}