RU

Lightweight LLMs on a GPU-less VPS: 4B models hit 90% accuracy in 3–5 seconds

Published: 2026-11-16 · Author: AI Release · @ai_release1
Lightweight LLMs on a GPU-less VPS: 4B models hit 90% accuracy in 3–5 seconds

⚡ The gist

🔍 What happened

Habr user vechersky published part two of his deep dive into lightweight LLMs on CPU. The test rig: a JustHost VPS with an Intel Xeon Gold 6148, 5 vCPUs, 8 GB RAM, 4 GB swap, Ubuntu 24.04, Ollama 0.35.0 in Docker, running a llama.cpp build of 161755f29. To the models from part one (QVikhr-3-4B, qwen3.5:2b and 4b, gemma3:4b, granite4:micro-h) he added five larger ones, all pulled from Hugging Face in Q4_K_M quantization: GigaChat 3.1 Lightning 10B at 6.5 GB (MoE, with only 1.8B parameters active per token), T-lite-it-2.1 8B at 5.0 GB, YandexGPT-5-Lite-8B-instruct at 4.9 GB, Qwen3-8B at 5.0 GB, and GLM-4-9B-0414 at 6.2 GB.

The headline number: 326 tickets, with nine out of ten models landing between 87% and 92% accuracy. In the first round there were only 20 examples, and seven models scored a perfect 100% — the gap between those two tests is the article's main takeaway. Now the mechanics. If every request starts with the same instruction, the model caches it and only reads the variable part — so you put the shared instruction at the top of the prompt and the ticket text at the end. Separately, the author tested the decision model d1-2B: accuracy-wise it matches the generative models, but it outputs an honest probability — and with a 0.9 threshold it would auto-handle 69% of tickets at 97% accuracy, routing the rest to a human.

🧠 Our take

The change in test scale is the most valuable thing here. Twenty examples and 326 examples give different answers to the same question — and that's about methodology, not the models. Our take: anyone picking an LLM based on a five-prompt demo is buying a lottery ticket. The 2–3 point spread between the leaders is within statistical noise, so you should be deciding on speed and memory footprint, not on a line in a table.

The second takeaway hits the "bigger is smarter" marketing head-on. On CPU, the 8–10B models didn't outperform the best 4B ones — and they ran slower; the only speed exception was GigaChat 3.1 Lightning thanks to MoE. We think that, for background text processing, this is effectively the death sentence for the "let's run a big model on a weak machine" idea. Classification in 3–5 seconds is fine; knowledge-base answers and summaries in 0.5–2.5 minutes are fine for a queue, but not for a live customer chat. Going from 4 to 8 GB of RAM eliminated out-of-memory crashes on long texts, and the fifth vCPU added 20–25% to prompt ingestion and about 10% to generation: memory here is cheaper than cores.

Who wins — VPS providers and teams for whom 8 GB is enough. Who loses — anyone selling "you need a GPU" for routine work. But let's be honest: one rig, one provider, one private set of tickets. I wouldn't generalize to your workload without re-testing.

⚠️ What we don't know

The source is one author, one VPS, and one set of 326 tickets — we don't know what those texts actually were or how much they resemble yours. Steal averaged 0.2% over 41 hours with a 6.4% peak — decent numbers, but your host neighbors change. There's no comparison against quantizations other than Q4_K_M and no measurement at 16 GB of RAM, so "8 GB is enough" applies to this rig, not to every workload. You need to validate on your own hundreds of examples.

💬 Worth debating

  • Run a 4B model and live with 90% accuracy, or use d1-2B with a threshold and hand the ambiguous cases to a human?
  • Save on memory (4 GB) and risk OOM crashes on long texts, or just get 8 GB and stop worrying?
  • Is it even worth hosting an LLM on your own VPS when the same classification costs pennies via an API?

Forecast: over the next quarter I expect a wave of similar "4B on 8 GB" benchmarks with published Ollama configs — the topic is too cheap not to get double-checked.

Disclaimer: a single source — one author's benchmarks on one rig, not officially verified.

Source: Habr

🤖 AI summary
  • Date: 2026-11-16
  • Source: habr.com
  • Tags: LLM, Ollama, VPS, Benchmarks, AI, neural networks, MachineLearning, ChatGPT
  • Key facts:
    • On a VPS with 8 GB of RAM and 5 vCPUs, 2–4B models classify support tickets in 3–5 seconds with roughly 90% accuracy.
    • Prompt caching in Ollama" speeds up responses 2–3x: QVikhr-3-4B drops from 12.4 seconds to 3.8 with the cache on.
    • The 8–10B models never beat the 4B ones on speed; on 8 GB of RAM the default cache eats memory — the author recommends LLAMA_ARG_CACHE_RAM=1024 .
    • Habr user vechersky published part two of his deep dive into lightweight LLMs on CPU.
    • The test rig: a JustHost VPS with an Intel Xeon Gold 6148, 5 vCPUs, 8 GB RAM, 4 GB swap, Ubuntu 24.04, Ollama 0.35.0 in Docker, running a llama.cpp build of 161755f29 .
    • To the models from part one (QVikhr-3-4B, qwen3.5:2b and 4b, gemma3:4b, granite4:micro-h) he added five larger ones, all pulled from Hugging Face in Q4_K_M quantization: GigaChat 3.1 Lightning 10B at 6.5 GB (MoE, with only 1.8B parameters active per token), T-lite-it-2.1 8B at 5.0 GB, YandexGPT-5-Lite-8B-instruct at 4.9 GB, Qwen3-8B at 5.0 GB, and GLM-4-9B-0414 at 6.2 GB.
    • The headline number: 326 tickets, with nine out of ten models landing between 87% and 92% accuracy.
    • In the first round there were only 20 examples, and seven models scored a perfect 100% — the gap between those two tests is the article's main takeaway.
#LLM#Ollama#VPS#бенчмарки#AI#нейросети#MachineLearning#ChatGPT
← PreviousCalifornia Sues OpenAI Over AI Agents Attacking Hugging Face

Source: habr.com · post in Telegram