LLAMA_ARG_CACHE_RAM=1024.Habr user vechersky published part two of his deep dive into lightweight LLMs on CPU. The test rig: a JustHost VPS with an Intel Xeon Gold 6148, 5 vCPUs, 8 GB RAM, 4 GB swap, Ubuntu 24.04, Ollama 0.35.0 in Docker, running a llama.cpp build of 161755f29. To the models from part one (QVikhr-3-4B, qwen3.5:2b and 4b, gemma3:4b, granite4:micro-h) he added five larger ones, all pulled from Hugging Face in Q4_K_M quantization: GigaChat 3.1 Lightning 10B at 6.5 GB (MoE, with only 1.8B parameters active per token), T-lite-it-2.1 8B at 5.0 GB, YandexGPT-5-Lite-8B-instruct at 4.9 GB, Qwen3-8B at 5.0 GB, and GLM-4-9B-0414 at 6.2 GB.
The headline number: 326 tickets, with nine out of ten models landing between 87% and 92% accuracy. In the first round there were only 20 examples, and seven models scored a perfect 100% — the gap between those two tests is the article's main takeaway. Now the mechanics. If every request starts with the same instruction, the model caches it and only reads the variable part — so you put the shared instruction at the top of the prompt and the ticket text at the end. Separately, the author tested the decision model d1-2B: accuracy-wise it matches the generative models, but it outputs an honest probability — and with a 0.9 threshold it would auto-handle 69% of tickets at 97% accuracy, routing the rest to a human.
The change in test scale is the most valuable thing here. Twenty examples and 326 examples give different answers to the same question — and that's about methodology, not the models. Our take: anyone picking an LLM based on a five-prompt demo is buying a lottery ticket. The 2–3 point spread between the leaders is within statistical noise, so you should be deciding on speed and memory footprint, not on a line in a table.
The second takeaway hits the "bigger is smarter" marketing head-on. On CPU, the 8–10B models didn't outperform the best 4B ones — and they ran slower; the only speed exception was GigaChat 3.1 Lightning thanks to MoE. We think that, for background text processing, this is effectively the death sentence for the "let's run a big model on a weak machine" idea. Classification in 3–5 seconds is fine; knowledge-base answers and summaries in 0.5–2.5 minutes are fine for a queue, but not for a live customer chat. Going from 4 to 8 GB of RAM eliminated out-of-memory crashes on long texts, and the fifth vCPU added 20–25% to prompt ingestion and about 10% to generation: memory here is cheaper than cores.
Who wins — VPS providers and teams for whom 8 GB is enough. Who loses — anyone selling "you need a GPU" for routine work. But let's be honest: one rig, one provider, one private set of tickets. I wouldn't generalize to your workload without re-testing.
The source is one author, one VPS, and one set of 326 tickets — we don't know what those texts actually were or how much they resemble yours. Steal averaged 0.2% over 41 hours with a 6.4% peak — decent numbers, but your host neighbors change. There's no comparison against quantizations other than Q4_K_M and no measurement at 16 GB of RAM, so "8 GB is enough" applies to this rig, not to every workload. You need to validate on your own hundreds of examples.
Forecast: over the next quarter I expect a wave of similar "4B on 8 GB" benchmarks with published Ollama configs — the topic is too cheap not to get double-checked.
Disclaimer: a single source — one author's benchmarks on one rig, not officially verified.
Source: Habr
Source: habr.com · post in Telegram