125B MoE Under the Desk: A Local LLM Delivers 25 Tokens/s Without the Cloud

⚡ The Gist in 5 Seconds

• A Habr author runs a 125B MoE model on a workstation: 22–25 tokens/s generation and 500+ t/s prefill.
• Quality — 93.1% agreement with an 8-bit reference, and not a single byte of data leaves the perimeter.
• The numbers come not from hardware but from tuning: a quant matched to the VRAM layout, a fresh engine, and a speculative decoder.

🔍 What Happened

User 4KAP posted on Habr a breakdown of his path to local inference. His clients are state corporations, banks, and industrial holdings — places where "trade secrets" appear on every other document, and the first question isn't "how much does it cost" but "where does the data go." The answer, in numbers: a 125B MoE lives on a workstation, delivering 22–25 tokens per second on generation and 500+ t/s on prefill, with quality at 93.1% agreement with an 8-bit reference.

Then the engineering begins. The quant is chosen not by quality but by the VRAM layout. The engine version is the cheapest upgrade: +79% to decode speed, but recalibration is needed after every update. On Windows, watch the commit, not the RAM: the pagefile gives +48%. The KV cache is nearly free; parallel slots are not. The author calls the speculative decoder a must-have. He's also honest about masking: a mandatory layer, but only partial — code, configs, and metadata have nothing to mask them with, and masked context can be reconstructed from indirect signals.

🧠 Our Take

"The security requirement isn't a whim" — that's the article's main thesis, and I agree with it. The debate over local inference has been stuck for years on "slow and expensive." Here, one person shows that 22–25 t/s is enough for an agent to write code, dig through incident traces, and build prototypes. Our assessment: the turning point came not from a new model, but from engines and proper tuning. Quant, slots, speculative decoder — this is now a craft, not "install and run."

The choice used to be binary: cloud or nothing. Now a third option is emerging — hybrid. The author predicts it himself: cloud for design and complex decisions, local for bulk waves and anything sensitive. We believe this is the most plausible scenario for dev teams. The real question is different: who will pay for the hardware and the engineer's time spent tuning this for months.

The winners are those with security breathing down their necks — banks, industry, the public sector. Cloud providers, it seems, aren't losing the whole market, just its sensitive part — and that's already significant. I'll also highlight the argument about the fallback channel: properly configured processes survive lost subscriptions, pulled deployments, and even an internet outage. That's the kind of thing people usually think about after a disaster, not before.

⚠️ What We Don't Know

The model isn't named — only "125B MoE." The hardware comes without specs: no CPU, no GPU, no VRAM size. "500+ t/s prefill" says little without a prompt length. The methodology behind the "93.1% agreement" figure isn't disclosed in the quoted excerpt. And this is one person with one workflow, not an independent test: someone else's harness won't work identically for you — you'll have to verify on your own tasks.

Prediction: within a year, at least one major client from banking or state corporations will write "agents inside the perimeter" into contractor requirements — by analogy with the domestic software registry. Verifiable via tender documents.

Disclaimer: a single source — the author's personal experience; no independent measurements.

Source: Habr