RU

Bonsai 2 27B on 8 GB VRAM Loses to Qwen3 8B: A Test on the P104-100

Published: 2026-11-16 · Author: AI Release · @ai_release1
Bonsai 2 27B on 8 GB VRAM Loses to Qwen3 8B: A Test on the P104-100

⚡ The gist

🔍 What happened

PrismML released Bonsai 27B in mid-July and claimed it was the first model of its size to fit on an iPhone. On September 17, Bonsai 2 27B arrived — this time ternary. The developers shrank 54 GB down to 5.9 GB — a ninefold reduction — and promise 98.2% of the original model's quality. The July version has been downloaded roughly 5 million times from Hugging Face. SkaldByte, writing in MTS's blog on Habr, put it through its paces on hardware that usually ends up in a landfill.

The mechanics are simple on paper. In a regular model, every weight is a 16-bit number; a ternary one has just three values: −1, 0, and +1. No multiplication needed — addition and sign flips are enough. Under the hood it's Qwen3-27B under Apache 2.0, with a rated context of 262K tokens and hybrid attention: roughly three-quarters of the layers are linear, which makes long-text processing cheaper than full attention. Size-wise: FP16 is about 54 GB (an 80 GB card), Q4 is roughly 15.4 GB (24 GB), PQ2_0 is 7.2 GB, and PTQ1_0 is 5.9 GB — both fit under 8 GB. PrismML says PTQ1_0 is faster on gaming cards like the RTX 4090, while PQ2_0 wins on H100, A100, and Blackwell. The first snag showed up on PC right away: stock llama.cpp and Ollama can't parse the ternary GGUF — the weights live in PQ2_0 and PTQ1_0 tensors, and only PrismML's fork understands those. The author ran a CUDA 12.4 build, benchmarked with llama-bench, polled the card's temperature, power draw, and memory usage every half second via nvidia-smi, and drove the phones over adb.

🧠 Our take

Ternary weights aren't magic — they're extremely aggressive compression. Old cards read every weight to produce a single token, so speed is bottlenecked by memory bandwidth, not compute. Hence the logic of the race: not "make it smarter," but "make the file fit in 8 GB." Our take: the 98.2% figure is a marketing benchmark from the developers themselves, and it shouldn't be judged in isolation from specific tasks.

What does this change for someone with an ancient card? On paper, the options just widened: 27B in 8 GB genuinely runs. In practice, between "fits" and "works" lies a pile of glue — a fork, patches, a specific Ollama version. Qwen3 8B fits into the same 8 GB with no forks or patches and, per the author's measurements, held its own — sometimes running faster and more stably. What bothers me here isn't the result but how it's framed: "27B on 8 GB" sounds like an upgrade, but in reality it's only an upgrade for certain workloads and with a lot of patience.

PrismML wins: five million downloads of the July version is real attention, not a line in a README. The llama.cpp ecosystem wins too — some of Bonsai's formats have already been merged there. The loser is anyone who falls for the file size and expects miracles from a 2017 card. From experience, old Pascal without AVX2 on the CPU side also means slow prompt processing — something the benchmark will honestly reveal if you look beyond generation speed.

⚠️ What we don't know

The source gives no exact tokens/sec figures per model or task — just the claim "on par, sometimes faster and more stable." One rig, one author; the methodology is described but hasn't been independently reproduced. The 98.2% quality claim comes straight from PrismML. There's no data in the excerpt on temperatures or throttling on the mining card. If you plan to replicate this, check for yourself: which Ollama version you're running and what tensor type is in the file.

💬 Worth debating

  • 27B in 8 GB with a fork, or 8B with zero patches: which would you pick for your workload?
  • Should "98.2% quality" count as an argument when the number comes from the model's own team?
  • Are ternary weights here to stay, or just an interim trick until the next quantization format?

Forecast: within a quarter, ternary tensors like PQ2_0 will land in mainline llama.cpp, and the debate will shift from "does it even run" to "how many tokens/sec on old hardware."

Disclaimer: one source, one author's measurements on one rig — not officially verified.

Source: Habr / MTS blog

🤖 AI summary
  • Date: 2026-11-16
  • Source: habr.com
  • Tags: local, троичные, llama.cpp, старое, AI, LLM, neural networks, Qwen3
  • Key facts:
    • PrismML's ternary Bonsai 2 27B technically fits in 8 GB of VRAM, but on old hardware it loses to plain Qwen3 8B.
    • Fast ternary PQ2_0 tensors only work in PrismML's fork: stock Ollama 0.40.6 can't read the file.
    • The author's test rig: a 2017 mining-grade P104-100, an i7-3770 without AVX2, 15.5 GB of RAM, plus two budget TECNO phones.
    • PrismML released Bonsai 27B in mid-July and claimed it was the first model of its size to fit on an iPhone.
    • On September 17, Bonsai 2 27B arrived — this time ternary.
    • The developers shrank 54 GB down to 5.9 GB — a ninefold reduction — and promise 98.2% of the original model's quality.
    • The July version has been downloaded roughly 5 million times from Hugging Face.
    • SkaldByte, writing in MTS's blog on Habr , put it through its paces on hardware that usually ends up in a landfill.
#локальные#троичные#llama.cpp#старое#AI#LLM#нейросети#Qwen3
← PreviousGeneva Weighs the Fate of 'Killer Robots': US and Russia Oppose a BanNext →AI Digest 11.16.2026: AI-Driven Scientific Progress… and 4 More

Source: habr.com · post in Telegram