Ternary Bonsai 2 27B: Qwen3.8-27B in 5.9GB, and where the 98% stops

PrismML released Ternary Bonsai 2 27B on 17 September 2026. It is a compressed build of Qwen3.8-27B in which almost every weight is one of three values, -1, 0 or +1, with an FP16 scale shared across each group. PrismML puts it at 1.76 effective bits per weight and a total footprint of 5.9GB, more than nine times smaller than the full-precision model, and says it retains 98.2% of the original's aggregate benchmark performance. The weights are Apache 2.0, on Hugging Face and GitHub, and run on NVIDIA GPUs through CUDA and on Apple devices through MLX.

When the base model shipped I argued in what Qwen3.8-27B changes for local AI that its Apache 2.0 licence mattered more than its benchmarks. This release keeps the licence and changes the other constraint: where the model physically fits. It is worth reading carefully, including the parts of PrismML's own documentation that the headline number smooths over.

What ternary actually means#

A normal FP16 weight takes 16 bits. A 4-bit quantisation keeps an approximation of each weight in about four. Ternary goes further: each weight is reduced to one of three values, and a single FP16 scale is stored for every group of 128 weights.

The model card sets out the arithmetic. A three-way value carries log₂3, about 1.585 bits of information. Add a 16-bit scale spread across 128 weights and the format costs about 1.71 bits per weight. A small set of tensors is deliberately kept at higher precision, 0.0976% of the language model, which takes the whole thing to 1.72 in theory and about 1.76 as actually shipped.

Two engineering details matter for anyone deploying it.

The weights live in a rotated basis. Each matrix is transformed with a blockwise Hadamard rotation before ternary assignment, and the runtime must apply the matching transform to activations. That is also why ordinary runtimes cannot load it.

There are two packings. PTQ1_0 packs the three-way values densely at 5.95GB. PQ2_0 stores each one in a 2-bit slot at 7.21GB. Neither is uniformly faster: the model card says PTQ1_0 wins on Ada-generation cards and the L4, and PQ2_0 wins on H100, A100 and Blackwell. The vision tower ships separately as an optional 0.63GB file, loaded only when an image arrives. Context stays at 262K tokens.

The hardware line it moves#

This is the reason to care.

PrismML's whitepaper puts it plainly: on a laptop with 16GB of RAM, the full-precision model at 54GB and even the Q4_K_XL build at 17.6GB do not fit. Both Bonsai packings do. A 4-bit build of this model needs more memory than a 16GB machine has, and a ternary one does not.

PrismML's first Bonsai 27B, released in July, already crossed that line. What this release changes is the cost of crossing it: a stronger base model, and a retention gap that PrismML says has closed from 95% to over 98%.

PrismML's throughput figures, all at batch size 1, show what that buys:

• Apple M5 Pro laptop: about 27.7 tokens per second
• Apple M5 Max laptop: 46.8 tokens per second
• RTX 4090: up to 96.7 tokens per second
• RTX 5090: up to 142.5 tokens per second
• NVIDIA L4 at 72W: about 32 tokens per second

I run agents on a Mac Studio, and the class of machine I have been considering for client deployments is much smaller, which I wrote about in why I am interested in Mac minis for local AI. The question with small machines has always been which models fit with room to spare. A 27B-class model in under 6GB of weights changes that answer.

One caution before anyone sizes hardware from the headline. The 5.9GB is the language model's weights alone. The key-value cache grows with the context you actually use, the operating system needs its share, and a 262K context is not free. PrismML lists sub-4-bit KV cache compression as roadmap work, not as something in this release. Size for the context you will run, not for the weights.

98.2% of what, exactly#

The headline figure comes from a 20-benchmark suite covering knowledge and reasoning, maths, coding, instruction following, tool calling and vision. Bonsai 2 averages 83.9 against 85.4 for the full-precision Qwen3.8-27B. It is also slightly above Qwen3.6-27B at full precision, which averages 83.6.

The comparison most people deploying locally actually need is against the 4-bit build they already run, and the model card has it, on a separate 14-benchmark set in thinking mode:

Build	Bits per weight	Size	Average
Qwen3.8-27B FP16	16.0	54GB	86.32
Qwen3.8-27B UD-Q4_K_XL ("4-bit")	5.2	17.6GB	85.18
Qwen3.8-27B IQ2_XXS ("2-bit")	2.16	7.27GB	72.59
Ternary Bonsai 2 27B	1.72	5.95GB	84.78

On that set, Bonsai 2 is within 0.4 points of the 4-bit build at a third of its size, and far ahead of the conventional 2-bit build at a smaller size.

The losses are not spread evenly. By PrismML's own category table, maths, coding and instruction following are essentially level. The gaps are in knowledge and reasoning, 83.95 against 86.66, and vision, 78.59 against 81.64. Individual benchmarks show it more sharply: GPQA Diamond falls from 90.51 to 85.76, and OCR Bench v2 from 60.99 to 56.88. If your workload is document reading or knowledge-heavy question answering, the aggregate flatters you.

Where the 98% stops#

The most important numbers in the release are in the whitepaper, not the announcement.

PrismML evaluated Bonsai 2 on two long-horizon agent benchmarks for the first time. On Terminal-Bench 2.1 it scores 52.8 against 69.7 for the full model. On SWE-bench Verified it scores 60.8 against 80.6. PrismML's own summary is that it retains "roughly three quarters of the full-precision performance on both benchmarks."

PrismML presents this as progress, and by its own account it is. The whitepaper notes that models at this footprint often make no meaningful progress on such tasks at all, and on τ²-Bench the new model reaches 80.2, up from 73.6 in the previous release. Both things are true. It is much better than it was, and on multi-step agent work it is still a quarter behind the model it came from.

The reasoning-effort catch#

Every headline result is measured at xhigh reasoning effort, the model's default and its most expensive setting. The whitepaper also publishes the full suite at medium, and the gap widens there. The average falls to 79.3 against 82.6 for the full model. τ²-Bench goes from 80.22 at xhigh to 70.86 at medium, against 79.50 for the full model at the same setting. AIME26 drops to 80.42 against 87.92, and GPQA Diamond to 75.56 against 84.34.

There is no cheaper floor either. The model card says low reasoning effort is not supported and, if selected, the model behaves close to xhigh.

This matters because of where the model is aimed. On a laptop generating around 28 tokens a second, long reasoning traces are slow in wall-clock time. The model card's own example leaves 16,384 tokens of room for the thinking trace. If a trace used all of that at 28 tokens a second, it would take close to ten minutes. On exactly the hardware this release makes viable, you will be tempted to turn reasoning down, and turning it down is where the quality gap grows.

The runtime is now a dependency you own#

The model card is blunt about this: "Stock llama.cpp will not run these files." It rejects the two packing formats as unknown types, and it "loads Q2_0 without any warning and produces garbage, because it has no Hadamard activation runtime."

Running Bonsai 2 means running PrismML's forks of llama.cpp, MLX or mlx-swift. The model card names its demo repository as "the source of truth for running these models" and says that where the card disagrees with it, the repository is right.

That is a reasonable way for a small team to ship fast-moving kernels. It is also a fork in the inference path of whatever you build, and someone has to track it, pin it and notice when it moves. A runtime that silently produces garbage is not a failure a monitoring dashboard catches.

Note too that the quality benchmarks ran through vLLM on NVIDIA H100s, not the llama.cpp Metal path a laptop deployment would use. And PrismML's two documents disagree in places: the model card and the whitepaper give different scores for the same conventional 2-bit build on the same benchmark, AIME26 at 57.5 in one and 78.6 in the other. Every number here is a claim about a specific harness on a specific day.

How I would test it before replacing a 4-bit quant#

If you already run Qwen3.8-27B at 4-bit and this looks tempting, a short evaluation settles it properly.

1. Run both on your own inputs. Fifty real cases from your domain, with a written definition of a correct answer before you start. Side by side, current 4-bit build against Bonsai 2.
2. Test at the reasoning effort you will actually use. If latency will push you to medium, evaluate at medium. The published gap there is wider.
3. Include long agent runs if you run agents. Multi-step tasks where the model calls tools and has to recover from its own mistakes. That is where the published retention drops to three quarters.
4. Test documents and images separately. Vision and OCR are where the category scores fall most.
5. Measure on the target machine at real context length. Weights, KV cache and everything else resident, on the hardware you would buy.
6. Pin the runtime. Pin the fork version, and confirm that a wrong file fails loudly in your setup rather than generating nonsense.

Where I think it fits is a small on-site machine doing bounded work such as drafting, classification, extraction and short tool calls, with data that should not leave the building. That is the split I described in the case for hybrid local and cloud AI. For long autonomous agent runs I would still want the full model, or a hosted one.

What to watch next#

Three things. Whether the ternary formats and the Hadamard runtime land in upstream llama.cpp, which would remove the fork dependency. Whether anyone independent reproduces the benchmark numbers, especially the agent ones. And the KV cache work on PrismML's roadmap, because on a 16GB machine the cache, not the weights, is what will decide how much of that 262K context is actually usable.