TL;DR

• Full ShapeLearn moves the measured quality-speed frontier beyond Lite. All five models in the new release sit on the frontier in each of our six GPU comparisons.
• GPU-5 is our default recommendation wherever it fits, reaching 99.63% of BF16’s aggregate benchmark score. If it does not fit with the context you need, GPU-4 is still very competitive: it reaches 98.72% of BF16 at a much smaller size (11.0 GB instead of 13.1 GB), and it is faster.
• ShapeLearn-Lite also performed better than its KLD ranking suggested: three of its six models sit on the frontier in the Lite-versus-Unsloth Dynamic v3 comparison.
• Speculative Decoding with MTP or DFlash2 increases throughput across every ShapeLearn model and GPU tested. DFlash2 is usually faster but requires more memory and does not support image inputs with llama.cpp. Choose DFlash2 for maximum text-only throughput when memory allows, and MTP when VRAM or multimodal support matters more.

Full ShapeLearn moves the frontier

We are releasing the full ShapeLearn run for Qwen 3.8 27B.

Within this release, larger models yield higher aggregate scores, while smaller models deliver higher throughput. That ordering holds across all six GPUs tested. Because this is a dense model and memory transfers are the bottleneck, lower BPW translates more directly into higher TPS than it does for MoEs.

The per-GPU comparisons also include AtomicChat, Bartowski, ISTA-DASLab, and Unsloth Dynamic v3. Bartowski’s latest models were released after our testing and are not included. Full ShapeLearn is labelled ByteShape in the figures.

All five ShapeLearn models remain on the measured frontier, with GPU-5 achieving the highest aggregate score among the plotted quants. Other teams also contribute competitive points. Notably, ISTA-DASLab’s excellent model (the yellow “d” on the graph below) also sits on the frontier.

By “frontier,” we mean that no other plotted model is both faster and more accurate.

96 GB: RTX Pro 6000

RTX PRO 6000, the GPU with the most memory, lets us show the full range of models tested.

GPU-5 is our default wherever you can fit it: it reaches 90.4 tok/s at 99.63% of the BF16 baseline.

32 GB: RTX 5090

The RTX 5090 tells a similar story, leading to the same recommendations.

Once again GPU-5 is our default choice, reaching 93.7 tok/s. Choose GPU-4 for slightly more context length or slightly better TPS.

24 GB: RTX 4090 and RTX 3090

Both 24 GB cards fit all five ShapeLearn models. We plot them separately because their throughput differs, but the ordering is the same on both.

RTX 4090

The RTX 4090 keeps the same pattern: GPU-5 is the default, reaching 59.2 tok/s.

RTX 3090

Older, but still fast in these measurements.

GPU-4 reaches 49.5 tok/s, compared with 45.8 tok/s for GPU-5. Moving to the larger model costs about 7.5% in throughput, while the aggregate score rises from 98.72% to 99.63% of BF16. That makes GPU-5 the default here as well.

16 GB: RTX 4080 and RTX 5060 Ti

With a tighter VRAM budget, the pragmatic choice is to leave room for the context you need, not just the model weights. These plots contain fewer competing configurations, but all five ShapeLearn models are represented.

RTX 4080

On the RTX 4080, GPU-4 reaches 52.4 tok/s, while GPU-5 reaches 45.7 tok/s.

RTX 5060 Ti

On the RTX 5060 Ti, the corresponding figures are 33.1 tok/s and 29.1 tok/s.

GPU-5 remains the default on both cards when the model, KV cache, and runtime buffers fit within your memory budget. When they do not, GPU-4 is still very competitive: almost 99% of BF16 at a much smaller size, and faster. A model appearing in these measurements does not establish that every context length or serving configuration will fit.

ShapeLearn-Lite, in retrospect

ShapeLearn-Lite uses a smaller optimization budget than full ShapeLearn. It let us get Qwen 3.8 27B onto 12 GB to 24 GB GPUs within a few days.

We released after targeted sanity checks and started the full evaluation afterwards. The full ShapeLearn models were ready before the benchmarking was finished. Evaluating both sets, along with the competing models, is what took most of the time.

Then Unsloth released its Dynamic v3 models. At similar sizes, several had lower KLD than Lite in our measurements. On KLD alone, Lite looked less competitive.

KLD looked decisive

KLD measures divergence between a quantized model’s predicted token distributions and the BF16 reference under a particular evaluation setup. It is useful for diagnosing substantial changes, but lower divergence does not automatically mean better task performance.

We measure KLD on a dataset of about 5 million tokens of prompt and response pairs, drawn from several benchmarks, including long-context and agentic tasks. We also changed how KLD is computed, so that it is closer to what we expect KLD to measure:

• KLD is measured on response tokens only, not on prompt tokens. We do not want to measure how well a model can generate prompts.
• KLD only considers the tokens that have a chance of being sampled during generation, the top-20, top-40, or top-60 tokens at each position. The tail tokens never get sampled, so they do not contribute.
• Requests have clear boundaries. Each prompt and response pair is scored as its own request, not as part of one long concatenated stream.

For example, Unsloth’s UD-IQ3_S (vii) has about 20% lower KLD than the similarly sized smallest Lite model (Lite-1): 0.028759 versus 0.035875. Yet its aggregate benchmark score is lower: 95.55% versus 97.33% of BF16.

If lower KLD were sufficient to rank these models by task performance, the benchmark ordering should have followed it.

It did not.

The point is not that KLD is useless. It is that a fidelity ranking is not a task-performance ranking. This is the distinction explored in our KLD evaluation blog. Our related paper on KLD and quantization fidelity metrics was also recently accepted to the EMNLP Industry Track.

Lite held up

Naturally, we made more plots.

Here, we show the RTX Pro 6000 because it can accommodate the full comparison. Each model’s benchmark score is reused across the GPU plots; the measured throughput and the set of displayed models change.

Leaving the full ShapeLearn models aside for a moment, three of the six ShapeLearn-Lite models sit on the Lite-versus-Unsloth frontier: the three smallest Lite models, the lighter orange bubbles labelled 1-3.

Of the twelve Unsloth v3 models shown, three also sit on that frontier: UD-IQ2_S (A), UD-Q2_K_XL (B), and UD-IQ4_XS (F). UD-IQ4_XS (F) is a strong higher-quality point, while Lite earns its places in the middle of the range.

Add the five full ShapeLearn models back in (the darker orange bubbles), and they take over the entire frontier.

Lite was never meant to be the final result. It still held its own where it mattered.

Speculative Decoding

We also evaluated MTP and DFlash2 with the new models, using 3 draft tokens for MTP and 7 draft tokens for DFlash2. Both methods increased throughput for all five ShapeLearn models on all six GPUs tested.

DFlash2 was faster than MTP in almost all cases. Across the full lineup, DFlash2 reached 1.34-2.10x the baseline next-token prediction (NTP) throughput, while MTP reached 1.28-1.66x.

We measured with the sampling parameters Qwen recommends for thinking mode, over a diverse set of agentic coding, mathematics, and general-knowledge requests. The speedups would likely be larger under greedy decoding, but temperature-based sampling better reflects real usage.

The figure below shows NTP, MTP, and DFlash2 throughput for each GPU. The quality axis is the target-model benchmark score reported above. These plots do not independently establish quality equivalence between decoding methods.

There is also a memory tradeoff between the two approaches. The embedded quantized MTP weights add only about 250 MB to the model, and if MTP is not used, these weights are not loaded into GPU memory. In comparison, the 4-bit DFlash2 draft model is about 1.1 GB, so enabling DFlash2 requires roughly 1.1 GB of additional GPU memory.

Packaging MTP as a separate GGUF file would largely eliminate this advantage. The standalone model would need its own MTP embedding and output layers, which are by far its largest tensors, bringing its memory footprint to roughly 1 GB as well.

In addition, DFlash2 in llama.cpp currently does not support image inputs, which is an important consideration for multimodal use cases.

Benchmarking Methodology

We evaluate all reported models across a set of instruct and thinking benchmarks.

Instruct benchmarks:

• GSM8K for math
• IFEval for instruction following
• MMLU for general knowledge
• LiveCodeBench V6* for coding
• Multi-IF for multi-turn and multilingual instruction following
• ACEBench for tool use and agentic tasks

Thinking benchmarks:

• ACEBench for tool use and agentic tasks
• Multiple HumanEval for coding
• BFCL V4* for tool calling and agentic tasks

For the thinking benchmarks, we used Qwen 3.8’s medium thinking setting.

For each benchmark, the score of a quantized model is normalized by the score of the corresponding BF16 model. The overall reported score is the average of these normalized benchmark scores.

Our LiveCodeBench V6* evaluation includes problems from January 1, 2024 onward, excluding the 2023 problems. We found the 2023 problems to be relatively easy for current models, with most models achieving very high scores on them. As a result, they provide limited discrimination between models while adding substantial evaluation time.

For BFCL V4*, we evaluate the following eight subsets:

• live_simple
• live_parallel
• live_parallel_multiple
• live_relevance
• multi_turn_base
• multi_turn_miss_func
• multi_turn_miss_param
• multi_turn_long_context

All evaluations were run with llama.cpp b10430. For both instruct and thinking experiments, we use the sampling parameters recommended by Qwen for the corresponding mode.

Conclusion

ShapeLearn-Lite did what it was designed to do. It got useful Qwen 3.8 27B quants onto 12 to 24 GB GPUs quickly, and it held up better than its KLD ranking suggested.

Full ShapeLearn goes further. It improves the measured quality-speed trade-offs over Lite and contributes five frontier models across all six tested GPUs.

GPU-5 is our default recommendation wherever it fits, reaching 99.63% of BF16’s aggregate benchmark score. When memory is tight, GPU-4 is still very competitive: almost 99% of BF16 at a much smaller size, and faster.

KLD remains useful, but it is not a task-performance leaderboard. Fidelity metrics tell us how much the model’s distributions changed under a particular measurement. Benchmarks tell us whether those changes matter on the tasks we tested.

We were impatient. This time, it worked out pretty well.