I gave Qwen3.8's MTP drafter another 69.2 MiB of precision. Throughput fell from 50.44 to 37.02 tokens per second. That result sums up the whole experiment: the best local inference setup is rarely made from the individually "best" parts.

I wanted a dense 27B model, its full 262,144-token context, multimodal input, maximum useful quality, and speculative decoding on an NVIDIA RTX PRO 4000 Blackwell SFF with 24 GB of VRAM. The server also had to survive real agent work after printing model loaded. The experiment followed a hunch I had written about earlier: careful operation may matter as much as moving to a larger model.

The finished system averages 50.44 tok/s in the current ten-run production series. On a strict runtime A/B, the custom llama.cpp build reaches 55.40 tok/s versus 45.42 for clean master, a 21.97% gain. Against target-only greedy decoding, embedded MTP moves 21.19 to 59.46 tok/s, or 2.81 times the throughput. At the far end of a genuinely occupied 256K cache, it still produces 12.61 tok/s without an out-of-memory failure.

Those numbers came from different gates and should stay separate. Combining them into one heroic speedup would make a better headline and a worse benchmark.

The winning setup came from the fit between the quant, drafter, CUDA kernels, memory layout, and workload. No component won on its own.

The target was deliberately unreasonable

Qwen3.8 27B is a 64-layer dense model. Its repeating pattern contains three Gated DeltaNet layers followed by one full-attention layer, giving 48 recurrent layers and 16 conventional attention layers. It has a native 262,144-token context, a one-layer MTP head, and a separate 27-layer vision encoder.

The hardware is lopsided in a useful way:

• GPU0: RTX PRO 4000 Blackwell SFF, 24 GB GDDR7 with ECC, a 192-bit memory interface, 432 GB/s peak memory bandwidth, 24,467 MiB reported capacity, and sm120a. It holds the target, embedded MTP, recurrent state, graphs, and the 256K KV cache.
• GPU1: RTX 2000 Ada, 15,996 MiB, sm89. It holds the F16 multimodal projector and other auxiliary services.
• Runtime: Debian 13, CUDA 12.9.86, GCC 14.2, dual-architecture CUDA build.

Only the 16 full-attention layers grow a conventional KV cache with sequence length, which makes 256K less absurd than it first appears. With Q4 K and V, that cache costs roughly 4.25 GiB before allocator overhead. DeltaNet adds recurrent state and checkpoints instead. Four checkpoints were the useful minimum; the default 32 spent memory I needed elsewhere.

NVIDIA quotes 432 GB/s of peak bandwidth. That is a hardware ceiling rather than an application metric from llama.cpp, but it matters here. Autoregressive decode repeatedly streams quantized weights, and the 16 attention layers add increasingly expensive KV reads as context fills. This is why the same profile averages about 50 tok/s on the production task and 12.61 tok/s at the far end of a 261.5K-token cache.

The original plan was simple: estimate the capacity, select a quant, then benchmark it. The machine immediately taught me that capacity estimates are just admission tickets. The real test begins after loading.

The first winner was Q4_0, and it was the wrong winner

I began with public GGUFs at 40K context. Q4_0 was surprisingly strong. Target-only decoding reached 22.40 tok/s, and MTP with n_max=3 reached 44.95. It beat smaller Q3_K_M and nominally smarter Q4_K_M variants because file size and quant label do not describe the CUDA kernel that actually runs.

Then quality testing spoiled the easy answer. On a short, identical WikiText-2 control, IQ4_XS scored 6.1175 perplexity while Q4_0 scored 6.3798. Q4_0 led the speed table. Hermes needed a main model, though, and that quality trade felt too expensive for a few hundred milliseconds. I would have been using a 27B model as oversized autocomplete.

The opposite extreme failed too. Q4_1 reached 6.1127 PPL, marginally ahead of IQ4_XS, but its memory footprint made 256K plus F16 vision uncomfortable. The useful point was somewhere between a fast blunt quant and a precise file that left no room for the rest of the system.

Loading 256K proves almost nothing

Early capacity tests looked excellent. Q4_0, MTP, Q4 KV, four recurrent checkpoints, and the F16 projector all allocated at 262,144 context. That still did not answer the question I cared about.

I filled the slot with 261,500 input tokens, generated another 256, and then reused the hot cache. No truncation. No OOM. The first Q4_0 profile decoded at 12.06 tok/s near the end of the cache, compared with 44.95 around 40K. GPU usage sat at 99 to 100%, while the server used roughly one CPU core. The bottleneck was the 16 full-attention layers reading a huge occupied KV cache, not a secret CPU fallback.

This changed the benchmark method for every run that followed. "262K loaded" was banned from the results table. A long-context claim had to include actual token fill, post-fill VRAM, hot decode, truncation state, and an output hash.

The ready-made NVFP4 quant failed the quality gate

Blackwell has native FP4 hardware, so a ready-made NVFP4-MEDIUM GGUF looked like the obvious route. Its bulk target matrices used NVFP4, with a Q8 output head, Q6 embeddings, and an IQ4_XS MTP layer. It reached 40.46 tok/s and fitted the complete 256K plus vision profile with about 1,055 MiB free.

Its PPL was 6.4949. Worse than Q4_0.

The conversion recipe was the problem. Attention and DeltaNet weights from the source FP8 checkpoint had been expanded and requantized into NVFP4 along with the large, tolerant matrices. Native arithmetic made the file quick, while indiscriminate low precision damaged sensitive parts of the model. Hardware format support does not tell you where to spend the bits.

That failure gave us the design for a custom quant: use NVFP4 for the bulk, then protect only the tensors that our own workload says matter.

I calibrated the model on how I actually use it

The calibration corpus started with 5,472 messages from 296 Hermes sessions. I placed that material before the generic corpus so the 153,600 processed tokens represented coding, Polish and English conversation, infrastructure work, tool calls, and the awkward mixtures my agents really see. A secret scan ran before calibration. No PEM keys, provider tokens, GitHub tokens, Slack tokens, or email addresses were present.

llama-imatrix collected importance data for 497 target weights. NVFP4 does not consume an iMatrix directly during block quantization, so I used the matrix as a map: large tolerant tensors stayed native NVFP4; selected attention, DeltaNet, and FFN tensors moved to Q5_K or Q6_K; embeddings became Q6_K; the output head stayed Q8_0.

The first 5.14 BPW hybrid was the quality winner at 6.0967 PPL. It was also slow at 34.19 tok/s and too large to keep the desired projector on GPU alongside 256K. Good experiment. Bad production model.

The second build was tighter:

• Size: 16,321.38 MiB, 5.01 BPW.
• Bulk matrices: native NVFP4.
• Sensitive target tensors: selected Q5_K and Q6_K using the Hermes iMatrix ranking.
• Embedding and output: Q6_K and Q8_0.
• Embedded MTP: native NVFP4.
• PPL: 6.1197, versus 6.1127 for Q4_1. The 0.11% gap is far below the error of this short control.

This became Qwen3.8-27B-Hermes-iMatrix-NVFP4-Balanced.gguf. It preserved the measured quality of the Q4_1 reference, ran faster, and left enough room for the actual serving stack.

MTP had a trapdoor at n=8

The early sweep suggested n_max=3. Values 4 through 7 got slower as rejected draft work accumulated. Then n=8 jumped to 49.31 tok/s.

The curve is jagged. Eight candidates hit a favorable batch and kernel shape. Nine was no faster, and each extra candidate cost about 150 MiB. With the full 256K allocation, n=9 at ubatch=256 failed on one more 162 MiB CUDA graph buffer. Reducing ubatch to 128 made it load, but throughput fell to 52.70 tok/s and 32K prefill suffered. N=10 failed on another 81 MiB. Even an experimental scheduler pool lost to a 31 MiB allocation.

I kept n=8 because it was the last fast point before the allocator started biting.

More accurate MTP made the system worse

I wanted a controlled rival for the NVFP4 MTP choice. A patched MTP-aware iMatrix run processed 300 Hermes-history chunks and added all eight MTP matrices. For the comparison, all 851 non-MTP tensors were verified byte-for-byte identical to production. Only the eight MTP weight tensors changed.

The higher-bit drafter may be closer to the BF16 source model. In production it had one job: predict this quantized target. The NVFP4 errors in the MTP head happened to align better with the NVFP4-heavy target, so its exact proposals survived more often. Standalone precision lost to quant-drafter alignment.

This changed how I treat drafter quality. The drafter and target form one quantized system, and their interaction decides acceptance and throughput. A standalone quality score for either half misses it.

DSpark had a whole second GPU and still lost

I also tested Qwen3.8-27B-DSpark, a 1.36B diffusion drafter with a Markov head and confidence head. A llama.cpp patch allowed the sidecar to run on GPU1 while the target stayed on GPU0.

The best version was Q8_0 at 26.49 tok/s, 34.5% above its 19.70 tok/s target-only reference. Embedded MTP reached 49.61 tok/s on the same comparison. DSpark was 46.6% slower and used more aggregate VRAM. The sidecar had been trained against an FP8 target, while ours was a mixed NVFP4/Q5/Q6 target. PCIe traffic and the weaker Ada card did the rest.

DSpark's diffusion mechanism worked, but this sidecar was trained for a different target and lost on this box.

The llama.cpp build mattered almost as much as the model

Once the model stabilized, I benchmarked runtime changes one branch at a time. N-gram speculation was disabled for every A/B because repeating a prompt taught the cache and pushed apparent throughput from 46.77 to roughly 180 tok/s. Useful in production, poison in a kernel comparison.

The final build pins #26001, #26048, #26705, #27173, #24891, and #25635 to audited commit hashes. The patches cover Gated DeltaNet, CUDA dispatch, faster Q4/Q5 speculative verification, chained MTP, recurrent-checkpoint correctness, and Flash Attention swizzling.

The Flash Attention patch deserves its own number. At 32K it moved prefill from 759.38 to 815.64 tok/s, up 7.41%, and hot decode from 37.26 to 38.23, up 2.61%. It helped where the attention workload was large enough to matter.

The builder refuses to continue if any PR head moves. It builds sm89 and sm120a, then runs sampling, quantization, Gated DeltaNet, Flash Attention, and NVFP4 matrix tests. Without those checks, the next upstream update could turn a fast private binary into an outage.

The final production profile

The active model is Qwen3.8-27B-iMatrix-NVFP4-256K-MTP. The important runtime settings are:

CUDA_VISIBLE_DEVICES=0,1
MTMD_BACKEND_DEVICE=CUDA1
LLAMA_SPEC_CHAIN=1
GGML_CUDA_GRAPH_OPT=1

llama-server \
  --model Qwen3.8-27B-Hermes-iMatrix-NVFP4-Balanced.gguf \
  --device CUDA0 --n-gpu-layers 999 --fit off \
  --ctx-size 262144 --parallel 1 --ctx-checkpoints 4 \
  --flash-attn on \
  --cache-type-k q4_0 --cache-type-v q4_0 \
  --batch-size 512 --ubatch-size 256 \
  --temp 0.6 \
  --spec-type draft-mtp --spec-default \
  --spec-draft-n-max 8 --spec-draft-n-min 0 --spec-draft-p-min 0 \
  --spec-draft-type-k f16 --spec-draft-type-v f16 \
  --spec-draft-backend-sampling \
  --mmproj mmproj-F16.gguf \
  --reasoning-preserve --jinja --metrics

--spec-default adds n-gram speculation in this build. It stays enabled because agent sessions repeat code, tool schemas, and prompt prefixes. It never appears in comparative benchmark numbers. Temperature is 0.6 in the stack, while reasoning effort remains request-controlled by Hermes. I do not force xhigh globally.

--agent is deliberately absent. Hermes owns tools and MCP. Enabling llama-server's agent layer would duplicate the tool loop and widen the code-execution surface. --fit off is deliberate too: automatic fitting would silently change context or offload to reserve its default VRAM margin, invalidating the profile we measured.

The F16 projector uses 982 MiB on GPU1. That is better than squeezing it onto GPU0, and much better than running a separate 4B vision model. The active 17 GiB model and binaries stay on local NVMe for fast restarts; inactive experiments live on slower network storage.

The numbers I kept

The full-context run generated 256 tokens without truncation or OOM. Cold and both hot outputs had the same hash. The ten-run production series was also deterministic within its configuration.

One correctness caveat remains. Target-only greedy and MTP n=8 produce different continuations on a quantized target, matching the open llama.cpp batch-invariance issue #25618. Both modes are internally deterministic and the outputs are coherent, but this max-TPS profile is not bitwise distribution-preserving relative to target-only decoding. N=1 is the safer lossless setting when that property matters.

The mistakes worth keeping

I would start with the production task distribution and three hard gates: quality, real context fill, and deterministic A/B. Quant labels would come later.

I also learned to leave the last few hundred MiB alone. We intentionally ran near 76 MiB free during one tuning phase. It worked until a 31 MiB scheduler allocation and graph fragmentation showed why arithmetic free memory is not operational headroom. The final roughly 500 MiB margin keeps the server alive through graph creation, vision requests, and allocator variation.

Next time I will test the drafter against the exact quantized target from day one. The 69 MiB "upgrade" settled the question: the higher-precision MTP weights reduced agreement, acceptance, and throughput.