The RX 9070 XT prefill cliff: a 25× Mesa regression hiding in plain sight

A new GPU arrived in the rack — a consumer Radeon RX 9070 XT, Navi 48, RDNA4, GFX1201, 16 GB. It is exactly the kind of card a local LLM engine should care about: new AMD silicon, desktop pricing, enough memory for useful models, and the same architectural family as the larger R9700 node. So the first test was simple: point ZINC at Qwen3.5-9B and read the number.

The number was 6.8 tok/s prefill.

On the same card, same model, and same driver stack, llama-bench read 973 tok/s. That is a 143× gap. The strange part was that decode was fine: ZINC decoded at 39.6 tok/s, comfortably ahead of llama’s 21.7. So the engine worked. The forward pass was correct. The GPU was busy. And yet prefill — the batched part where the prompt should amortize work across tokens — had fallen off a cliff.

This is the story of finding the 25× regression hiding inside that 143× gap, why it took a second RDNA4 GPU to prove it, and what happened after the obvious fix: a clock-ramp noise trap, a mid-cycle crash, an autonomous loop driven by GLM-5.2, and a stack of small keeps that moved the 9B from a flatline to a real curve.

The scene, and the first (wrong) theory

The 9070 XT is an awkward card to benchmark for one specific reason: it is so new that the only driver that knows its device ID is Mesa 26.0.0-devel. The project’s reference RDNA4 node — a Radeon AI PRO R9700 — is deliberately pinned to Mesa 25.0.7, because 25.2.8 already cost it ~14% in a known RADV regression. So the two RDNA4 cards in the rack run different Mesa major versions, and there is no way around it: older Mesa literally does not enumerate GFX1201.

The first theory was the boring one, and it was wrong. Qwen3.5-9B is a dense SSM+attention hybrid, and an older optimization effort had already documented that one fast prefill path was shape-locked to the 27B model. A predicate named isQwen36DenseHybrid27B gated the layer-major machinery on hidden_dim == 5120, which the 9B (4096) fails. The hypothesis wrote itself: the 9B was falling through to prefillBatch, the per-token “token-major” path, and token-major was simply slow.

Easy to check. Probe the GPU while prefill runs:

sample 1: gpu= 18%  cpu= 0.5%   (warming up)
sample 2: gpu=100%  cpu= 4.2%
sample 3: gpu=100%  cpu= 3.2%
sample 4: gpu=100%  cpu= 4.8%

The GPU is pinned at 100% busy for the entire prefill, CPU nearly idle. This is not a fallback to a slow host path. The GPU is hard at work. It is just doing the wrong work, slowly. Theory one was dead.

So the profiler. ZINC has a built-in ZINC_PREFILL_PROFILE=1 that bucketizes the prefill wall clock into GPU phases. For a 64-token prompt (the “decode-extended” site scenario), the totals over a 9.5-second prefill:

Prefill GPU phases totals: attn=43.1  ssm=172.3  dense_ffn=9223.6  tail=2.3 ms
dense_ffn subphases: gateup_matmul=9123.4 (generic=9123.4  q4=0.0  q6=0.0)

Ninety-seven percent of the prefill is in one sub-phase, the dense feed-forward gate+up matmul, and it is running the generic path — not the specialized q4 path — for the full 9.1 seconds. Everything else (attention, the entire SSM stack) is a rounding error next to it. Whatever is wrong is wrong in exactly one matmul.

The two-GPU natural experiment

The move that cracked it only works if you have a control machine.

The Radeon AI PRO R9700 node is the same architecture — GFX1201, 64 CUs, same wave size, same cooperative-matrix silicon — but pinned to the stable Mesa 25.0.7. Same ZINC binary. Same Qwen3.5-9B Q4_K_M weights. Same Vulkan path. The only things that differ between the two cards are the things you cannot make equal: the Mesa version, and one capability bit.

Run the identical 64-token prefill on both:

	R9700 (Mesa 25.0.7)	9070 XT (Mesa 26.0-devel)
Prefill, 64 tok	160 ms / 392 tok/s	9479 ms / 6.8 tok/s
gateup matmul phase	~0 ms	9095 ms
cooperative_matrix exposed	no	yes

Same shader. Same dispatch. Same model. Fifty-eight times apart. The profiler’s smoking gun: the gateup matmul time on the R9700 is effectively zero because it runs in the fused path and is attributed elsewhere, while on the 9070 XT the same code path logs 9.1 seconds under the generic phase label.

There is exactly one knob that distinguishes “use the int8 DP4a (shaderIntegerDotProduct) kernel” from “use the branchless Q4_K kernel”: a predicate called qwenDenseFfnDp4aEnabled. The decisive experiment takes one line:

ZINC_QWEN_DENSE_FFN_DP4A=0  ./zig-out/bin/zinc --model-id qwen35-9b-q4k-m ...

Disabling DP4a on the 9070 XT takes prefill from 6.8 → 169 tok/s. Twenty-five times faster. Identical output (Command shape: benchmark --preset ..., the correct Long Coding Plan answer). The regression is not in ZINC’s logic at all. It is in the compiler: on Mesa 26.0.0-devel, when the device also exposes VK_KHR_cooperative_matrix, RADV emits a catastrophically slow SPIR-V→AMDGPU lowering for the int8 dot-product accumulate kernel (mul_mm_q4k_gate_up_swiglu_full_dp4a). The same kernel is fast on RDNA4 without cooperative matrix (Mesa 25.0.7). It is a driver regression on brand-new hardware, and it was quietly absorbing 97% of the prefill wall clock.

This is why the two-node comparison mattered. On one card, “the DP4a kernel is slow” looks identical to “my kernel is badly tuned” or “the 9B is hitting a slow path.” You can spend a week retuning a shader that was never the problem. The R9700 was the control that proved the shader logic was correct and the environment was the variable.

The fix, and the scoreboard

The fix is small and deliberately narrow. qwenDenseFfnDp4aEnabled now defaults the Qwen3.5-9B to the branchless Q4_K path when amd_rdna4 and cooperative_matrix are both exposed — i.e. exactly this configuration — and lets the already-loaded mul_mm_q4k_gate_up_swiglu_full kernel run instead. It is scoped to the 9B so the 27B and Gemma paths are untouched, and ZINC_QWEN_DENSE_FFN_DP4A=1 forces the DP4a path back on for re-testing the day Mesa 26 ships stable.

Qwen3.5-9B prefill on the 9070 XT	before	after
64-token (decode-extended)	6.8 tok/s	~170 tok/s	25×
326-token (context-long)	5.8 tok/s	~153 tok/s	26×
781-token	—	~142 tok/s	—
decode (256 tok)	39.6 tok/s	39.6 tok/s	unchanged — still beats llama’s 21.7

So the 143× gap was really a 25× driver regression sitting on top of an honest structural gap to llama’s batched GEMM. The regression was the part you could fix in an afternoon, once you could see it.

Then: a noise trap, and a clock that would not sit still

With the regression fixed, the next move was to point an autonomous optimization loop at the remaining gap and let it grind. The project already had a harness for exactly this — optimize_perf.ts — but it ran claude/codex agents. The first harness work was adding opencode + GLM-5.2 as a first-class agent (opencode run --model zai-coding-plan/glm-5.2 --variant max), plus a crash-restart supervisor, plus an RX 9070 XT effort doc seeding the loop with everything above so it would not re-tread the DP4a discovery.

The loop started up, ran the baseline at a healthy 212 tok/s, and then… reverted every single cycle. Cycle 1: reverted. Cycle 2: reverted. Cycle 3: reverted. Each with a hopeful-looking change that came back inside the noise floor. This is the optimizer’s job — it should not keep a change that does not measurably win — but the shape of the reverts was wrong: the samples inside one cycle were swinging from 160 to 210 tok/s on identical code.

The 9070 XT’s pp_dpm_sclk reads three states: 500 MHz, 0 MHz, 2520 MHz. In dynamic (auto) power mode the card drops to that 0 MHz idle state the moment a kernel stops issuing, and ramps back to 2.5 GHz over ~0.4 seconds when work arrives. A 64-token prefill burst lasts ~0.37 seconds. The burst and the ramp are the same length. So whether a given sample catches the clock cold (mid-ramp, ~160 tok/s) or warm (already boosted, ~210 tok/s) is a coin flip, and the loop’s median-of-five flipped with it. Three cycles of “real” improvements were drowned in that coin flip.

The fix is the sysfs one-liner the harness now runs before every benchmark:

echo high > /sys/class/drm/card1/device/power_dpm_force_performance_level

Pin the card to high-performance and the ramp disappears — consecutive samples come back at 205.23, 205.97, 205.74 tok/s, a variance under 0.6%. (And high mode survives a 30-second idle that drops the reading back to 0 MHz: the ramp is what it kills, so even a cold-start burst now lands at full boost.) With a stable floor, the loop’s keep threshold — max(+3 tok/s, +2%) — finally means something. As a bonus, the old 212 tok/s “baseline” turned out to be a lucky-warm outlier from the noisy era; the real stable speed was 205, and a fresh re-baseline reset the target to something achievable.

The crash, and the supervisor

Of course the loop then crashed.

Midway through cycle 1 of the clean run — agent reading the dense-down shader, heartbeat ticking at seven minutes — the process simply vanished. No error. No stack trace. No OOM kill, no SIGKILL in the system log, no crash report. One transient exit, and the entire multi-hour effort was dead. The existing harness treated any exit as terminal.

The lesson is that a long autonomous loop needs to be restartable by construction, the way the project’s overnight zinc_rt-autopilot loop already is. A small supervisor (loops/supervise_perf.sh) now wraps the effort: any exit — crash, timeout, or cycles exhausted — is followed by a --resume restart from the last committed cycle boundary. State is written only after a keep/revert decision, so a mid-cycle crash never corrupts it. The next crash costs one cycle, not the whole run.

(Along the way, two macOS-as-host footguns bit the GLM-5.2 agent: it kept wrapping commands in timeout — which does not exist on macOS — and one launch used setsid, which also does not exist. The effort doc now carries an “Agent environment” section: no timeout, no setsid, raise the shell-tool per-command timeout for builds, the model is remote-only, the controller does the authoritative build so a local one is just a sanity check. Small frictions, but each one was costing the agent a full turn.)

The keeps, honestly

Stable floor, restartable loop, max-reasoning agent. From there the loop did what loops are supposed to do: grind.

cycle	change	prefill tok/s
baseline	(regression fix, stable floor)	205.44
1	dense-FFN tuning	208.10
2	+1.88%	212.02
3	foundation: SSM conv1d recurrent state → shared memory	—
4	+2.75%	217.84
5	foundation: profiling timestamp artifact fix	—
8	foundation: fused SSM gated-norm shader (one WG/head, all tokens internal)	—
9	+0.40%	218.71

205 → 218.71 tok/s, about +6.5%, stacked entirely on top of the 25× regression fix. The keeps are real — each cleared the max(+3, +2%) bar against a tight noise floor — and they are the kind of incremental, profile-driven work (cache the conv1d state in shared memory, fuse the gated-norm dispatch, fix a profiling-timestamp artifact that was mis-attributing time) that an autonomous loop is genuinely good at once its measurements are trustworthy.

The honest ceiling

The 9070 XT went from 6.8 to ~219 tok/s on Qwen3.5-9B. Decode was never the problem and still leads llama (39.6 vs 21.7). And yet llama-bench pp512 on this same card reads 973 tok/s. ZINC is still ~4.5× behind on prefill, and that last gap is the hard one: structural kernel efficiency. llama.cpp fuses weight dequantization into the GEMM (MMQ), so each weight block is read once and consumed on the fly; ZINC’s batched path dequantizes into a wider scratch and reads it back. Closing that means real GEMM work — a persistent fp16 weight cache, or an MMQ-style fused-dequant kernel — not another predicate flip. The loop is now grinding on exactly that, on a node whose measurements finally hold still long enough to trust a 2% win.

Two lessons, neither new, both worth retelling. A single GPU cannot diagnose its own driver regression: “my kernel is slow” and “my compiler is broken” are indistinguishable without a control, and the control here was a second card running an older Mesa. And a noisy benchmark is worse than no benchmark: it does not just fail to find improvements, it manufactures confident-looking reverts of real wins. Fix the floor before you trust the scoreboard. After that, an autonomous loop with a max-reasoning model and a restartable harness can grind out the 6% you would have found yourself, and keep going through the crash you would not have.