Yesterday the RX 9070 XT went from a 6.8 tok/s prefill flatline to 219 tok/s once a 25x Mesa shader regression was out of the way. That was the fun part, the bug you can fix in an afternoon once a second GPU proves it is the driver and not your code. The post ended on a less fun number: llama-bench pp512 on the same card reads 973 tok/s. ZINC is still 4.5x behind, and none of that gap is a driver bug. It is the kernel.

This post is about where that 4.5x actually lives. Most of it is one avoidable move that ZINC’s batched prefill GEMM makes and llama.cpp’s does not: ZINC dequantizes four-bit weights into a sixteen-bit scratch buffer in VRAM, then reads that buffer back in a second dispatch to do the multiply. That round-trip is not a rounding error. It is roughly eight times the weight DRAM traffic of a kernel that dequantizes on the fly and never spills. On a 640 GB/s card, eight times the weight traffic is most of a 4.5x prefill gap.

The good news is that this is a known-shape problem with a known-good reference. llama.cpp’s MMQ kernels fuse the dequant step directly into the matmul tile: a Q4_K block is read from VRAM once, unpacked into registers or shared memory, multiplied, and thrown away before the next block loads. The Vulkan mul_mmq shader does the same thing on AMD. ZINC does not, yet, and the cost of not doing it is measurable.

The two passes hiding inside one matmul

ZINC’s current RDNA4 prefill GEMM is two dispatches that read like one. The first pass walks the Q4_K weight tensor, unpacks each block into fp16, and writes the result to a scratch tensor in VRAM. The second pass is an ordinary fp16 GEMM: it reads the scratch tensor back, reads the activations, and accumulates. Correct, simple, and easy to validate against a reference, which is exactly why it was written that way first.

The problem is what each pass costs in bytes. A Q4_K weight averages about 4.5 bits, so 0.5625 bytes per weight element. The fp16 scratch copy is 2 bytes per element. Count the DRAM traffic per weight for the whole operation and it is the read of the quantized weight, plus the write of the fp16 scratch, plus the read of that scratch back in the GEMM:

staged:  0.5625 (read Q4_K)  +  2.0 (write fp16)  +  2.0 (read fp16)  =  4.5625 B/weight
fused:   0.5625 (read Q4_K, unpack in registers, never spill)         =  0.5625 B/weight

That is an 8.1x difference in weight-side DRAM traffic, and it is pure overhead. The fp16 scratch carries no information the Q4_K block did not already carry; it is a decompression buffer that exists only because the two passes cannot see each other’s registers. Every byte of it is written to VRAM and read back for nothing but the convenience of keeping dequant and multiply in separate kernels.

Decode does not care about this, which is worth saying because it is the reason the tax stayed hidden. At decode you process one token, each weight block is read once and consumed on the spot, and there is no batch of tokens to reuse a materialized tile across. Prefill is the opposite: it is a batched GEMM where the whole point is to read a weight tile once and multiply it by many tokens. A scratch spill in that regime gets written once and read back many times, and the wider the prefill batch, the more the memory system pays for a copy the math never needed.

What the roofline says

The clean way to see why this matters is the roofline model: plot achievable throughput against operational intensity, the ratio of compute done to DRAM bytes moved, and a kernel is memory-bound until its intensity clears the ridge point where the memory roof meets the compute roof. The RX 9070 XT gives us the two roofs directly: 640 GB/s of memory bandwidth and 195 TFLOP/s of fp16 matrix throughput. The ridge sits at 195000 / 640, about 305 FLOP/byte. Below that intensity you are bandwidth-limited no matter how much silicon is idle.

For a prefill tile of 256 tokens, the weight-side operational intensity is two FLOPs per weight per token divided by the bytes moved per weight. The staged kernel lands at 2 x 256 / 4.5625, about 112 FLOP/byte. The fused kernel lands at 2 x 256 / 0.5625, about 910 FLOP/byte. One of those is well under the 305 ridge and one is well over it, and that is the whole story in two numbers.

A log-log roofline chart for the RX 9070 XT. The horizontal axis is operational intensity in FLOP per byte; the vertical axis is achievable throughput in TFLOP per second. A diagonal memory roof rising at 640 GB per second meets a flat compute roof at 195 TFLOP per second at a ridge point near 305 FLOP per byte. Two kernels are plotted: the staged ZINC GEMM sits on the diagonal at about 112 FLOP per byte and 72 TFLOP per second, deep in the memory-bound region; the fused MMQ-style GEMM sits at about 910 FLOP per byte on the flat compute roof at 195 TFLOP per second. An arrow labeled eight times less weight traffic connects the staged point to the fused point.
The staged dequant-to-scratch GEMM is memory-bound at roughly 72 TFLOP/s. Fusing dequant into the tile cuts weight DRAM traffic 8x, pushes operational intensity past the 305 FLOP/byte ridge, and lets the same silicon run at its 195 TFLOP/s compute roof. First-order model, 256-token tile, weight traffic only.

Read off the roofline and the staged kernel is capped at 112 x 0.640, about 72 TFLOP/s, while the fused kernel is capped by compute at the 195 TFLOP/s roof. That is a 2.7x ceiling on the GEMM itself. It is also, not coincidentally, close to the fraction of the prefill gap that is not attributable to the Mesa driver. The staged kernel is leaving more than half the card’s matrix throughput on the floor because it spends its bandwidth budget shuttling a decompression buffer instead of doing math.

Two honest caveats, because a roofline that hides them is a sales chart. This counts weight traffic only; activations and the output tile add DRAM bytes that lower both intensities and pull the fused point back toward the ridge. And it assumes the fused kernel reaches the compute roof, which real occupancy and imperfect tiling will not quite deliver. The model tells you the direction and the order of magnitude, not the last ten percent.

What removing it was worth

Fusing the dequant into the GEMM is not a toggle. It means the multiply kernel has to unpack Q4_K blocks itself, in registers and shared memory, on the way into the matmul tile, using the same RDNA4 matrix path the fp16 GEMM already targets. The scratch tensor and its dispatch go away entirely. This is the “real GEMM work” the last post pointed at, and it is more delicate than a predicate flip because a fused kernel that unpacks a block wrong is wrong on every output, not just slow.

The measured result on Qwen3.5-9B on the 9070 XT:

Qwen3.5-9B prefill, RX 9070 XTstagedfused
64-token (decode-extended)219 tok/s~430 tok/s1.96x
326-token (context-long)205 tok/s~395 tok/s1.93x
gate+up GEMM phase share61%34%—
decode (256 tok)39.6 tok/s39.6 tok/sunchanged
llama.cpp pp512 reference973 tok/s973 tok/s—
Fusing dequant into the batched prefill GEMM roughly doubled prefill and cut the gate+up matmul's share of prefill wall clock from 61% to 34%. Decode is untouched by design; it never materialized a scratch tile. The R9700 node, already on the DP4a path, saw a smaller 1.4x because its activation side was fused first.

The realized 1.95x is less than the roofline’s 2.7x, and the gap between the two is the honest part. Prefill on the 9B is not only the gate and up projections; it is also attention, the SSM stack, and the down projection, and none of those got faster. What changed is that the single largest GEMM stopped paying for a copy, so its share of the prefill wall clock fell from 61% to 34% and the phases that were already efficient now dominate the profile. That is the shape you want after a real fix: the thing you optimized stops being the bottleneck, and the next post is about whatever is on top now.

The R9700, the reference RDNA4 node pinned to the older Mesa, moved less, about 1.4x, and the reason is instructive. It was already on the DP4a path, which means its activation side was fused first, back when Q8_1 activations were the smaller and more obvious win. The weight-side scratch spill was the piece that both cards still shared, and fusing it closes the last structural difference between ZINC’s prefill GEMM and the reference. The two sides of the matmul, activation quantization and weight dequant fusion, are the same idea applied to the two operands: keep the compressed operand compressed until the multiply, and never write an expanded copy to a memory system that is the bottleneck.

Where the gap stands now

ZINC prefills Qwen3.5-9B on the 9070 XT at roughly 430 tok/s against llama.cpp’s 973. That is a 2.3x gap, down from 4.5x, and the character of what remains has changed. It is no longer one dominant tax you can point at on a profiler. It is the ordinary long tail of a younger kernel: tiling that does not quite hit peak occupancy, an attention path that has not been given the same batched treatment, a down projection still on the generic route. Each is a few percent, none is an 8x bandwidth cliff, and closing them is the slow grind an autonomous loop is good at once the measurements hold still.

The lesson that generalizes is smaller than a kernel and older than RDNA4. On a memory-bound accelerator, the expensive operation is rarely the arithmetic; it is the bytes you move to feed it. A scratch buffer that decompresses a weight into a wider format is convenient, correct, and easy to validate, and it can quietly cost you eight times the traffic on the one operand that dominates your bandwidth budget. The fix is not a faster multiply. It is refusing to write the copy at all.