I'm building ZINC, a local LLM inference engine for consumer GPUs. This site is where I write about the work — GPU kernels, Vulkan, Metal, CUDA, quantization, and the systems engineering underneath.
A month of kernel work moved one RDNA4 decode token from 25.2 ms to a modeled 18.0 ms. Adding a second concurrent sequence would have added more aggregate throughput than all of it, because 10.7 of those 18 milliseconds are weight bytes that every sequence in a batch reads exactly once. Local engines are built around a single user typing, and the workload on top of them is turning into four agents fanning out at the same time.
Command buffer reuse took the CPU rebuild off yesterday's decode token and left about 4 ms of GPU barrier bubbles. Those bubbles are not kernel launches. Each one is a pipeline barrier that flushes the write cache to L2 and invalidates L1 before the next kernel can read, and at batch one the compute units sit idle through the round trip. Kernel fusion removes them by keeping the intermediate in registers, which is why fusion's real decode win is the barrier it deletes, not the launch it saves.
Yesterday's decode budget left 7 ms of every 25.2 ms token in pure dispatch and barriers. NVIDIA already deleted most of that on CUDA with graph capture, worth 14 percent on a 7B. RDNA4 has no CUDA graphs, so ZINC rebuilds the whole decode command buffer every token on Vulkan. Recording it once and resubmitting it is the cheap half of the fix. The other half, the barrier bubbles, needs kernel fusion, and that is the part command buffer reuse cannot touch.
Yesterday's post said a chat turn is 96 percent decode and left decode stuck at 39.6 tok/s, a third of the RX 9070 XT's 116 tok/s bandwidth ceiling. This is where the other two thirds go. Streaming the weights is only 10.7 ms of a 25.2 ms decode token. The rest is latency the batch of one cannot hide, and the memory bus is innocent for most of it.
The last six posts took Qwen3.5-9B prefill on the RX 9070 XT from 219 to 962 tok/s, a 4.4x win that closed the llama.cpp gap. On a normal 2k-in, 2k-out chat turn it shaved the wall clock by 13 percent, and the last post in the series was worth 0.7 percent. Prefill is now 4 percent of a chat turn. Decode is the other 96, and decode is a bandwidth problem the matrix core cannot touch.