In July this blog worked through an eight-agent swarm on a 16 GB RX 9070 XT and concluded that a Q8 KV cache is close to free. Halve the bytes, push the crossover where private KV reads overtake the shared weight stream from 5k tokens to 10k, roughly double the context the card can hold. It was a storage-format change with no kernel work and it paid on both axes.

On August 17 we turned the same optimization off by default on Apple Silicon. Muse Glimmer 30B at Q4_K_M decodes at 22.3 tok/s on the unquantized cache and 18.4 tok/s on Q8_0, measured back to back on an M4 Max at 257 tokens of context. Across the runs the quantized cache was 15 to 22 percent slower.

The part worth writing down is not that the number went the other way on a different backend. It is why. Q8_0 packs a KV element into 1.06 bytes against the unquantized path’s 4, so the slower path reads 3.8 times less data. A cache that reads a quarter of the bytes and loses seventeen percent of decode is telling you something specific: the kernel doing the reading was never bandwidth-bound to begin with.

The bytes are not the story, which is the story

ZINC reserves the cache for the model’s 131,072-token maximum context, which comes to about 14 GB unquantized and 3.7 GB at Q8_0. Both figures back out to the same geometry, 256 key values and 256 value values per layer per token across the 52 layers, so the two formats really are describing the same tensor. Those are the numbers that make KV quantization look obviously correct, and on a card where they bind, they are.

Now put them next to the workload. At 257 tokens of context the resident cache is 257 of those 131,072 positions, roughly 27 MB. The decode step reads about 15 GiB of weights per token at an effective 378 GB/s, a figure from the same per-kernel profile that produced yesterday’s lm-head result. Even allowing for grouped-query attention re-reading each key and value group once per query head, the KV traffic is a rounding error against the weight stream.

So the 9.5 milliseconds per token that Q8 costs, which is 54.3 ms against 44.8 ms once you invert the two rates, cannot be bytes. There are not enough bytes involved for a 3.8 times reduction in them to move anything, in either direction. Whatever is happening is happening per element, inside the kernel, and it scales with the number of positions rather than with the number of gigabytes.

A diagram on a pale cool-grey background titled 'The smaller KV cache costs more to read', subtitled with Muse Glimmer 30B at Q4_K_M on ZINC's Metal backend, M4 Max, 257 tokens of context, and noting it shows one 32-value slice of the key or value cache as the batched flash-attention kernel consumes it. The left column is headed 'Bytes on the wire, drawn to scale' and shows two horizontal byte strips for the same 32 elements. The upper strip, in deep teal and labelled 'unquantized f32, 128 bytes per 32 values', is a long bar divided by faint rules into eight equal 16-byte segments, captioned '8 x float4, 16 bytes each. One aligned vector load per float4.' Below it a much shorter plum strip labelled 'Q8_0, 34 bytes per 32 values' runs about a quarter the width, made of a solid 2-byte scale block at its left edge followed by a lighter run of int8 cells, captioned '2-byte block scale, then 32 x int8. Two loads per float4, the second dependent.' A dashed bracket spans the gap between the two strip ends and is labelled '3.8x fewer bytes', with a note that both strips are drawn at one pixel per 0.32 bytes so the widths are the real ratio. Beneath them sits a pale blue callout box headed 'Why the byte saving cannot be the story', reading that at 257 tokens the whole cache is roughly 27 MB against about 15 GiB of weights read per decode token, so the bytes saved are noise. The right column is headed 'Work per float4, exploded' and holds two rounded call-out panels. The teal 'f32 path' panel contains a single node reading 'one 16-byte vector load' with a short arrow into a node reading 'dot()'. The taller plum 'Q8_0 path' panel is a vertical chain: '4-byte packed char4 load', then '2-byte block scale load', then '4 int to float, 4 multiplies', then 'dot()'. A dashed curved arrow runs from the block scale node back up to the packed load node, annotated 'dependent, re-read for all 8 vectors in the block'. Across the bottom is a band headed 'The two axes disagree' with two paired-bar groups. The left group, 'Decode rate, 257 tokens of context', shows a teal bar at 22.3 tok/s for f32 above a shorter plum bar at 18.4 tok/s labelled 'Q8_0, 17% slower'. The right group, 'KV cache reserved at the 131,072-token maximum', shows a long teal bar at 14 GB for f32 above a much shorter plum bar at 3.7 GB labelled 'Q8_0, 10.3 GB smaller'. A footer states that both decode rates were measured back to back on the same prompt, that both cache sizes were measured, and that the byte strips and load counts are read off the Q8_0 block layout in ggml of 34 bytes per 32 values.
The same 32 key or value elements, in both cache formats. The strips on the left are drawn to scale, so the Q8_0 cache really is about a quarter the width. The exploded view on the right is what the flash-attention kernel has to execute to consume each four-element vector, and it is where the seventeen percent goes. The two bars at the bottom are the trade in its honest form: Q8 wins the capacity axis by 10.3 GB and loses the speed axis by 3.9 tok/s, and which of those you care about is a property of the machine, not of the format.

What the figure is meant to show is that the two panels are measuring different things and only one of them was ever in dispute. Nobody doubted the byte strip. The exploded view on the right is the part that was assumed rather than measured, and it is where the decision went wrong.

Two loads where there was one

ZINC’s Metal backend carries the batched flash-attention kernel in two versions. The unquantized one reads a key vector like this, once per four elements of head dimension:

const float4 kv = *(device const float4*)(k_cache + kv_base + (i << 2));
score += dot(qv, kv);

One aligned 16-byte load, one dot product. The Q8_0 version has to unpack a ggml Q8_0 block, which is a 2-byte half-precision scale followed by 32 signed bytes, 34 bytes for 32 values:

inline float4 loadQ8_0Vec4(device const uchar* base, uint vec4_idx) {
    const uint block_idx = vec4_idx >> 3u;          // 8 vec4s per 32-element block
    device const uchar* block = base + block_idx * 34u;
    const float scale = float(as_type<half>(*(device const ushort*)(block)));
    device const packed_char4* quants = (device const packed_char4*)(block + 2u);
    const char4 q = char4(quants[vec4_in_block]);
    return float4(float(q[0]), float(q[1]), float(q[2]), float(q[3])) * scale;
}

Count what changed. One vector load became a 4-byte packed load plus a separate 2-byte scalar load of the block header, and the second load is dependent: the address arithmetic for the quants comes from the same block pointer, and the multiply at the end cannot retire until the scale arrives. Then four integer-to-float conversions and four multiplies that the unquantized path does not perform at all.

The scale load is also redundant eight times over. There are eight four-element vectors in a 32-value block and each one re-reads the same 2-byte header. That read almost certainly hits cache, but it is still an instruction, an address computation, and a dependency edge in a loop that runs once per key position per head per layer, 52 layers deep.

This is the same failure shape as the lm-head kernel gate from yesterday, viewed from the other side. There, the kernel that read more bytes per instruction sat unused and cost us six percent. Here, the format that reads fewer bytes per element was switched on and cost us seventeen. In both cases the thing that mattered was the shape of the load, not the size of the tensor.

Where the crossover actually is

PropertyUnquantized f32 cacheQ8_0 cache
Bytes per KV element41.06
Cache reserved at 131,072 context14 GB3.7 GB
Device loads per four elements12, one dependent
Conversions per four elements04
Decode at 257 tokens, M4 Max22.3 tok/s18.4 tok/s

The table is the whole argument in five rows. Q8 wins everything about storage and loses everything about execution, and on this machine execution is the only side that is scarce. The M4 Max under test has 64 GB of unified memory holding a 16 GB model. Even the full 14 GB reservation at maximum context fits, and a realistic session is nowhere near maximum context.

Compare that to the RDNA4 case where this series first reached for Q8. Eight agents on a 16 GB card, 5.5 GB of that spent on resident weights, leaving about 10 GB of KV budget for all eight. An FP16 cache exhausts it near 9k tokens per agent. There, the quantized cache is not an optimization, it is the only configuration that runs.

The honest generalization is that KV quantization trades instructions for capacity at a fixed rate, and the exchange rate is set by the kernel while the value of each side is set by the machine. Both of this blog’s earlier KV posts held that view implicitly. The July post measured a card where capacity was the binding constraint. The August post on where KV quantization stops being free measured where the quality side breaks down below 8 bits. Neither of them checked what the dequant costs in a kernel that is not bandwidth-limited, and that is the term that dominated here.

The default we shipped is still one format too wide

The fix was five lines in defaultKvCacheQ8Enabled: return false for the Muse Glimmer architecture, leave ZINC_METAL_KV_Q8=1 as the opt-in for memory-constrained setups. Output stayed correct on the standing gates, and speculative decode stayed byte-identical against the per-token path.

It is not the right answer, only the better of two available ones. ZINC’s unquantized Metal cache is f32. The reference build we measure against, llama.cpp, defaults both halves of the cache to f16, which its server documentation lists as the default for --cache-type-k and --cache-type-v alike. Half precision is the format that should actually win this argument, because a half-to-float widening happens in the load itself rather than through a block-scale lookup. It keeps the single-load shape, drops no dependency edge into the inner loop, and still cuts the 14 GB reservation in half. We do not implement it on this path, so we defaulted to the wide, exact, fast thing instead of the narrow, exact, fast thing.

There is also a context length we have not measured. Attention cost grows with position count while the weight stream does not, which is the entire premise of Flash-Decoding and the reason the analytical inference models in Pope et al. treat the KV term separately from the parameter term. Far enough out, KV bytes stop being noise. What is unclear is whether Q8 ever crosses back over on this kernel, because the dequant cost is per position and therefore scales with context on exactly the same slope as the byte saving. If both terms grow linearly, the ratio does not move and Q8 stays slower forever, just with a larger absolute gap. That is a prediction from the shape of the code, not a measurement, and it is the next thing to run.

What to take from a knob that reversed sign

The generalizable claim is narrow and I want to keep it narrow. It is not that KV quantization is bad, and it is not that measurements do not transfer between backends. It is that a compression format is two decisions wearing one name. One decision is how many bytes the data occupies, which is a property of the format. The other is how much work it takes to consume a byte, which is a property of the kernel that consumes it. Marketing a format by the first number and inheriting the second by accident is how you ship a default that costs seventeen percent.

The cheap defense is to make the second number visible. We knew the cache sizes to the gigabyte before we enabled Q8 and had never once measured the two attention kernels against each other on the same prompt. That comparison takes one run. It is now the gate on any storage-format change in this engine, alongside the byte-identical correctness check that every math-preserving cycle already has to clear.