The number this series keeps quoting for the RX 9070 XT is 640 GB/s, and it is worth pausing on why that is the honest figure rather than a conservative one. RDNA4’s headline memory feature is not the GDDR6 bus. It is the 64 MB Infinity Cache, a large on-die SRAM last-level cache whose entire job in a game is to catch re-reads and make the card behave as if it had far more bandwidth than its bus provides. During local LLM decode it catches almost nothing.

That is the claim I want to defend. A decode step is a streaming read with no reuse, and a cache that is 88 times smaller than the thing being streamed cannot change a streaming read. So the 640 GB/s is not the bus rate with a cautious asterisk; it is the rate decode actually sees, because the Infinity Cache sits idle through the part of the token that costs the most.

The twist, and the reason this is worth a post rather than a footnote, is that the agent swarm the last two weeks have been about is the first local workload that finally gives that cache something to do. Not the weights. A shared prompt.

A cache pays off on reuse, and decode has none

A cache does exactly one thing: it holds recently touched bytes so a later read of the same address lands in fast SRAM instead of slow DRAM. That only helps if the same bytes are read more than once before they are evicted. Volume is irrelevant; reuse is everything.

Now look at what a decode step reads. To produce one token, ZINC streams the entire Qwen3.5-9B Q4_K_M model out of VRAM, about 5.5 GB, and that stream is 10.7 ms of a 25.2 ms token, the single largest slice. Each matrix is read once, multiplied into the running activation, and never touched again inside that token. The next token re-reads all 5.5 GB from scratch, roughly 25 ms later, long after the first bytes have been pushed out of any 64 MB structure. There is no temporal reuse to catch.

Put the two facts next to each other and the cache’s fate is settled. The working set is 5.5 GB, the cache is 64 MB, a ratio of about 88 to 1. Even if the access pattern were friendly, only a little over one percent of the stream could be resident at a time, and the pattern is not friendly, it is a single linear sweep. Every line is a compulsory miss. The Infinity Cache contributes no measurable effective bandwidth to the weight stream, and decode runs at the raw GDDR6 rate. That is the 640 GB/s.

This is the memory-side mirror of a point made earlier in the series about the compute side. RDNA4’s matrix cores sit out the decode loop because decode has too little arithmetic to feed them; the Infinity Cache sits out the same loop because decode has too little reuse to fill it. Decode is thin in both currencies the card is built to spend.

Private KV is the same story

The KV cache does not rescue the cache either, at least not for a single sequence. Each agent’s attention reads back every key and value it has stored, and Qwen3.5-9B costs about 147 KB per fp16 KV token even after grouped-query attention has folded many query heads onto each key-value head. An agent holding a 6,900-token working set is reading roughly a gigabyte of KV every decode step.

That gigabyte is also read once per step. Within a token, each stored KV entry is visited a single time by attention and then dropped. So the private KV read has the same zero-reuse shape as the weight stream, and at 16 times the size of the cache it gets the same non-help. A single agent, no matter how it is tuned, hands the Infinity Cache a workload with nothing to cache.

A schematic on a near-black slate background titled 'A 64 MB Infinity Cache against a 5.5 GB decode read', subtitled 'RDNA4 (RX 9070 XT) memory path during Qwen3.5-9B decode: what the on-die cache can and cannot hold'. The left half, labelled 'the read path', shows three stacked tiers: a 'Compute units plus WMMA' box at top that consumes weights and KV then discards them, a cyan-outlined 'Infinity Cache, 64 MB SRAM' box in the middle that only pays off on bytes read more than once, and an amber-outlined 'GDDR6, 16 GB, 640 GB/s' box at bottom where every weight byte lives, read once per token. A thick amber arrow labelled 'weight stream 5.5 GB, no reuse, cache skipped' runs straight from GDDR through the cache tier to compute, bypassing it. A cyan arrow labelled 'shared-prefix KV' runs from GDDR into the cache and loops there, annotated 'eight agents, one GDDR read'. The right half, labelled 'working set versus the 64 MB line', has a dashed cyan vertical line marking the 64 MB cache and four horizontal bars: 'weight stream' at 5.5 GB, eighty-eight times the cache; 'private KV, one agent at 6,900 tokens' at 1.0 GB, sixteen times; 'shared prompt KV, 400 tokens' at 59 MB, which fits; and 'shared prompt KV, 2,000 tokens' at 294 MB, which spills with only about 445 tokens staying hot. A callout box reads '64 MB divided by 147 KB per KV token is about 445 tokens, the longest shared prefix the cache can keep resident'.
Modeled for Qwen3.5-9B Q4_K_M on one RX 9070 XT: a 5.5 GB weight stream per decode step, 147 KB per fp16 KV token, 640 GB/s GDDR6, and a 64 MB Infinity Cache. Streaming reads with no reuse gain nothing from a last-level cache; only the shared-prefix KV, read once per agent in a batched step, has the reuse a 64 MB cache can capture, and only while it stays under 64 MB.

Read the left side as a decision about which arrow touches the middle tier. The weight stream, the biggest arrow, drives straight past the Infinity Cache because none of its bytes will be asked for twice. The private KV does the same. The only arrow that loops through the cache is the shared-prefix KV, and the right side explains why: it is the only read whose bar sits near or below the 64 MB line, which is the same as saying it is the only read a 64 MB cache can hold onto.

The swarm creates the one reuse pattern that exists

Here is where the last two weeks pay off. When you run eight agents on one card and they share a system prompt, that prompt’s KV is no longer read once per step. It is read once per agent inside the same batched attention step, because each agent’s attention has to attend over the shared prefix as part of its own context. Eight agents, eight reads of the same physical bytes, within a window of a few milliseconds.

That is reuse, and it is exactly what a last-level cache is for. The first agent’s attention pulls the shared prefix out of GDDR and into the Infinity Cache; the other seven find it already resident and hit SRAM. The eight GDDR reads collapse to one, and they collapse whether or not the engine explicitly deduplicates the prefix in the kernel. The prefix sharing that RadixAttention formalizes as a software structure, the Infinity Cache delivers for free at the hardware level, as long as the shared bytes fit.

The catch is that qualifier. A 64 MB cache holds only so much KV, and the arithmetic is unforgiving: 64 MB divided by 147 KB per token is about 445 tokens. That is the longest shared prefix the cache can keep resident across a batched step. Below it the swarm gets the collapse; above it the prefix spills, and most of it is back to being read once per agent from GDDR.

Read in a decode stepBytesReads per stepWorking set vs 64 MBInfinity Cache payoff
Weight stream, Q4_K_M5.5 GB188xnone, runs at raw 640 GB/s
Private KV, one agent at 6,900 tokens1.0 GB116xnone
Shared prompt KV, 8 agents, 400 tokens59 MB80.9x, fitsup to ~8x fewer GDDR reads on the prefix
Shared prompt KV, 8 agents, 2,000 tokens294 MB84.6x, spillsonly the first ~445 tokens stay hot

The table is the whole argument in four rows. The two reads that dominate a decode token, the weights and the private KV, are far larger than the cache and read only once, so the cache does nothing for them. The shared-prefix read is the only one with both a small enough footprint and enough reuse to matter, and even it earns its keep only while the prompt stays under a few hundred tokens. A 400-token system prompt is comfortably inside the line; a 2,000-token one, which is an ordinary agent preamble, is nearly five times too big and mostly spills.

What this changes about tuning a local swarm

The first consequence is to stop expecting the Infinity Cache to show up in a decode benchmark. It will not. A single-stream token generator on this card is a GDDR6 machine with 64 MB of SRAM going almost entirely unused, and any decode tuning that assumes the cache is quietly buying effective bandwidth is tuning against a number that is not there. The memory-bound regime that Pope and colleagues treat as the master variable for decode is set by the bus, full stop.

The second consequence is more useful. If the shared-prefix KV is the only read the cache can help, then keeping that prefix short and hot is a real lever, and it is a different lever from the one the system-prompt post pulled. That post was about capacity: storing the shared prompt once in VRAM instead of eight times so it does not crowd out the private KV. This is about traffic: a shared prompt small enough to live in 64 MB is read from GDDR once per step across the whole batch, not once per agent. The two effects stack. Deduplicating the prefix saves the space, and fitting it under the cache line saves the bandwidth, and a swarm that keeps its common preamble under a few hundred tokens gets both.

There is a design tension worth naming, because it points somewhere. Quantizing the KV cache to fp8, the decode-bandwidth cut RDNA4 already has the WMMA for, halves the 147 KB per token, which roughly doubles how much shared prefix fits in the cache, from about 445 tokens to about 890. So the same feature that shrinks private KV traffic also widens the window where the Infinity Cache captures shared-prefix reuse. That is a rare case of one change paying off in two ledgers at once, and it is the sort of thing the PagedAttention line of work keeps surfacing: the KV cache, not the weights, is where a serving engine’s memory decisions actually bite.

The framing I want to leave is that RDNA4’s biggest cache is a locality machine, and local decode is a workload almost without locality. For a single user generating tokens, the 64 MB of Infinity Cache is 53.9 billion transistors’ worth of SRAM doing very little, and the honest bandwidth number is the raw bus. The agent swarm is the exception, and a narrow one: it manufactures reuse out of a shared prompt, and the cache pays it back exactly up to 64 MB and not a byte further. Knowing that line exists, and keeping the shared context under it, is the difference between a cache that quietly helps every step and one that watches the bus do all the work.