Little's Law caps a responsive RDNA4 agent swarm near twelve

The last two posts argued about who should get a decode slot. Fair queuing stops one agent from starving the rest, and reading-speed awareness says the agent a human is watching needs only a thin slice of one. Both took the number of agents as fixed and fought over how to divide the card between them.

There is a question those posts skipped. How many agents should be in the session in the first place? On one RX 9070 XT the KV cache fits about eight concurrent decode slots, and it is tempting to read that as the answer: eight agents, one per slot, done. That is the wrong number, and it is wrong in a useful direction. The card can keep more than eight agents feeling instant, and the reason is a sixty-year-old result about queues.

The number that matters is not how many agents decode at once. It is how many can be alive in a coding session before the card stops keeping up with them. Those are different because an agent is not decoding most of the time. It decodes a burst, then goes off to run a test, read a file, or wait on a compiler, and while it does that its slot is free. Counting agents by slots ignores all that idle time, and the idle time is where the extra capacity lives.

An agent is a loop, not a stream

Start with what one agent actually does. It is not a firehose of tokens. It is a loop: generate a turn of output, hand that output to a tool, wait for the tool, fold the result back into context, then generate the next turn. A local coding agent reads far more than it writes, and every one of those reads is time the agent is not on the GPU.

Put numbers on the loop. A mean agent turn on this card runs about 354 tokens, which is the average of the mixed workload this series has been using, everything from a 40-token tool check to a 900-token refactor plan. At the card’s per-slot rate of roughly 38.9 tokens per second, that turn takes about 9.1 seconds to decode. Call the time between turns, the tool call and the reading, the think time. Say it is 4 seconds, which is modest for an agent that runs a test or reads a couple of files between generations.

So each agent spends 9.1 seconds needing the GPU and 4 seconds not needing it, a cycle of 13.1 seconds. For a bit less than a third of its life, its decode slot is available to somebody else. That fraction is the whole argument. If you only ever put eight agents on the card, the slot sits idle during every one of those 4-second gaps, and the card is doing less work than it could.

Little’s Law turns the loop into a capacity number

This is exactly the setting John Little formalized in 1961. Little’s Law says that for any stable system, the average number of items inside it equals the average arrival rate times the average time each item spends there: L equals lambda times W. It is almost embarrassingly general, holding for any arrival pattern and any service discipline as long as the averages are finite, which is why it shows up everywhere from checkout lines to CPU schedulers.

Applied to a closed loop of agents, Little’s Law rearranges into a form that performance engineers call the interactive response-time law. If N agents each cycle through a decode turn and a think time Z, and the whole system delivers throughput X turns per second, then the response time an agent waits for its turn is R equals N divided by X, minus Z. That single equation carries the entire story, because X cannot grow forever. It rises as you add agents, right up until the card’s decode slots are all busy, and then it flattens at the ceiling this series keeps hitting, 311 tokens per second, or about 0.88 agent-turns per second.

Below that ceiling, adding an agent raises X and R stays pinned at its floor of one turn’s decode time. Once X is flat, the equation has only one free variable left, so every new agent has to show up as a bigger R. The crossover, the point where responsiveness stops being free and starts costing wait, sits at roughly the slot count times one plus the think-to-decode ratio. Here that is 8 times 1 plus 4 over 9.1, about 11.5. Round it and the card keeps about twelve agents responsive, not eight.

The two curves bend at the same place, and that is the point of the chart. Left of the knee, the gold throughput line is doing the work: each agent you add is worth about 27 more tokens per second, and the terracotta response line does not move, because there is always a free slot waiting. Right of the knee, the gold line is flat. The card is already producing every token it can, so the thirteenth agent does not add throughput. It adds queue, and the terracotta line turns upward and climbs for the rest of the chart.

What the knee costs on the wrong side

The linear rise past the knee is easy to underrate, so it is worth reading off the exact numbers. Below saturation each agent’s turn completes in its bare decode time, about 9.1 seconds, with no waiting. Above saturation the response time grows by the bottleneck’s per-turn demand, which on eight slots is 9.1 divided by 8, about 1.1 seconds, for every agent added.

Live agents	Aggregate throughput	Response time per turn	Regime
4	108 tok/s	9.1 s	slots idle, card underused
8	216 tok/s	9.1 s	still below the knee
12	311 tok/s	~9.7 s	the knee, card just saturated
16	311 tok/s	14.2 s	throughput pinned, wait growing
24	311 tok/s	23.3 s	every added agent is pure latency

Read the throughput column down and it stops moving at twelve. Read the response-time column and it does the opposite, more than doubling from the knee to 24 agents while the card produces not one extra token. That is the trap of sizing a swarm by eye. Doubling the agent count from twelve to 24 feels like more parallelism, and on a throughput dashboard nothing looks wrong, but the actual effect is that every agent now waits two and a half times as long for a turn that used to be instant. The parallelism was already spent at the knee.

Why the practical target sits below the knee

The response-time law describes averages, and averages hide the thing a user actually feels, which is the bad moment. Agent workloads are not smooth. Turn lengths swing across an order of magnitude, and agents tend to arrive at the GPU in waves, several of them finishing a shared tool call at the same instant and all demanding a slot together. That burstiness does not change the average throughput, but it does change the waiting.

Kingman’s formula is the standard way to see it. For a general single-queue system, mean waiting time is approximately the utilization over one minus utilization, times a variability factor that averages the squared coefficients of variation of the arrival and service processes, times the service time. The utilization term is the part that bites: as the card approaches full, that ratio explodes toward infinity, and any variability at all multiplies the explosion. A workload with bursty arrivals and highly variable turn lengths starts feeling the blowup well before utilization reaches one.

The practical consequence is to leave headroom. Running the card at its exact saturation point means a single burst has no slack to absorb into, and the burst becomes a visible stall. Targeting something like 80 percent of the knee, roughly nine or ten agents rather than twelve here, keeps the average utilization comfortable and gives bursts somewhere to go. The knee is the ceiling; the target is a step below it.

Where this sits in the stack

None of this replaces the scheduling posts. It sits under them. Continuous batching sets the 311 tokens per second ceiling the throughput curve flattens against. Chunked prefill keeps a big incoming prompt from freezing the agents already decoding. Fair queuing and reading-speed awareness decide how the eight live slots get divided once more than eight agents want them. The response-time law answers the question that comes before all of those: how many agents to admit to the session at all.

There is a control knob hiding in the derivation, and it is not the GPU. The knee moved past the slot count only because of the think time between turns, the ratio Z over S. An agent that reads a lot between short generations has a high ratio and stretches the knee well past eight; an agent that emits long outputs with little tool use has a low ratio and saturates near the slot count. That means the way to fit more agents on one card is not always a faster kernel or a bigger KV budget. Sometimes it is to shape the workload so agents spend a healthy fraction of each cycle off the GPU, which is exactly what tool-heavy coding agents already do.

Production engines expose almost none of this. vLLM lets you cap concurrency with max-num-seqs and pick a scheduling policy, but the number you put there is a guess unless you have measured your agents’ think-to-decode ratio, and the default is set for server traffic, not a handful of long-lived local agents. An engine like zinc that owns its own submission path can do better: measure Z and S from the running session, compute the knee, and refuse to over-admit past it. The Orca line of work made the scheduler operate at iteration granularity precisely so it could reason about the batch this finely. On a single consumer card, where there is no autoscaler to hide a bad guess, knowing that eight slots means about twelve agents, and that the twelfth is the last free one, is the difference between a swarm that feels instant and one that quietly queues up behind itself.