Stepan
Zolotukhin

Systems engineer working on GPU inference.

I build fast local model inference for consumer GPUs.

ZINC is the engine. It runs GGUF models through Vulkan, ROCm, or Metal and includes a command line, browser chat, and an OpenAI-compatible API.

I publish the benchmark data and write down the kernel work, including the experiments that do not improve performance.

ZINC

A small engine for local inference.

ZINC is tuned on the hardware it claims to support: AMD Radeon, Intel Arc, and Apple Silicon. AMD users can choose either Vulkan or ROCm; NVIDIA CUDA remains experimental.

AMD / Vulkan & ROCm  ·   Intel / Vulkan  ·   Apple / Metal

Qwen 3.8 decode
32.43 tok/s
29.92 in llama.cpp
Qwen 3.8 prefill
391.3 tok/s
150.9 in llama.cpp
Workloads ahead
4/4
R9700 ROCm suite

Same Radeon AI PRO R9700, model file, prompts, and reusable-server method. See the complete run and provenance →

Models

What I am testing or following now.

Qwen 3.8 27B Local / dense

Current ROCm decode and speculative-decoding target.

 Muse Glimmer 30B Local / dense

30.1 tok/s decode in the published ROCm suite.

 Gemma 4 Local / dense + MoE

Dense and sparse checkpoints that exercise different GPU paths.

Recent writing

All 114 posts →

Q8 KV cache: a quarter of the bytes, seventeen percent slower decode on Metal

In July this blog argued a Q8 KV cache is close to free on an RDNA4 card. On August 17 we turned it off by default for Muse Glimmer on Apple Silicon, because it was costing 15 to 22 percent of decode. Both results are correct. Q8_0 stores a KV element in 1.06 bytes instead of 4, so the quantized path reads 3.8 times fewer bytes and still loses, which means the flash-attention kernel at ordinary context lengths is not bandwidth-bound at all. KV quantization is a capacity decision that a decode kernel charges you for in time.

The fastest Q5_K lm-head kernel was already in the binary, gated off by one constant

A commit on August 17 made Muse Glimmer decode about six percent faster on an M4 Max without adding a kernel. The faster Q5_K matvec was already compiled, already loaded, and never dispatched, because its threadgroup input cache was declared for K<=4096 and Muse's lm-head is K=6656. The lm-head is 925 MB of the roughly 15 GiB the decoder reads per token, and it was running on a translated shader that reads weights a byte at a time. The interesting part is not the fix, it is why the kernel was marked intentionally unused: a correct measurement about dispatch fusion got generalized into a wrong decision about a kernel that was never about dispatch count.

Qwen 3.8 27B Shipped Yesterday. Here's What ZINC's Day-One Support Actually Took.

Qwen 3.8 27B landed on Hugging Face on August 14 and ran through ZINC the next day. Two weeks later, the tuned ROCm path beats llama.cpp ROCm across all four published workloads. This is what architecture reuse bought us, what the new fused RMS-norm and Q8 path changed, and why Metal decode still needs work.

The KV cache lever the swarm keeps pulling stops being free below 8 bits

Four posts in this series reached for the same lever: quantize the KV cache to q8, halve every byte, double the headroom. That lever is real, and at 8 bits it is close to free. The trouble starts when the swarm wants more and reaches below 8 bits, because the KV cache is not one number to round off. Keys carry large per-channel outliers and values do not, so a single uniform 4-bit or 2-bit knob falls off a quality cliff that the papers only climb back up with per-channel key quantization and explicit outlier isolation. KIVI keeps 2-bit quality only by quantizing keys per-channel and values per-token; KVQuant holds 3-bit to under 0.1 perplexity loss only with pre-RoPE per-channel keys and dense-and-sparse outlier handling. On a single 16 GB RX 9070 XT that means the free win ends at 8 bits, and everything below it is an engineering project with a per-model accuracy budget, not a config flag.

Eight agents share one system prompt, so store the prefix KV once

Yesterday's post found the swarm had about 2 GiB of VRAM headroom on a 16 GB RX 9070 XT, and treated that as scarce. It is scarcer than it needs to be, because a good chunk of it is spent storing the same bytes eight times. Every agent in a coding swarm carries an identical prefix, the system prompt plus the tool schemas, and the naive engine caches a separate copy of that prefix's KV for each of the eight decode slots. A 2,000-token shared prefix is 0.30 GiB per copy at fp16, so eight copies cost 2.44 GiB while one shared copy costs 0.30 GiB. Storing it once reclaims 2.14 GiB, which is roughly the entire headroom the swarm was fighting for. This is the prefix-sharing case an earlier post said a single-user engine could mostly skip, and a swarm cannot.