Gemma 4 Local Inference

When This Page Helps

Use this page when you are deciding whether Gemma is a model-family port, a benchmark target, or a writing cluster. The useful reader intent is specific: what breaks differently from Qwen, what fits locally, and what an AMD or Metal engine has to optimize next.

Status Snapshot

Best current answer

Gemma is the right second pillar for ZINC coverage because it proves the engine is not only tuned for Qwen-shaped models.

Reader problem

People landing here need to know which Gemma variant fits, why it is architecturally different, and what performance number is credible on their GPU.

Main bottleneck

For larger Gemma runs, time-to-first-token is the risk area: sliding-window attention, asymmetric Q/KV dimensions, and command submission overhead show up before decode looks bad.

Action Plan

1. Start from the model shape Separate dense Gemma runs from A4B MoE runs before comparing numbers. They stress different kernels and memory paths.
2. Measure prefill first Gemma can look healthy in decode while still losing badly on time-to-first-token because prompt batching and attention shape are the hard parts.
3. Keep Gemma posts practical The strongest future posts should answer a concrete local-running question: head dimensions, sliding windows, MoE routing, Metal parity, or RDNA4 prefill.
4. Link back to benchmarks Readers landing from search need the current ZINC result, the llama.cpp baseline if available, and the exact hardware class.

Operator Checklist

• Name the checkpoint Use the exact model id, quantization, and whether it is dense or A4B MoE before making any claim.
• Confirm backend support Check whether the run is using the batched Gemma path or a fallback path; the difference changes the conclusion.
• Capture fit and context Record VRAM budget, reserved KV cache, active context, and whether any experts or tensors are offloaded.
• Compare user-visible latency Report TTFT and prompt throughput before decode tokens per second. Gemma pain is often visible before generation starts.

What Matters

• Gemma 4 uses sliding-window attention for most layers, so long-context memory does not grow the same way on every layer.
• Full-attention Gemma layers can use different Q and KV head dimensions, which breaks kernels that assume one shared head_dim.
• Gemma MoE is not identical to Qwen MoE: activations, norms, and residual placement differ.
• Prefill performance depends on batching the prompt correctly; a per-token path wastes bandwidth on large Gemma models.

What To Measure

TTFT

The first metric for Gemma posts should be time-to-first-token on a fixed prompt length.

Prompt tok/s

Prefill throughput tells whether the sliding-window and full-attention layers are batched correctly.

Decode tok/s

Decode still matters, but it should be paired with model size, active parameters, and context length.

Memory shape

Show weights, runtime buffers, reserved KV cache, and any offload mode separately.

Common Traps

• Assuming Qwen kernels are enough Gemma changes attention dimensions, activation behavior, norm placement, and sometimes MoE layout. A generic transformer path is not a full answer.
• Publishing decode-only conclusions Large Gemma prompts are often prefill-bound. Decode tokens per second alone can make the local experience look better than it is.
• Writing generic model coverage Searchers do not need another broad Gemma overview. The opportunity is local inference mechanics on real hardware.

Useful Next Posts

• Gemma 4 on 16 GB vs 32 GB RDNA4 A practical fit guide for RX 9070 XT-class cards versus R9700-class cards.
• Sliding-window attention in local inference Explain what Gemma saves, what it does not save, and where full-attention layers still dominate.
• Gemma A4B MoE versus Qwen A3B MoE A direct comparison of routing, active parameters, memory residency, and prefill behavior.

Deep Reads

Run It

Common Questions