Last updated: 2026-07-19

Inference Runtime

Batching

All API Sections

Tenant-aware batch planning for ZINC_RT.

This module owns admission, quotas, and prefill/decode batch selection. It is intentionally independent of the current host-assisted `forward_zinc_rt` execution path so it can be validated before the M3 continuous-batching executor consumes it.

9 exports 14 methods src/zinc_rt/batching.zig

9 exports shown

constant

TenantId

# 
pub const TenantId = u32

Stable tenant identifier supplied by the API/server layer.

src/zinc_rt/batching.zig:11

constant

RequestId

# 
pub const RequestId = u64

Monotonic request identifier assigned at admission.

src/zinc_rt/batching.zig:13

struct

TenantLimits

# 
pub const TenantLimits = struct

Per-tenant admission and scheduling limits.

src/zinc_rt/batching.zig:16

struct

RequestConfig

# 
pub const RequestConfig = struct

Request metadata needed by the ZINC_RT batch planner.

src/zinc_rt/batching.zig:26

enum

RequestState

# 
pub const RequestState = enum

Lifecycle tracked by the ZINC_RT batch planner.

src/zinc_rt/batching.zig:36

struct

Slot

# 
pub const Slot = struct

One request slot in the multitenant planner.

src/zinc_rt/batching.zig:50

Methods

2

enum

BatchKind

# 
pub const BatchKind = enum

Type of work represented by a batch entry.

src/zinc_rt/batching.zig:78

struct

BatchEntry

# 
pub const BatchEntry = struct

One request inside a selected prefill or decode batch.

src/zinc_rt/batching.zig:84

struct

BatchScheduler

# 
pub const BatchScheduler = struct

Fixed-capacity multitenant scheduler for ZINC_RT prefill/decode batches.

src/zinc_rt/batching.zig:102

Methods

12

method

BatchScheduler.init

# 
pub fn init(allocator: std.mem.Allocator, max_slots: u32, max_tenants: u32) !BatchScheduler

Initialize a fixed-capacity scheduler.

Parameters
allocator
Allocator for slot and tenant arrays.
max_slots
Maximum live requests across all tenants.
max_tenants
Maximum registered tenants.

src/zinc_rt/batching.zig:114

method

BatchScheduler.registerTenant

# 
pub fn registerTenant(self: *BatchScheduler, tenant_id: TenantId, limits: TenantLimits) !void

Register or update limits for a tenant.

Parameters
self
Scheduler to mutate.
tenant_id
Tenant identifier.
limits
New tenant limits.

src/zinc_rt/batching.zig:136

method

BatchScheduler.submit

# 
pub fn submit(self: *BatchScheduler, config: RequestConfig) !u32

Submit a request into the first free slot.

Returns

Assigned slot ID.

src/zinc_rt/batching.zig:161

method

BatchScheduler.selectPrefillBatch

# 
pub fn selectPrefillBatch(self: *BatchScheduler, out: []BatchEntry, max_prompt_tokens: u32) []BatchEntry

Select queued/prefilling requests for prompt prefill.

Parameters
out
Caller-owned scratch for entries.
max_prompt_tokens
Total prompt-token budget for this batch.
Returns

A slice of `out` with selected prefill entries.

src/zinc_rt/batching.zig:193

method

BatchScheduler.advancePrefill

# 
pub fn advancePrefill(self: *BatchScheduler, slot_id: u32, tokens: u32) !void

Account for completed prompt work.

Parameters
slot_id
Slot that consumed prompt tokens.
tokens
Number of prompt tokens consumed.

src/zinc_rt/batching.zig:240

method

BatchScheduler.selectDecodeBatch

# 
pub fn selectDecodeBatch(self: *BatchScheduler, out: []BatchEntry, max_slots: u32) []BatchEntry

Select active decode requests fairly across slots and tenant limits.

Parameters
out
Caller-owned scratch for entries.
max_slots
Maximum entries to select.
Returns

A slice of `out` with selected decode entries.

src/zinc_rt/batching.zig:254

method

BatchScheduler.recordDecodeToken

# 
pub fn recordDecodeToken(self: *BatchScheduler, slot_id: u32) !void

Account for one generated decode token.

Parameters
slot_id
Slot that emitted a token.

src/zinc_rt/batching.zig:290

method

BatchScheduler.release

# 
pub fn release(self: *BatchScheduler, slot_id: u32) void

Release a completed or cancelled slot.

src/zinc_rt/batching.zig:308

method

BatchScheduler.tenantActiveCount

# 
pub fn tenantActiveCount(self: *const BatchScheduler, tenant_id: TenantId) u32

Number of live requests owned by `tenant_id`.

src/zinc_rt/batching.zig:331