Last updated: 2026-09-07

Scheduler

Scheduler

All API Sections

Continuous-batching scheduler groundwork for concurrent inference requests.

Today this module owns request slot accounting and state collection only. The HTTP serving hot path still serializes generation behind ServerState.generation_mutex; the batched prefill/decode dispatch loop is not wired yet.

1 exports 14 methods src/scheduler/scheduler.zig

1 exports shown

struct

Scheduler

# 
pub const Scheduler = struct

Fixed-capacity pool of request slots used to track concurrent inference requests.

Each slot holds at most one active `Request`; slots are reused once released.

src/scheduler/scheduler.zig:16

Methods

14

method

Scheduler.init

# 
pub fn init(allocator: std.mem.Allocator, max_parallel: u32) !Scheduler

Initialize the scheduler with a fixed number of concurrent request slots.

Parameters
allocator
Allocator for the slot array.
max_parallel
Maximum number of concurrent requests.
Returns

A Scheduler with all slots initially empty.

src/scheduler/scheduler.zig:37

method

Scheduler.enqueue

# 
pub fn enqueue(self: *Scheduler, prompt_tokens: []const u32, params: GenerationParams) !u64

Enqueue a new request without assigning a slot (continuous-batching path).

The request sits in `pending` (state `.pending`) until `admitNext` moves it into a free slot. Unlike `submit`, this never fails on a full slot array — arrivals queue and are admitted as slots free, which is what lets a running batch admit/evict sequences between decode steps.

Returns

The new request's unique id.

src/scheduler/scheduler.zig:58

method

Scheduler.admitNext

# 
pub fn admitNext(self: *Scheduler) !?u32

Admit the oldest pending request into the first free slot, if any.

Moves it out of the `pending` queue, assigns `slot_id`, and transitions it to `.prefilling`. The caller then runs prefill for every slot reported by `pendingPrefill` and transitions those to `.decoding`.

Returns

The assigned slot index, or null if no pending request or no free slot.

src/scheduler/scheduler.zig:72

method

Scheduler.isIdle

# 
pub fn isIdle(self: *const Scheduler) bool

True if there is no outstanding work: every slot empty and no waiters.

src/scheduler/scheduler.zig:96

method

Scheduler.submit

# 
pub fn submit(self: *Scheduler, prompt_tokens: []const u32, params: GenerationParams) !u32

Submit a new request and assign it to the first free slot.

Parameters
self
Scheduler to submit to.
prompt_tokens
Tokenized prompt for the request.
params
Generation parameters (max_tokens, temperature, etc.).
Returns

The slot index that was assigned; pass this value to `release` when the request completes.

Notes

Returns `error.AllSlotsBusy` if every slot is occupied.

src/scheduler/scheduler.zig:106

method

Scheduler.isFull

# 
pub fn isFull(self: *const Scheduler) bool

Check if all slots are occupied.

Parameters
self
Scheduler to query.
Returns

True if every slot holds an active request.

src/scheduler/scheduler.zig:125

method

Scheduler.activeCount

# 
pub fn activeCount(self: *const Scheduler) u32

Get the number of active (non-null) requests.

Parameters
self
Scheduler to query.
Returns

Count of occupied slots.

src/scheduler/scheduler.zig:132

method

Scheduler.transition

# 
pub fn transition(self: *Scheduler, slot_id: u32, new_state: RequestState) !void

Transition a live slot through the request state machine.

Parameters
self
Scheduler to query.
slot_id
Slot index to update.
new_state
Target request state.
Returns

error.InvalidSlot if the slot is out of range or empty.

src/scheduler/scheduler.zig:145

method

Scheduler.collectByState

# 
pub fn collectByState(self: *const Scheduler, state: RequestState, out: []u32) []u32

Collect slot IDs whose request currently has `state`.

Parameters
self
Scheduler to query.
state
Request state to match.
out
Caller-owned scratch buffer for slot IDs.
Returns

A slice of `out` containing the collected slot IDs.

src/scheduler/scheduler.zig:159

method

Scheduler.pendingPrefill

# 
pub fn pendingPrefill(self: *Scheduler) []u32

Slot IDs of requests in the `.prefilling` state (admitted, prompt not yet processed).

The driver runs prefill for each, then transitions it to `.decoding`. activeDecoding call.

Returns

A slice into `self.scratch`, valid until the next pendingPrefill /

src/scheduler/scheduler.zig:178

method

Scheduler.activeDecoding

# 
pub fn activeDecoding(self: *Scheduler) []u32

Slot IDs of requests in the `.decoding` state (the running decode batch).

The driver gathers (token, position, slot) per id and issues ONE batched decode step over them. activeDecoding call.

Returns

A slice into `self.scratch`, valid until the next pendingPrefill /

src/scheduler/scheduler.zig:196

method

Scheduler.release

# 
pub fn release(self: *Scheduler, slot_id: u32) void

Release a completed or cancelled request's slot, freeing its resources.

Parameters
self
Scheduler to release from.
slot_id
Slot index to free (the value returned by `submit`).
Notes

Silently does nothing if `slot_id` is out of range or the slot is already empty.

src/scheduler/scheduler.zig:213

method

Scheduler.deinit

# 
pub fn deinit(self: *Scheduler) void

Tear down all active and pending requests and free owned buffers.

Parameters
self
Scheduler to destroy.

src/scheduler/scheduler.zig:225