Last updated: 2026-07-19
Model Format & Loading
Loader Cuda
CUDA-specific model loading — mmap the GGUF, upload every tensor to the NVIDIA device, and expose them to the CUDA forward pass.
Unlike the Metal loader (Apple unified memory, zero-copy newBufferWithBytesNoCopy), CUDA device memory is NOT CPU-visible, so each tensor is copied host->device once at load time via `cuda_buffer.uploadMmap`. The file mapping is kept alive after upload because the embedding lookup (`dequantEmbeddingRow`) dequantizes a single `token_embd.weight` row on the CPU directly out of the mapping.
GGUF parsing and config extraction are backend-agnostic; the config extractor below is copied verbatim from loader_metal.zig (pure metadata reads, no backend dependency). `dequantRow`/`getScaleMinK4` are copied from compute/forward_metal.zig to avoid importing the Metal forward pass (which would drag in Metal-only dependencies).
4 exports shown
struct
LoadedTensor
pub const LoadedTensor = structA GGUF tensor descriptor paired with the device buffer holding its weights.
`info` carries the on-disk shape/quant/offset metadata; `gpu_buffer` is the device-local copy uploaded at load time.
struct
Model
pub const Model = structRuntime model state: parsed config, the live CUDA context, every weight tensor resident on the GPU, and the still-mapped GGUF file (kept alive for CPU-side embedding dequantization).
Methods
5method
Model.load
pub fn load(allocator: std.mem.Allocator, ctx: ?*shim.CudaCtx, path: []const u8) !ModelOpen + mmap the GGUF at `path`, parse it, extract the config, and upload every tensor to `ctx`.
The mapping is retained for embedding lookups.
method
Model.get
pub fn get(self: *const Model, name: []const u8) ?*const LoadedTensorLook up a tensor by its exact GGUF name (e.g.
`"blk.3.attn_q.weight"`).
method
Model.getLayer
pub fn getLayer(self: *const Model, layer: u32, suffix: []const u8) ?*const LoadedTensorLook up a per-layer tensor by formatting `"blk.{layer}.{suffix}"`.
method
Model.dequantEmbeddingRow
pub fn dequantEmbeddingRow(self: *const Model, token_id: u32, out: []f32) voidDequantize one row of `token_embd.weight` to f32 on the CPU.
Reads the raw tensor bytes directly out of the live mapping (no device round-trip). `token_id` is clamped to the last valid row.
method
Model.deinit
pub fn deinit(self: *Model) voidRelease device buffers, the GGUF metadata, the name map, and the mapping.
function
inspectConfig
pub fn inspectConfig(path: []const u8, allocator: std.mem.Allocator) !ModelConfigInspect a GGUF model's config without uploading any weights to the GPU.
mmaps the file, parses the header, extracts the `ModelConfig`, then unmaps — used by `zinc --check` to report architecture/dims on the CUDA backend.
function
dequantRow
pub fn dequantRow(raw_data: []const u8, row: u32, cols: u32, quant_type: GGMLType, output: []f32) voidDequantize one row of quantized weight data to f32 values.
Supports f32, f16, Q5_0, Q5_1, Q8_0, Q4_K, Q5_K, Q6_K, and MXFP4. Unsupported types log a warning and zero the output slice.