Last updated: 2026-11-17

Tokenization

Bench Tokenize

All API Sections

CPU-only tokenizer benchmark; no tensor upload or GPU initialization.

Measures warmed prompt encoding against one GGUF vocabulary and verifies every timed token sequence against the initial result.

1 exports 0 methods src/bench_tokenize.zig

1 exports shown

function

main

#
pub fn main(process_init: std.process.Init) !void

Usage: zinc-tokenize-bench <model.gguf> <text-file> [runs].

Parse the vocabulary once, warm it, then measure complete prompt encoding.

src/bench_tokenize.zig:12