Last updated: 2026-11-17
Tokenization
Bench Tokenize
CPU-only tokenizer benchmark; no tensor upload or GPU initialization.
Measures warmed prompt encoding against one GGUF vocabulary and verifies every timed token sequence against the initial result.
1 exports shown
function
main
pub fn main(process_init: std.process.Init) !void Usage: zinc-tokenize-bench <model.gguf> <text-file> [runs].
Parse the vocabulary once, warm it, then measure complete prompt encoding.