Last updated: 2026-11-17
Tokenization
Unicode Classes
Unicode classes for code points >= 0x80 used by the Qwen-family pretokenizer: \p{L}, \p{M}, \p{N} and \s.
ASCII is classified inline by the caller. The ranges are generated from the reference implementation's unicode-data.cpp (reference master 5f436dddb) so chunk boundaries match it.
8 exports shown
constant
letter_ranges
pub const letter_ranges = [_][2]u32{ No public doc comment yet.
constant
mark_ranges
pub const mark_ranges = [_][2]u32{ No public doc comment yet.
constant
number_ranges
pub const number_ranges = [_][2]u32{ No public doc comment yet.
constant
space_ranges
pub const space_ranges = [_][2]u32{ No public doc comment yet.
function
isLetter
pub fn isLetter(cp: u32) bool Returns true when a non-ASCII code point is a letter (\p{L}).
function
isMark
pub fn isMark(cp: u32) bool Returns true when a non-ASCII code point is a combining mark (\p{M}).
function
isNumber
pub fn isNumber(cp: u32) bool Returns true when a non-ASCII code point is a number (\p{N}).
function
isSpace
pub fn isSpace(cp: u32) bool Returns true when a non-ASCII code point is whitespace (\s).