Last updated: 2026-11-17

Tokenization

Unicode Classes

All API Sections

Unicode classes for code points >= 0x80 used by the Qwen-family pretokenizer: \p{L}, \p{M}, \p{N} and \s.

ASCII is classified inline by the caller. The ranges are generated from the reference implementation's unicode-data.cpp (reference master 5f436dddb) so chunk boundaries match it.

8 exports shown

constant

letter_ranges

#
pub const letter_ranges = [_][2]u32{

No public doc comment yet.

src/model/unicode_classes.zig:7

constant

mark_ranges

#
pub const mark_ranges = [_][2]u32{

No public doc comment yet.

src/model/unicode_classes.zig:175

constant

number_ranges

#
pub const number_ranges = [_][2]u32{

No public doc comment yet.

src/model/unicode_classes.zig:256

constant

space_ranges

#
pub const space_ranges = [_][2]u32{

No public doc comment yet.

src/model/unicode_classes.zig:293

function

isLetter

#
pub fn isLetter(cp: u32) bool

Returns true when a non-ASCII code point is a letter (\p{L}).

Parameters

cp
Unicode code point, expected >= 0x80.

Returns

Whether the code point is in a letter range.

src/model/unicode_classes.zig:317

function

isMark

#
pub fn isMark(cp: u32) bool

Returns true when a non-ASCII code point is a combining mark (\p{M}).

Parameters

cp
Unicode code point, expected >= 0x80.

Returns

Whether the code point is in a mark range.

src/model/unicode_classes.zig:324

function

isNumber

#
pub fn isNumber(cp: u32) bool

Returns true when a non-ASCII code point is a number (\p{N}).

Parameters

cp
Unicode code point, expected >= 0x80.

Returns

Whether the code point is in a number range.

src/model/unicode_classes.zig:331

function

isSpace

#
pub fn isSpace(cp: u32) bool

Returns true when a non-ASCII code point is whitespace (\s).

Parameters

cp
Unicode code point, expected >= 0x80.

Returns

Whether the code point is in the whitespace set.

src/model/unicode_classes.zig:338