UTF-8 and Unicode segmentation support for String's Character layer.
_utf8_decode returns one packed Int: scalar * 8 + size, where size is the number of bytes consumed (1-4). Packing avoids a per-scalar record allocation on the hot iteration path.
Ill-formed input consumes a maximal subpart (Unicode 17.0 §3.9.6, the same policy as Rust's fromutf8lossy and WHATWG): a truncated-but-valid prefix is one error unit, a lone continuation or invalid lead is one byte. Error units decode as U+FFFD for classification, but callers keep viewing the raw bytes — iteration over any byte string is lossless.
API Reference
Structs
struct Character
One extended grapheme cluster, represented as a borrowed UTF-8 byte range.
struct CharacterIterator
Walks extended grapheme clusters across a borrowed UTF-8 range.
struct ScalarIterator
Walks Unicode scalar values across a borrowed UTF-8 range.
Globals
let _WB_OTHER: Int
Internal code for UAX #29 word-break category Other.
let _WB_CR: Int
Internal code for UAX #29 word-break category CR.
let _WB_LF: Int
Internal code for UAX #29 word-break category LF.
let _WB_NEWLINE: Int
Internal code for UAX #29 word-break category Newline.
let _WB_EXTEND: Int
Internal code for UAX #29 word-break category Extend.
let _WB_ZWJ: Int
Internal code for UAX #29 word-break category ZWJ.
let _WB_REGIONAL_INDICATOR: Int
Internal code for UAX #29 word-break category Regional_Indicator.
let _WB_FORMAT: Int
Internal code for UAX #29 word-break category Format.
let _WB_KATAKANA: Int
Internal code for UAX #29 word-break category Katakana.
let _WB_HEBREW_LETTER: Int
Internal code for UAX #29 word-break category Hebrew_Letter.
let _WB_ALETTER: Int
Internal code for UAX #29 word-break category ALetter.
let _WB_SINGLE_QUOTE: Int
Internal code for UAX #29 word-break category Single_Quote.
let _WB_DOUBLE_QUOTE: Int
Internal code for UAX #29 word-break category Double_Quote.
let _WB_MIDNUMLET: Int
Internal code for UAX #29 word-break category MidNumLet.
let _WB_MIDLETTER: Int
Internal code for UAX #29 word-break category MidLetter.
let _WB_MIDNUM: Int
Internal code for UAX #29 word-break category MidNum.
let _WB_NUMERIC: Int
Internal code for UAX #29 word-break category Numeric.
let _WB_EXTENDNUMLET: Int
Internal code for UAX #29 word-break category ExtendNumLet.
let _WB_WSEGSPACE: Int
Internal code for UAX #29 word-break category WSegSpace.
let _WB_EXTENDED_PICTOGRAPHIC: Int
Internal code for the extended-pictographic override used by UAX #29.
Functions
func _utf8_decode(storage: ByteStorage, offset: Int, end: Int) -> Int
Decodes one UTF-8 scalar and packs its value with the consumed byte count. Malformed input yields U+FFFD and consumes one maximal invalid subpart.
func _grapheme_category(scalar: Int) -> Int
The scalar's grapheme-cluster-break category (a GC* constant), via binary search over the generated boundary list. O(log n) with ~12 probes; ASCII short-circuits.
func scalar_is_extended_pictographic(scalar: Int) -> Bool
Returns whether the scalar has the Unicode extended pictographic property.
func scalar_is_whitespace(scalar: Int) -> Bool
Tests the scalar against Unicode's White_Space property.
func scalar_is_alphabetic(scalar: Int) -> Bool
Tests the scalar against Unicode's Alphabetic property.
func scalar_is_numeric(scalar: Int) -> Bool
Returns true for decimal, digit, or other numeric Unicode scalars.
func scalar_is_alphanumeric(scalar: Int) -> Bool
Returns true when the scalar is Unicode alphabetic or numeric.
func scalar_is_ascii_digit(scalar: Int) -> Bool
Returns true when scalar is the code point for 0...9.
func scalar_is_ascii_hexdigit(scalar: Int) -> Bool
Returns true for ASCII 0...9, a...f, or A...F.
func scalar_is_lowercase(scalar: Int) -> Bool
Tests the scalar against Unicode's Lowercase property.
func scalar_is_uppercase(scalar: Int) -> Bool
Tests the scalar against Unicode's Uppercase property.
func scalar_is_titlecase(scalar: Int) -> Bool
Returns true when the scalar has Unicode general category Lt.
func scalar_is_decimal(scalar: Int) -> Bool
Returns true when the scalar has Unicode general category Nd.
func scalar_is_digit(scalar: Int) -> Bool
Tests whether Unicode assigns the scalar decimal or digit numeric status.
func scalar_is_punctuation(scalar: Int) -> Bool
Returns true for any Unicode punctuation general category.
func scalar_is_symbol(scalar: Int) -> Bool
Returns true for any Unicode symbol general category.
func scalar_is_control(scalar: Int) -> Bool
Returns true when the scalar has Unicode general category Cc.
func scalar_is_printable(scalar: Int) -> Bool
Excludes control, format, surrogate, private-use, and unassigned categories.
func scalar_is_cased(scalar: Int) -> Bool
Tests Unicode's derived Cased property used by case conversion.
func scalar_is_case_ignorable(scalar: Int) -> Bool
Tests Unicode's Case_Ignorable property used when finding cased neighbors.
func _word_break_category(scalar: Int) -> Int
Looks up the Unicode word-break category assigned to scalar.
func _case_mapping(scalar: Int, kind: Int) -> Substring?
Borrows the full Unicode case mapping for one scalar.
func _decomposition(scalar: Int, compatibility: Bool) -> Substring?
Looks up a normalization decomposition without allocating.
func _combining_class(scalar: Int) -> Int
Returns scalar's canonical combining class.
func _compose_pair(starter: Int, combining: Int) -> Int?
Looks up canonical composition for a starter and following combining scalar.
func _next_boundary(storage: ByteStorage, offset: Int, end: Int) -> Int
The byte offset just past the extended grapheme cluster starting at offset, which must itself be a cluster boundary. Because scans start at true boundaries, all UAX #29 rule state (GB11's emoji sequence, GB9c's conjunct chain, GB12/13's regional-indicator parity) is local to this one scan — the iterator carries none.