The self-hosted lexer (ADR 0043 stage 2). Token values stay opaque to user code outside the syntax API. Procedural invocation records cross the private macro-service seam with their canonical kind tags and ranges. lex is a validation export rendering the frozen dump-format token section compared byte-for-byte by the differential harness.
API Reference
Structs
struct Token
One canonical lexical token with complete and parser-facing byte ranges.
struct Comment
A dropped line comment's span, preserved for the parse dump's comments section.
struct ScanResult
The partial products of one deterministic scanner pass.
struct Group
A balanced token-tree group with explicit opening and closing tokens.
struct TokenRange
Canonical tokens owned by one half-open source extent.
struct MacroInput
A procedural macro invocation's source identity, text, and balanced token tree.
struct TokenTreeResult
Completed top-level trees plus the first delimiter-balance failure.
Enums
enum TokenKind
Every lexical category emitted by the canonical scanner.
enum Delimiter
The paired delimiters that can own a token-tree group.
enum TokenTree
A balanced macro token tree.
enum TokenTreeError
A delimiter-balance failure encountered while building token trees.
Functions
func kind_name(kind: TokenKind) -> String
Returns the frozen PascalCase name used by validation dumps.
func kind_spelling(kind: TokenKind) -> String
The user-facing spelling of a kind (the reference TokenKind::as_str): literal spellings verbatim, token classes as placeholders.
func scan(source: String) -> ScanResult
Scans UTF-8 source into canonical tokens and detached line comments.
func append(mut buf: [Byte], text: String) -> Void
Appends the UTF-8 bytes of text to a dump buffer.
func append_int(mut buf: [Byte], value: Int) -> Void
Appends the decimal spelling of value to a dump buffer.
func append_snippet(mut buf: [Byte], view: UTF8View, start: Int, end: Int) -> Void
Appends a diagnostic snippet for a short, single-line byte range.
func lex(source: String) -> String
Scans source and renders the frozen token validation format.
func capture_token_trees(tokens: [Token]) -> TokenTreeResult
Capture a flat token stream into balanced trees. Delimiter failures are structured so a macro caller can attach diagnostics without parsing text.
func decode_token_records(encoded: String) -> [Token]?
Decode canonical token records without imposing tree structure: the wrapper macro ABI hands over a declaration's token run, which is not a single delimited group. Empty input decodes to an empty token list.
func capture_macro_input(source_id: Int, source: String, encoded: String) -> MacroInput?
Decode the parser-recorded canonical token records supplied by the macro service ABI. This does not inspect token spelling or run the lexer again.
func token_tree_range(tree: &TokenTree) -> TokenRange
Flatten one tree back to its canonical tokens, preserving every token and byte range. This is a token operation, not source reconstruction.
func group_contents(group: &Group) -> TokenRange
The tokens inside a group, excluding its delimiters. Empty groups still retain a meaningful source extent for parser EOF diagnostics.
func indent(mut buf: [Byte], depth: Int) -> Void
Appends indentation for the requested nesting depth.
func trees(source: String) -> String
Scans, balances, and renders macro token trees in the validation format.