Token

struct Token

One canonical lexical token with complete and parser-facing byte ranges.

span covers the full spelling. lexeme excludes sigils and identifier quotes when name resolution needs only the payload; for ordinary tokens the ranges are equal. Both are half-open UTF-8 byte offsets into the same source.

Properties

let kind: TokenKind

The lexical category assigned by the scanner.

let span: Range<Int>

The complete token spelling as a half-open byte range.

let lexeme: Range<Int>

The parser-facing value as a half-open byte range. This excludes sigils and identifier quotes; for ordinary tokens it equals span.