diff options
Diffstat (limited to 'AGENTS/CONVENTIONS/LEXING.md')
| -rw-r--r-- | AGENTS/CONVENTIONS/LEXING.md | 13 |
1 files changed, 12 insertions, 1 deletions
diff --git a/AGENTS/CONVENTIONS/LEXING.md b/AGENTS/CONVENTIONS/LEXING.md index 81b01d1..654d7ae 100644 --- a/AGENTS/CONVENTIONS/LEXING.md +++ b/AGENTS/CONVENTIONS/LEXING.md @@ -20,7 +20,8 @@ identifier. Assignment is `<-` (ASCII only), equality is single `=`, inequality concatenation `&` (reaches the parser as raw ASCII 38 — no token). Identifiers use ASCII case-insensitive lookup/comparison through `IdentifierMap`, `IdentifierSet` and `sameIdentifier`; retain source spelling in ASTs and diagnostics. This includes -functions, types, enum values, fields and builtins, but not C_TYPE import strings. +functions, types, enum values, fields and builtins, but not C_TYPE import strings +or preprocessor switch names. Integer tokens carry a checked unsigned magnitude through lookahead; the parser applies the sign and i64 range check, including the minimum value. String/char literals support C++17 byte escapes and Unicode escapes (UTF-8 in STRING, ASCII only @@ -32,3 +33,13 @@ reserved in every user identifier, including EXTERN declarations. **The lexer reads a caller-owned stream.** Construct it with `Lexer(std::istream &)`; the stream must outlive the lexer. Lookahead, line number and error state belong to each instance. The driver opens the source file; stdin is not used for compilation. + +**Preprocessing precedes lexing.** `preprocess(std::istream &)` returns masked source, +per-line __LIB states and an error flag. PP_DEFINE/PP_UNDEF change case-sensitive, +file-local switches; PP_IFDEF/PP_IFNDEF/PP_ENDIF maintain a nested conditional stack. +Directives occupy a line, with optional leading whitespace and trailing // comments. +Disabled ordinary source is never tokenized; DEFINE/UNDEF in it have no effect. +Masking preserves byte lengths, CRLF and original source line numbers. The driver +owns the processed stream passed to Lexer. PP_* keywords remain reserved outside +valid directives. There is no macro expansion, value, PP_ELSE or switch propagation +through INCLUDE; line-leading # directives are errors. |