From 0085ea1c2c7ddf32f2ad239da93f27257cf2bf6d Mon Sep 17 00:00:00 2001 From: _ov4 Date: Wed, 21 Oct 2026 12:42:53 +0000 Subject: docs(agents): split CONVENTIONS.md into topic files under AGENTS/CONVENTIONS Co-Authored-By: Claude Sonnet 5.5 --- AGENTS/CONVENTIONS/LEXING.md | 34 ++++++++++++++++++++++++++++++++++ 1 file changed, 34 insertions(+) create mode 100644 AGENTS/CONVENTIONS/LEXING.md (limited to 'AGENTS/CONVENTIONS/LEXING.md') diff --git a/AGENTS/CONVENTIONS/LEXING.md b/AGENTS/CONVENTIONS/LEXING.md new file mode 100644 index 0000000..81b01d1 --- /dev/null +++ b/AGENTS/CONVENTIONS/LEXING.md @@ -0,0 +1,34 @@ +# Conventions: text, tokens and lexing + +Part of [Load-bearing conventions](../CONVENTIONS.md). + +**Project text must be in English.** This applies to documentation, source comments, +user-facing messages, examples and test fixtures. Unicode test data must not contain +Chinese text. Keep user instructions in README files concise; put architecture, +implementation notes and AI guidance in the relevant `AGENTS/` directory. + +**Tokens are raw `int`.** `Lexer::gettok()` returns negative values for the `Token` +enum (`include/cps/Lexer.h`) and non-negative values for literal ASCII chars (`'('`, +`'+'`, `'<'`). Parser and CodeGen switch on both forms interchangeably; +`BinaryExprAST` stores this same int as its operator. Payloads (identifier text, +literal values) arrive out-of-band via public `Lexer` fields (`IdentifierStr`, +`StringVal`, `NumVal`, `RealVal`, `CharVal`) that must be read before the next +`gettok()`. + +**Keywords are UPPERCASE and case-sensitive.** `declare` silently lexes as an +identifier. Assignment is `<-` (ASCII only), equality is single `=`, inequality `<>`, +concatenation `&` (reaches the parser as raw ASCII 38 — no token). Identifiers use +ASCII case-insensitive lookup/comparison through `IdentifierMap`, `IdentifierSet` +and `sameIdentifier`; retain source spelling in ASTs and diagnostics. This includes +functions, types, enum values, fields and builtins, but not C_TYPE import strings. +Integer tokens carry a checked unsigned magnitude through lookahead; the parser +applies the sign and i64 range check, including the minimum value. String/char +literals support C++17 byte escapes and Unicode escapes (UTF-8 in STRING, ASCII only +in CHAR). Comments are `//` to end of line. + +Identifiers may begin with an underscore. The case-insensitive `__cps_` prefix is +reserved in every user identifier, including EXTERN declarations. + +**The lexer reads a caller-owned stream.** Construct it with `Lexer(std::istream &)`; +the stream must outlive the lexer. Lookahead, line number and error state belong to +each instance. The driver opens the source file; stdin is not used for compilation. -- cgit v1.3.1-4-g156e