docs(agents): split CONVENTIONS.md into topic files under AGENTS/CONVENTIONS

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>

1 files changed, 34 insertions, 0 deletions

diff --git a/AGENTS/CONVENTIONS/LEXING.md b/AGENTS/CONVENTIONS/LEXING.md
new file mode 100644
index 0000000..81b01d1
--- /dev/null
+++ b/AGENTS/CONVENTIONS/LEXING.md
@@ -0,0 +1,34 @@
+# Conventions: text, tokens and lexing
+
+Part of [Load-bearing conventions](../CONVENTIONS.md).
+
+**Project text must be in English.** This applies to documentation, source comments,
+user-facing messages, examples and test fixtures. Unicode test data must not contain
+Chinese text. Keep user instructions in README files concise; put architecture,
+implementation notes and AI guidance in the relevant `AGENTS/` directory.
+
+**Tokens are raw `int`.** `Lexer::gettok()` returns negative values for the `Token`
+enum (`include/cps/Lexer.h`) and non-negative values for literal ASCII chars (`'('`,
+`'+'`, `'<'`). Parser and CodeGen switch on both forms interchangeably;
+`BinaryExprAST` stores this same int as its operator. Payloads (identifier text,
+literal values) arrive out-of-band via public `Lexer` fields (`IdentifierStr`,
+`StringVal`, `NumVal`, `RealVal`, `CharVal`) that must be read before the next
+`gettok()`.
+
+**Keywords are UPPERCASE and case-sensitive.** `declare` silently lexes as an
+identifier. Assignment is `<-` (ASCII only), equality is single `=`, inequality `<>`,
+concatenation `&` (reaches the parser as raw ASCII 38 — no token). Identifiers use
+ASCII case-insensitive lookup/comparison through `IdentifierMap`, `IdentifierSet`
+and `sameIdentifier`; retain source spelling in ASTs and diagnostics. This includes
+functions, types, enum values, fields and builtins, but not C_TYPE import strings.
+Integer tokens carry a checked unsigned magnitude through lookahead; the parser
+applies the sign and i64 range check, including the minimum value. String/char
+literals support C++17 byte escapes and Unicode escapes (UTF-8 in STRING, ASCII only
+in CHAR). Comments are `//` to end of line.
+
+Identifiers may begin with an underscore. The case-insensitive `__cps_` prefix is
+reserved in every user identifier, including EXTERN declarations.
+
+**The lexer reads a caller-owned stream.** Construct it with `Lexer(std::istream &)`;
+the stream must outlive the lexer. Lookahead, line number and error state belong to
+each instance. The driver opens the source file; stdin is not used for compilation.