From e7320359c12b97ccb85ebfb06b2540124630df7b Mon Sep 17 00:00:00 2001 From: _ov4 Date: Sun, 18 Oct 2026 08:00:31 +0000 Subject: feat(lexer): support C++ escape sequences in literals Co-Authored-By: Claude Opus 5.5 --- AGENTS/CONVENTIONS.md | 3 +-- AGENTS/PITFALLS.md | 4 +++- 2 files changed, 4 insertions(+), 3 deletions(-) (limited to 'AGENTS') diff --git a/AGENTS/CONVENTIONS.md b/AGENTS/CONVENTIONS.md index d1f018d..41b049f 100644 --- a/AGENTS/CONVENTIONS.md +++ b/AGENTS/CONVENTIONS.md @@ -22,8 +22,7 @@ ASCII case-insensitive lookup/comparison through `IdentifierMap`, `IdentifierSet and `sameIdentifier`; retain source spelling in ASTs and diagnostics. This includes functions, types, enum values, fields and builtins, but not C_TYPE import strings. Integer tokens carry a checked unsigned magnitude through lookahead; the parser -applies the sign and i64 range check, including the minimum value. No escape sequences -in string/char literals. Comments are `//` to end of line. +applies the sign and i64 range check, including the minimum value. String/char literals support C++17 byte escapes and Unicode escapes (UTF-8 in STRING, ASCII only in CHAR). Comments are `//` to end of line. **The lexer reads a caller-owned stream.** Construct it with `Lexer(std::istream &)`; the stream must outlive the lexer. Lookahead, line number and error state belong to diff --git a/AGENTS/PITFALLS.md b/AGENTS/PITFALLS.md index 4059767..820fa55 100644 --- a/AGENTS/PITFALLS.md +++ b/AGENTS/PITFALLS.md @@ -169,10 +169,12 @@ test programs. SET types (`SET`/`DEFINE` are reserved with dedicated errors), random files (OPENFILE...FOR RANDOM/SEEK/GETRECORD/PUTRECORD — TYPE is done, so these are unblocked now; the RANDOM keyword is reserved and gives a dedicated parse error), -DATE, string escape sequences, +DATE, array-typed record fields, whole-array RETURN and class inheritance (INHERITS/SUPER). Compiler fixtures now live in `tests/` (see TESTING.md). Nightly CI runs the compiler suites, packaged EMBEDDED smoke checks and existing plugin tests. The initial standard conflicts and missing diagnostic locations are fixed on Linux. WHILE accepts both the standard header without DO and the older WHILE DO form. + +- **Literal escapes.** C++17 escapes are decoded by the lexer. Unicode escapes encode UTF-8 in strings; CHAR Unicode escapes must be ASCII. Built-in STRING remains NUL-terminated: `"a\0b"` is observed as `"a"`. -- cgit v1.3.1-4-g156e