feat(lexer): support C++ escape sequences in literals

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

18 files changed, 158 insertions, 46 deletions

diff --git a/AGENTS/CONVENTIONS.md b/AGENTS/CONVENTIONS.md
index d1f018d..41b049f 100644
--- a/AGENTS/CONVENTIONS.md
+++ b/AGENTS/CONVENTIONS.md
@@ -22,8 +22,7 @@ ASCII case-insensitive lookup/comparison through `IdentifierMap`, `IdentifierSet
 and `sameIdentifier`; retain source spelling in ASTs and diagnostics. This includes
 functions, types, enum values, fields and builtins, but not C_TYPE import strings.
 Integer tokens carry a checked unsigned magnitude through lookahead; the parser
-applies the sign and i64 range check, including the minimum value. No escape sequences
-in string/char literals. Comments are `//` to end of line.
+applies the sign and i64 range check, including the minimum value. String/char literals support C++17 byte escapes and Unicode escapes (UTF-8 in STRING, ASCII only in CHAR). Comments are `//` to end of line.
 
 **The lexer reads a caller-owned stream.** Construct it with `Lexer(std::istream &)`;
 the stream must outlive the lexer. Lookahead, line number and error state belong to
diff --git a/AGENTS/PITFALLS.md b/AGENTS/PITFALLS.md
index 4059767..820fa55 100644
--- a/AGENTS/PITFALLS.md
+++ b/AGENTS/PITFALLS.md
@@ -169,10 +169,12 @@ test programs.
 SET types (`SET`/`DEFINE` are reserved with dedicated errors),
 random files (OPENFILE...FOR RANDOM/SEEK/GETRECORD/PUTRECORD — TYPE is done, so these
 are unblocked now; the RANDOM keyword is reserved and gives a dedicated parse error),
-DATE, string escape sequences,
+DATE,
 array-typed record fields, whole-array RETURN
 and class inheritance (INHERITS/SUPER). Compiler fixtures now live in `tests/` (see TESTING.md).
 Nightly CI runs the compiler suites, packaged EMBEDDED smoke checks and existing
 plugin tests. The initial standard conflicts and missing diagnostic locations are fixed on
 Linux.
 WHILE accepts both the standard header without DO and the older WHILE DO form.
+
+- **Literal escapes.** C++17 escapes are decoded by the lexer. Unicode escapes encode UTF-8 in strings; CHAR Unicode escapes must be ASCII. Built-in STRING remains NUL-terminated: `"a\0b"` is observed as `"a"`.
diff --git a/doc/language/cpsc-language-overview.7.md b/doc/language/cpsc-language-overview.7.md
index a66ce69..ff3d1f0 100644
--- a/doc/language/cpsc-language-overview.7.md
+++ b/doc/language/cpsc-language-overview.7.md
@@ -16,7 +16,7 @@ Precedence, from highest to lowest: unary operators, `* / DIV MOD`, `+ - &`, com
 
 Parenthesized type casts, such as (C_INT)Number, have unary precedence. C_INT, C_CHAR, C_STRING and C_DOUBLE are native C aliases; TYPE name = C_TYPE("...") imports other supported C base types and pointers. Conversions between C and pseudocode types must be explicit. These are compiler extensions; see [cpsc-keyword-c_type(7)](keyword/cpsc-keyword-c_type.7.md). EXTERN "C" declares C functions from objects named with INCLUDE_OBJ; see [cpsc-keyword-extern(7)](keyword/cpsc-keyword-extern.7.md).
 
-Literals include integers, decimal real numbers, TRUE/FALSE, strings in double quotes, and single-byte characters in single quotes. Negative numbers use unary minus. There are no string or character escape sequences. Comments begin with `//` and continue to the end of the line.
+Literals include integers, decimal real numbers, TRUE/FALSE, strings in double quotes, and single-byte characters in single quotes. Negative numbers use unary minus. String and character literals support C++17 simple escapes (`\' \" \? \\ \a \b \f \n \r \t \v`), one to three octal digits, and `\x` followed by hexadecimal digits (byte values only). Unicode escapes `\uXXXX` and `\UXXXXXXXX` encode valid Unicode scalar values as UTF-8 in strings; in CHAR they must be ASCII. Unknown escapes are errors. Built-in STRING is NUL-terminated, so `"a\0b"` is observed as `"a"`. Comments begin with `//` and continue to the end of the line.
 
 Declare variables before use. See DECLARE for built-in types and arrays, TYPE for records/enumerations/pointers, and CLASS for public/private fields and methods.
 
@@ -81,7 +81,7 @@ Closing and auxiliary keywords are documented with their main constructs: THEN/E
 
 # UNSUPPORTED FEATURES
 
-DATE, SET types and DEFINE, random-file operations (RANDOM/SEEK/GETRECORD/PUTRECORD), INT, RAND, string escapes, whole-array assignment or return, array fields, and class inheritance (INHERITS/SUPER).
+DATE, SET types and DEFINE, random-file operations (RANDOM/SEEK/GETRECORD/PUTRECORD), INT, RAND, whole-array assignment or return, array fields, and class inheritance (INHERITS/SUPER).
 
 The individual pages note relevant differences from the Cambridge 2026 pseudocode guide. This manual describes the compiler's current support.
 
diff --git a/include/cps/Lexer.h b/include/cps/Lexer.h
index 449ff8d..1c4723e 100644
--- a/include/cps/Lexer.h
+++ b/include/cps/Lexer.h
@@ -128,6 +128,7 @@ class Lexer {
         char Character;
     };
     std::deque<TokenData> Lookahead;
+    std::string readEscape(bool IsChar);
     int readToken();
     TokenData captureToken(int Kind) const;
     void restoreToken(const TokenData &Token);
diff --git a/lib/Lexer/Lexer.cc b/lib/Lexer/Lexer.cc
index cebf335..edc77de 100644
--- a/lib/Lexer/Lexer.cc
+++ b/lib/Lexer/Lexer.cc
@@ -10,6 +10,85 @@
 
 using namespace cps;
 
+std::string Lexer::readEscape(bool IsChar) {
+    auto Error = [&](const char *Reason) {
+        cps::sourceDiagnostic(TokenLine, "error: %s at line %d\n", Reason, TokenLine);
+        HadError = true;
+        return std::string("?");
+    };
+    int Escape = Input.get();
+    if (Escape == EOF || Escape == '\n' || Escape == '\r') {
+        LastChar = Escape;
+        return Error("Unterminated escape sequence");
+    }
+    LastChar = Input.get();
+    switch (Escape) {
+    case '\'': return "'";
+    case '"': return "\"";
+    case '?': return "?";
+    case '\\': return "\\";
+    case 'a': return "\a";
+    case 'b': return "\b";
+    case 'f': return "\f";
+    case 'n': return "\n";
+    case 'r': return "\r";
+    case 't': return "\t";
+    case 'v': return "\v";
+    default: break;
+    }
+    auto Hex = [](int C) -> int {
+        if (C >= '0' && C <= '9') return C - '0';
+        if (C >= 'a' && C <= 'f') return C - 'a' + 10;
+        if (C >= 'A' && C <= 'F') return C - 'A' + 10;
+        return -1;
+    };
+    uint32_t Value = 0;
+    if (Escape >= '0' && Escape <= '7') {
+        Value = Escape - '0';
+        for (int I = 1; I < 3 && LastChar >= '0' && LastChar <= '7'; ++I) {
+            Value = Value * 8 + LastChar - '0';
+            LastChar = Input.get();
+        }
+        if (Value > 255) return Error("Octal escape exceeds one byte");
+        return std::string(1, static_cast<char>(Value));
+    }
+    if (Escape == 'x') {
+        if (Hex(LastChar) < 0) return Error("Hex escape requires a digit");
+        bool Overflow = false;
+        while (Hex(LastChar) >= 0) {
+            Value = Value * 16 + Hex(LastChar);
+            if (Value > 255) { Overflow = true; Value = 0; }
+            LastChar = Input.get();
+        }
+        if (Overflow) return Error("Hex escape exceeds one byte");
+        return std::string(1, static_cast<char>(Value));
+    }
+    if (Escape != 'u' && Escape != 'U') return Error("Unknown escape sequence");
+    for (int I = 0; I < (Escape == 'u' ? 4 : 8); ++I) {
+        if (Hex(LastChar) < 0) return Error("Incomplete Unicode escape");
+        Value = Value * 16 + Hex(LastChar);
+        LastChar = Input.get();
+    }
+    if (Value > 0x10ffff || (Value >= 0xd800 && Value <= 0xdfff))
+        return Error("Invalid Unicode code point");
+    if (IsChar && Value > 127) return Error("Unicode CHAR escape must be ASCII");
+    std::string Result;
+    if (Value <= 0x7f) Result += static_cast<char>(Value);
+    else {
+        if (Value <= 0x7ff) Result += static_cast<char>(0xc0 | (Value >> 6));
+        else {
+            if (Value <= 0xffff) Result += static_cast<char>(0xe0 | (Value >> 12));
+            else {
+                Result += static_cast<char>(0xf0 | (Value >> 18));
+                Result += static_cast<char>(0x80 | ((Value >> 12) & 63));
+            }
+            Result += static_cast<char>(0x80 | ((Value >> 6) & 63));
+        }
+        Result += static_cast<char>(0x80 | (Value & 63));
+    }
+    return Result;
+}
+
 int Lexer::readToken() {
     while (isspace(LastChar)) {
         if (LastChar == '\n') CurrentLine++;
@@ -18,54 +97,31 @@ int Lexer::readToken() {
 
     TokenLine = CurrentLine;
 
-    if (LastChar == '"') {
+    if (LastChar == '"' || LastChar == '\'') {
+        const int Quote = LastChar;
+        const bool IsChar = Quote == '\'';
         StringVal.clear();
         LastChar = Input.get();
-
-        while (LastChar != '"' && LastChar != EOF && LastChar != '\n') {
-            StringVal += static_cast<char>(LastChar);
-            LastChar = Input.get();
-        }
-
-        if (LastChar == '"') {
-            LastChar = Input.get();
-        } else {
-            cps::sourceDiagnostic(CurrentLine, "error: Unterminated string literal at line %d\n", CurrentLine);
-            HadError = true;
+        while (LastChar != Quote && LastChar != EOF && LastChar != '\n') {
+            if (LastChar == '\\') StringVal += readEscape(IsChar);
+            else {
+                StringVal += static_cast<char>(LastChar);
+                LastChar = Input.get();
+            }
         }
-
-        return tok_string_literal;
-    }
-
-    if (LastChar == '\'') {
-        int CharCode = Input.get();
-        if (CharCode == EOF || CharCode == '\n') {
-            cps::sourceDiagnostic(CurrentLine, "error: Unterminated char literal at line %d\n", CurrentLine);
+        if (LastChar == Quote) LastChar = Input.get();
+        else {
+            cps::sourceDiagnostic(TokenLine, "error: Unterminated %s literal at line %d\n", IsChar ? "char" : "string", TokenLine);
             HadError = true;
-            CharVal = '?';
-            StringVal.assign(1, CharVal);
-            // Leave the newline/EOF for the main loop so line counting stays right.
-            LastChar = CharCode;
-            return tok_char_literal;
         }
-
-        int ClosingQuote = Input.get();
-        if (ClosingQuote != '\'') {
-            cps::sourceDiagnostic(CurrentLine, "error: CHAR literal must contain exactly one character (line %d)\n", CurrentLine);
-            HadError = true;
-            while (ClosingQuote != EOF && ClosingQuote != '\n' && ClosingQuote != '\'') {
-                ClosingQuote = Input.get();
+        if (IsChar) {
+            if (StringVal.size() != 1) {
+                cps::sourceDiagnostic(TokenLine, "error: CHAR literal must contain exactly one character at line %d\n", TokenLine);
+                HadError = true;
             }
-            CharVal = static_cast<char>(CharCode);
-            StringVal.assign(1, CharVal);
-            LastChar = (ClosingQuote == '\'') ? Input.get() : ClosingQuote;
-            return tok_char_literal;
+            CharVal = StringVal.size() == 1 ? StringVal[0] : '?';
         }
-
-        CharVal = static_cast<char>(CharCode);
-        StringVal.assign(1, CharVal);
-        LastChar = Input.get();
-        return tok_char_literal;
+        return IsChar ? tok_char_literal : tok_string_literal;
     }
 
     if (isalpha(LastChar)) {
diff --git a/tests/i/escape-sequences/case.json b/tests/i/escape-sequences/case.json
new file mode 100644
index 0000000..0967ef4
--- /dev/null
+++ b/tests/i/escape-sequences/case.json
@@ -0,0 +1 @@
+{}
diff --git a/tests/i/escape-sequences/expected.txt b/tests/i/escape-sequences/expected.txt
new file mode 100644
index 0000000..da16637
--- /dev/null
+++ b/tests/i/escape-sequences/expected.txt
@@ -0,0 +1,4 @@
+quote:" slash:\ question:?
+AABé😀
+10:9:255
+a
diff --git a/tests/i/escape-sequences/program.cps b/tests/i/escape-sequences/program.cps
new file mode 100644
index 0000000..a616449
--- /dev/null
+++ b/tests/i/escape-sequences/program.cps
@@ -0,0 +1,4 @@
+OUTPUT "quote:\" slash:\\ question:\?"
+OUTPUT "A\101\x42\u00e9\U0001f600"
+OUTPUT ASC('\n'), ":", ASC('\t'), ":", ASC('\377')
+OUTPUT "a\0b"
diff --git a/tests/iii/escape-hex/case.json b/tests/iii/escape-hex/case.json
new file mode 100644
index 0000000..a40e503
--- /dev/null
+++ b/tests/iii/escape-hex/case.json
@@ -0,0 +1,8 @@
+{
+  "errors": [
+    {
+      "reason": "exceeds one byte",
+      "line": 1
+    }
+  ]
+}
diff --git a/tests/iii/escape-hex/program.cps b/tests/iii/escape-hex/program.cps
new file mode 100644
index 0000000..3769c04
--- /dev/null
+++ b/tests/iii/escape-hex/program.cps
@@ -0,0 +1 @@
+OUTPUT "\x100"
diff --git a/tests/iii/escape-octal/case.json b/tests/iii/escape-octal/case.json
new file mode 100644
index 0000000..a40e503
--- /dev/null
+++ b/tests/iii/escape-octal/case.json
@@ -0,0 +1,8 @@
+{
+  "errors": [
+    {
+      "reason": "exceeds one byte",
+      "line": 1
+    }
+  ]
+}
diff --git a/tests/iii/escape-octal/program.cps b/tests/iii/escape-octal/program.cps
new file mode 100644
index 0000000..7e56ef1
--- /dev/null
+++ b/tests/iii/escape-octal/program.cps
@@ -0,0 +1 @@
+OUTPUT "\400"
diff --git a/tests/iii/escape-surrogate/case.json b/tests/iii/escape-surrogate/case.json
new file mode 100644
index 0000000..eb1882d
--- /dev/null
+++ b/tests/iii/escape-surrogate/case.json
@@ -0,0 +1,8 @@
+{
+  "errors": [
+    {
+      "reason": "Invalid Unicode",
+      "line": 1
+    }
+  ]
+}
diff --git a/tests/iii/escape-surrogate/program.cps b/tests/iii/escape-surrogate/program.cps
new file mode 100644
index 0000000..bee5004
--- /dev/null
+++ b/tests/iii/escape-surrogate/program.cps
@@ -0,0 +1 @@
+OUTPUT "\ud800"
diff --git a/tests/iii/escape-unicode-char/case.json b/tests/iii/escape-unicode-char/case.json
new file mode 100644
index 0000000..7bc1560
--- /dev/null
+++ b/tests/iii/escape-unicode-char/case.json
@@ -0,0 +1,8 @@
+{
+  "errors": [
+    {
+      "reason": "must be ASCII",
+      "line": 1
+    }
+  ]
+}
diff --git a/tests/iii/escape-unicode-char/program.cps b/tests/iii/escape-unicode-char/program.cps
new file mode 100644
index 0000000..258852a
--- /dev/null
+++ b/tests/iii/escape-unicode-char/program.cps
@@ -0,0 +1 @@
+OUTPUT '\u00e9'
diff --git a/tests/iii/escape-unknown/case.json b/tests/iii/escape-unknown/case.json
new file mode 100644
index 0000000..2ba3033
--- /dev/null
+++ b/tests/iii/escape-unknown/case.json
@@ -0,0 +1,8 @@
+{
+  "errors": [
+    {
+      "reason": "Unknown escape",
+      "line": 1
+    }
+  ]
+}
diff --git a/tests/iii/escape-unknown/program.cps b/tests/iii/escape-unknown/program.cps
new file mode 100644
index 0000000..25a4b1d
--- /dev/null
+++ b/tests/iii/escape-unknown/program.cps
@@ -0,0 +1 @@
+OUTPUT "\q"