test(diagnostics): require accurate causes and source lines

Check native and IR rejection without creating or replacing output files.
Keep nine missing-location failures visible rather than accepting them.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

51 files changed, 279 insertions, 0 deletions

diff --git a/tests/iii/argument-count/case.json b/tests/iii/argument-count/case.json
new file mode 100644
index 0000000..b503ec6
--- /dev/null
+++ b/tests/iii/argument-count/case.json
@@ -0,0 +1,8 @@
+{
+  "errors": [
+    {
+      "reason": "Incorrect\\ \\#\\ arguments\\ passed\\ to\\ Identity",
+      "line": 4
+    }
+  ]
+}
diff --git a/tests/iii/argument-count/notes.md b/tests/iii/argument-count/notes.md
new file mode 100644
index 0000000..eff0df3
--- /dev/null
+++ b/tests/iii/argument-count/notes.md
@@ -0,0 +1,5 @@
+# argument-count
+
+CIE pp. 23-24 function signature and call must agree.
+
+Require exit 1, the stated cause and source line on the same diagnostic, exactly one compilation termination line, and preservation of absent/existing outputs for both native and IR requests. Missing block delimiters are located at EOF (the line after the final newline).
diff --git a/tests/iii/argument-count/program.cps b/tests/iii/argument-count/program.cps
new file mode 100644
index 0000000..edc4b7e
--- /dev/null
+++ b/tests/iii/argument-count/program.cps
@@ -0,0 +1,4 @@
+FUNCTION Identity(X : INTEGER) RETURNS INTEGER
+    RETURN X
+ENDFUNCTION
+OUTPUT Identity(1, 2)
diff --git a/tests/iii/builtin-arity/case.json b/tests/iii/builtin-arity/case.json
new file mode 100644
index 0000000..3b9e9a7
--- /dev/null
+++ b/tests/iii/builtin-arity/case.json
@@ -0,0 +1,8 @@
+{
+  "errors": [
+    {
+      "reason": "LENGTH\\ expects\\ 1\\ arg",
+      "line": 2
+    }
+  ]
+}
diff --git a/tests/iii/builtin-arity/notes.md b/tests/iii/builtin-arity/notes.md
new file mode 100644
index 0000000..99cdc60
--- /dev/null
+++ b/tests/iii/builtin-arity/notes.md
@@ -0,0 +1,5 @@
+# builtin-arity
+
+CIE p. 16 LENGTH signature.
+
+Require exit 1, the stated cause and source line on the same diagnostic, exactly one compilation termination line, and preservation of absent/existing outputs for both native and IR requests. Missing block delimiters are located at EOF (the line after the final newline).
diff --git a/tests/iii/builtin-arity/program.cps b/tests/iii/builtin-arity/program.cps
new file mode 100644
index 0000000..36bf085
--- /dev/null
+++ b/tests/iii/builtin-arity/program.cps
@@ -0,0 +1,2 @@
+// LENGTH takes exactly one string.
+OUTPUT LENGTH("a", "b")
diff --git a/tests/iii/byref-literal/case.json b/tests/iii/byref-literal/case.json
new file mode 100644
index 0000000..0a98e22
--- /dev/null
+++ b/tests/iii/byref-literal/case.json
@@ -0,0 +1,8 @@
+{
+  "errors": [
+    {
+      "reason": "BYREF\\ argument\\ to\\ Change\\ must\\ be\\ a\\ variable",
+      "line": 4
+    }
+  ]
+}
diff --git a/tests/iii/byref-literal/notes.md b/tests/iii/byref-literal/notes.md
new file mode 100644
index 0000000..6d55d4d
--- /dev/null
+++ b/tests/iii/byref-literal/notes.md
@@ -0,0 +1,5 @@
+# byref-literal
+
+CIE p. 24 reference parameters require a writable caller location.
+
+Require exit 1, the stated cause and source line on the same diagnostic, exactly one compilation termination line, and preservation of absent/existing outputs for both native and IR requests. Missing block delimiters are located at EOF (the line after the final newline).
diff --git a/tests/iii/byref-literal/program.cps b/tests/iii/byref-literal/program.cps
new file mode 100644
index 0000000..400e217
--- /dev/null
+++ b/tests/iii/byref-literal/program.cps
@@ -0,0 +1,4 @@
+PROCEDURE Change(BYREF X : INTEGER)
+    X <- X + 1
+ENDPROCEDURE
+CALL Change(7)
diff --git a/tests/iii/case-order/case.json b/tests/iii/case-order/case.json
new file mode 100644
index 0000000..ca00d69
--- /dev/null
+++ b/tests/iii/case-order/case.json
@@ -0,0 +1,8 @@
+{
+  "errors": [
+    {
+      "reason": "OTHERWISE\\ must\\ be\\ last",
+      "line": 5
+    }
+  ]
+}
diff --git a/tests/iii/case-order/notes.md b/tests/iii/case-order/notes.md
new file mode 100644
index 0000000..36b71cb
--- /dev/null
+++ b/tests/iii/case-order/notes.md
@@ -0,0 +1,5 @@
+# case-order
+
+CIE p. 19 requires OTHERWISE to be the last CASE clause.
+
+Require exit 1, the stated cause and source line on the same diagnostic, exactly one compilation termination line, and preservation of absent/existing outputs for both native and IR requests. Missing block delimiters are located at EOF (the line after the final newline).
diff --git a/tests/iii/case-order/program.cps b/tests/iii/case-order/program.cps
new file mode 100644
index 0000000..dedf5cd
--- /dev/null
+++ b/tests/iii/case-order/program.cps
@@ -0,0 +1,6 @@
+DECLARE X : INTEGER
+X <- 1
+CASE OF X
+    OTHERWISE : OUTPUT 0
+    1 : OUTPUT 1
+ENDCASE
diff --git a/tests/iii/char-width/case.json b/tests/iii/char-width/case.json
new file mode 100644
index 0000000..5bf46ff
--- /dev/null
+++ b/tests/iii/char-width/case.json
@@ -0,0 +1,8 @@
+{
+  "errors": [
+    {
+      "reason": "CHAR\\ literal\\ must\\ contain\\ exactly\\ one\\ character",
+      "line": 2
+    }
+  ]
+}
diff --git a/tests/iii/char-width/notes.md b/tests/iii/char-width/notes.md
new file mode 100644
index 0000000..e8d50ba
--- /dev/null
+++ b/tests/iii/char-width/notes.md
@@ -0,0 +1,5 @@
+# char-width
+
+CIE p. 7 CHAR literal definition.
+
+Require exit 1, the stated cause and source line on the same diagnostic, exactly one compilation termination line, and preservation of absent/existing outputs for both native and IR requests. Missing block delimiters are located at EOF (the line after the final newline).
diff --git a/tests/iii/char-width/program.cps b/tests/iii/char-width/program.cps
new file mode 100644
index 0000000..f79602a
--- /dev/null
+++ b/tests/iii/char-width/program.cps
@@ -0,0 +1,2 @@
+// CHAR contains exactly one character.
+OUTPUT 'ab'
diff --git a/tests/iii/constant-expression/case.json b/tests/iii/constant-expression/case.json
new file mode 100644
index 0000000..63614df
--- /dev/null
+++ b/tests/iii/constant-expression/case.json
@@ -0,0 +1,8 @@
+{
+  "errors": [
+    {
+      "reason": "CONSTANT\\ initializer\\ must\\ be\\ a\\ literal,\\ not\\ an\\ expression",
+      "line": 2
+    }
+  ]
+}
diff --git a/tests/iii/constant-expression/notes.md b/tests/iii/constant-expression/notes.md
new file mode 100644
index 0000000..11fb0ae
--- /dev/null
+++ b/tests/iii/constant-expression/notes.md
@@ -0,0 +1,5 @@
+# constant-expression
+
+CIE p. 8 forbids expressions in constant declarations.
+
+Require exit 1, the stated cause and source line on the same diagnostic, exactly one compilation termination line, and preservation of absent/existing outputs for both native and IR requests. Missing block delimiters are located at EOF (the line after the final newline).
diff --git a/tests/iii/constant-expression/program.cps b/tests/iii/constant-expression/program.cps
new file mode 100644
index 0000000..f1c6c25
--- /dev/null
+++ b/tests/iii/constant-expression/program.cps
@@ -0,0 +1,2 @@
+// Constants accept literals only.
+CONSTANT Limit = 1 + 2
diff --git a/tests/iii/constant-write/case.json b/tests/iii/constant-write/case.json
new file mode 100644
index 0000000..01fde66
--- /dev/null
+++ b/tests/iii/constant-write/case.json
@@ -0,0 +1,8 @@
+{
+  "errors": [
+    {
+      "reason": "Constant\\ Limit\\ is\\ not\\ a\\ writable\\ location",
+      "line": 2
+    }
+  ]
+}
diff --git a/tests/iii/constant-write/notes.md b/tests/iii/constant-write/notes.md
new file mode 100644
index 0000000..144568d
--- /dev/null
+++ b/tests/iii/constant-write/notes.md
@@ -0,0 +1,5 @@
+# constant-write
+
+CIE p. 8 constant semantics; assignment is not writable.
+
+Require exit 1, the stated cause and source line on the same diagnostic, exactly one compilation termination line, and preservation of absent/existing outputs for both native and IR requests. Missing block delimiters are located at EOF (the line after the final newline).
diff --git a/tests/iii/constant-write/program.cps b/tests/iii/constant-write/program.cps
new file mode 100644
index 0000000..aca2dc0
--- /dev/null
+++ b/tests/iii/constant-write/program.cps
@@ -0,0 +1,2 @@
+CONSTANT Limit = 3
+Limit <- 4
diff --git a/tests/iii/function-byref/case.json b/tests/iii/function-byref/case.json
new file mode 100644
index 0000000..e879655
--- /dev/null
+++ b/tests/iii/function-byref/case.json
@@ -0,0 +1,8 @@
+{
+  "errors": [
+    {
+      "reason": "BYREF\\ parameters\\ are\\ not\\ allowed\\ in\\ a\\ FUNCTION",
+      "line": 2
+    }
+  ]
+}
diff --git a/tests/iii/function-byref/notes.md b/tests/iii/function-byref/notes.md
new file mode 100644
index 0000000..4063bff
--- /dev/null
+++ b/tests/iii/function-byref/notes.md
@@ -0,0 +1,5 @@
+# function-byref
+
+CIE p. 24 explicitly prohibits passing function parameters by reference.
+
+Require exit 1, the stated cause and source line on the same diagnostic, exactly one compilation termination line, and preservation of absent/existing outputs for both native and IR requests. Missing block delimiters are located at EOF (the line after the final newline).
diff --git a/tests/iii/function-byref/program.cps b/tests/iii/function-byref/program.cps
new file mode 100644
index 0000000..1cc89cb
--- /dev/null
+++ b/tests/iii/function-byref/program.cps
@@ -0,0 +1,4 @@
+// BYREF is procedure-only.
+FUNCTION Bad(BYREF X : INTEGER) RETURNS INTEGER
+    RETURN X
+ENDFUNCTION
diff --git a/tests/iii/missing-colon/case.json b/tests/iii/missing-colon/case.json
new file mode 100644
index 0000000..5127698
--- /dev/null
+++ b/tests/iii/missing-colon/case.json
@@ -0,0 +1,8 @@
+{
+  "errors": [
+    {
+      "reason": "Expected\\ ':'\\ after\\ variable\\ name",
+      "line": 2
+    }
+  ]
+}
diff --git a/tests/iii/missing-colon/notes.md b/tests/iii/missing-colon/notes.md
new file mode 100644
index 0000000..39bf132
--- /dev/null
+++ b/tests/iii/missing-colon/notes.md
@@ -0,0 +1,5 @@
+# missing-colon
+
+CIE p. 8 declaration syntax.
+
+Require exit 1, the stated cause and source line on the same diagnostic, exactly one compilation termination line, and preservation of absent/existing outputs for both native and IR requests. Missing block delimiters are located at EOF (the line after the final newline).
diff --git a/tests/iii/missing-colon/program.cps b/tests/iii/missing-colon/program.cps
new file mode 100644
index 0000000..def8db0
--- /dev/null
+++ b/tests/iii/missing-colon/program.cps
@@ -0,0 +1,3 @@
+// DECLARE requires a colon.
+DECLARE Count INTEGER
+OUTPUT 1
diff --git a/tests/iii/missing-endif/case.json b/tests/iii/missing-endif/case.json
new file mode 100644
index 0000000..8eb640c
--- /dev/null
+++ b/tests/iii/missing-endif/case.json
@@ -0,0 +1,8 @@
+{
+  "errors": [
+    {
+      "reason": "expected\\ ENDIF",
+      "line": 4
+    }
+  ]
+}
diff --git a/tests/iii/missing-endif/notes.md b/tests/iii/missing-endif/notes.md
new file mode 100644
index 0000000..e37304e
--- /dev/null
+++ b/tests/iii/missing-endif/notes.md
@@ -0,0 +1,5 @@
+# missing-endif
+
+CIE p. 18 IF requires ENDIF.
+
+Require exit 1, the stated cause and source line on the same diagnostic, exactly one compilation termination line, and preservation of absent/existing outputs for both native and IR requests. Missing block delimiters are located at EOF (the line after the final newline).
diff --git a/tests/iii/missing-endif/program.cps b/tests/iii/missing-endif/program.cps
new file mode 100644
index 0000000..f5d97ac
--- /dev/null
+++ b/tests/iii/missing-endif/program.cps
@@ -0,0 +1,3 @@
+// The IF block never closes.
+IF TRUE THEN
+    OUTPUT 1
diff --git a/tests/iii/missing-parenthesis/case.json b/tests/iii/missing-parenthesis/case.json
new file mode 100644
index 0000000..3ccd080
--- /dev/null
+++ b/tests/iii/missing-parenthesis/case.json
@@ -0,0 +1,8 @@
+{
+  "errors": [
+    {
+      "reason": "expected\\ '\\)'",
+      "line": 3
+    }
+  ]
+}
diff --git a/tests/iii/missing-parenthesis/notes.md b/tests/iii/missing-parenthesis/notes.md
new file mode 100644
index 0000000..21265cc
--- /dev/null
+++ b/tests/iii/missing-parenthesis/notes.md
@@ -0,0 +1,5 @@
+# missing-parenthesis
+
+Unbalanced expression, expected closing delimiter at EOF.
+
+Require exit 1, the stated cause and source line on the same diagnostic, exactly one compilation termination line, and preservation of absent/existing outputs for both native and IR requests. Missing block delimiters are located at EOF (the line after the final newline).
diff --git a/tests/iii/missing-parenthesis/program.cps b/tests/iii/missing-parenthesis/program.cps
new file mode 100644
index 0000000..0cbe2c2
--- /dev/null
+++ b/tests/iii/missing-parenthesis/program.cps
@@ -0,0 +1,2 @@
+// The closing parenthesis is missing.
+OUTPUT (1 + 2
diff --git a/tests/iii/missing-then/case.json b/tests/iii/missing-then/case.json
new file mode 100644
index 0000000..67e69e7
--- /dev/null
+++ b/tests/iii/missing-then/case.json
@@ -0,0 +1,8 @@
+{
+  "errors": [
+    {
+      "reason": "expected\\ THEN",
+      "line": 2
+    }
+  ]
+}
diff --git a/tests/iii/missing-then/notes.md b/tests/iii/missing-then/notes.md
new file mode 100644
index 0000000..c04bebd
--- /dev/null
+++ b/tests/iii/missing-then/notes.md
@@ -0,0 +1,5 @@
+# missing-then
+
+CIE p. 18 IF requires THEN.
+
+Require exit 1, the stated cause and source line on the same diagnostic, exactly one compilation termination line, and preservation of absent/existing outputs for both native and IR requests. Missing block delimiters are located at EOF (the line after the final newline).
diff --git a/tests/iii/missing-then/program.cps b/tests/iii/missing-then/program.cps
new file mode 100644
index 0000000..e61e2e0
--- /dev/null
+++ b/tests/iii/missing-then/program.cps
@@ -0,0 +1,3 @@
+// THEN is required on the IF header.
+IF TRUE OUTPUT 1
+ENDIF
diff --git a/tests/iii/multiple-errors/case.json b/tests/iii/multiple-errors/case.json
new file mode 100644
index 0000000..32ec325
--- /dev/null
+++ b/tests/iii/multiple-errors/case.json
@@ -0,0 +1,12 @@
+{
+  "errors": [
+    {
+      "reason": "Unterminated\\ string\\ literal",
+      "line": 1
+    },
+    {
+      "reason": "Expected\\ ':'\\ after\\ variable\\ name",
+      "line": 2
+    }
+  ]
+}
diff --git a/tests/iii/multiple-errors/notes.md b/tests/iii/multiple-errors/notes.md
new file mode 100644
index 0000000..257f882
--- /dev/null
+++ b/tests/iii/multiple-errors/notes.md
@@ -0,0 +1,5 @@
+# multiple-errors
+
+Lexer and parser must both diagnose independent errors and recover to subsequent statements without hanging or producing output.
+
+Require exit 1, the stated cause and source line on the same diagnostic, exactly one compilation termination line, and preservation of absent/existing outputs for both native and IR requests. Missing block delimiters are located at EOF (the line after the final newline).
diff --git a/tests/iii/multiple-errors/program.cps b/tests/iii/multiple-errors/program.cps
new file mode 100644
index 0000000..ed85208
--- /dev/null
+++ b/tests/iii/multiple-errors/program.cps
@@ -0,0 +1,3 @@
+OUTPUT "unterminated
+DECLARE Number INTEGER
+OUTPUT "after errors"
diff --git a/tests/iii/nominal-type-mismatch/case.json b/tests/iii/nominal-type-mismatch/case.json
new file mode 100644
index 0000000..19b147a
--- /dev/null
+++ b/tests/iii/nominal-type-mismatch/case.json
@@ -0,0 +1,8 @@
+{
+  "errors": [
+    {
+      "reason": "Cannot\\ use\\ a\\ value\\ of\\ type\\ First\\ where\\ Second\\ is\\ expected",
+      "line": 7
+    }
+  ]
+}
diff --git a/tests/iii/nominal-type-mismatch/notes.md b/tests/iii/nominal-type-mismatch/notes.md
new file mode 100644
index 0000000..858a1c7
--- /dev/null
+++ b/tests/iii/nominal-type-mismatch/notes.md
@@ -0,0 +1,5 @@
+# nominal-type-mismatch
+
+Project nominal user-type contract: distinct pointer declarations remain incompatible.
+
+Require exit 1, the stated cause and source line on the same diagnostic, exactly one compilation termination line, and preservation of absent/existing outputs for both native and IR requests. Missing block delimiters are located at EOF (the line after the final newline).
diff --git a/tests/iii/nominal-type-mismatch/program.cps b/tests/iii/nominal-type-mismatch/program.cps
new file mode 100644
index 0000000..ecad322
--- /dev/null
+++ b/tests/iii/nominal-type-mismatch/program.cps
@@ -0,0 +1,7 @@
+TYPE First = ^INTEGER
+TYPE Second = ^INTEGER
+DECLARE Value : INTEGER
+DECLARE A : First
+DECLARE B : Second
+A <- ^Value
+B <- A
diff --git a/tests/iii/string-type-mismatch/case.json b/tests/iii/string-type-mismatch/case.json
new file mode 100644
index 0000000..199909e
--- /dev/null
+++ b/tests/iii/string-type-mismatch/case.json
@@ -0,0 +1,8 @@
+{
+  "errors": [
+    {
+      "reason": "STRING\\ comparison\\ requires\\ two\\ STRING\\ operands",
+      "line": 2
+    }
+  ]
+}
diff --git a/tests/iii/string-type-mismatch/notes.md b/tests/iii/string-type-mismatch/notes.md
new file mode 100644
index 0000000..7ffaecb
--- /dev/null
+++ b/tests/iii/string-type-mismatch/notes.md
@@ -0,0 +1,5 @@
+# string-type-mismatch
+
+Project strict STRING comparison contract.
+
+Require exit 1, the stated cause and source line on the same diagnostic, exactly one compilation termination line, and preservation of absent/existing outputs for both native and IR requests. Missing block delimiters are located at EOF (the line after the final newline).
diff --git a/tests/iii/string-type-mismatch/program.cps b/tests/iii/string-type-mismatch/program.cps
new file mode 100644
index 0000000..9595f38
--- /dev/null
+++ b/tests/iii/string-type-mismatch/program.cps
@@ -0,0 +1,2 @@
+// Mixed STRING/numeric comparison is not defined.
+OUTPUT "a" = 1
diff --git a/tests/iii/unknown-name/case.json b/tests/iii/unknown-name/case.json
new file mode 100644
index 0000000..e90b979
--- /dev/null
+++ b/tests/iii/unknown-name/case.json
@@ -0,0 +1,8 @@
+{
+  "errors": [
+    {
+      "reason": "Unknown\\ variable\\ name\\ Missing",
+      "line": 2
+    }
+  ]
+}
diff --git a/tests/iii/unknown-name/notes.md b/tests/iii/unknown-name/notes.md
new file mode 100644
index 0000000..36449f5
--- /dev/null
+++ b/tests/iii/unknown-name/notes.md
@@ -0,0 +1,5 @@
+# unknown-name
+
+Project declaration requirement. The output use, not EOF, is the diagnostic location.
+
+Require exit 1, the stated cause and source line on the same diagnostic, exactly one compilation termination line, and preservation of absent/existing outputs for both native and IR requests. Missing block delimiters are located at EOF (the line after the final newline).
diff --git a/tests/iii/unknown-name/program.cps b/tests/iii/unknown-name/program.cps
new file mode 100644
index 0000000..576da1c
--- /dev/null
+++ b/tests/iii/unknown-name/program.cps
@@ -0,0 +1,2 @@
+// No declaration introduces Missing.
+OUTPUT Missing
diff --git a/tests/iii/unterminated-string/case.json b/tests/iii/unterminated-string/case.json
new file mode 100644
index 0000000..8d13200
--- /dev/null
+++ b/tests/iii/unterminated-string/case.json
@@ -0,0 +1,8 @@
+{
+  "errors": [
+    {
+      "reason": "Unterminated\\ string\\ literal",
+      "line": 2
+    }
+  ]
+}
diff --git a/tests/iii/unterminated-string/notes.md b/tests/iii/unterminated-string/notes.md
new file mode 100644
index 0000000..5d97371
--- /dev/null
+++ b/tests/iii/unterminated-string/notes.md
@@ -0,0 +1,5 @@
+# unterminated-string
+
+Invalid string literal; CIE p. 7 and project lexer contract.
+
+Require exit 1, the stated cause and source line on the same diagnostic, exactly one compilation termination line, and preservation of absent/existing outputs for both native and IR requests. Missing block delimiters are located at EOF (the line after the final newline).
diff --git a/tests/iii/unterminated-string/program.cps b/tests/iii/unterminated-string/program.cps
new file mode 100644
index 0000000..867d6c0
--- /dev/null
+++ b/tests/iii/unterminated-string/program.cps
@@ -0,0 +1,3 @@
+// A newline cannot terminate a string.
+OUTPUT "broken
+OUTPUT "after"