ci(compiler): gate platform packages on reproducible compiler tests

Run all three suites on Linux, macOS and Windows with the push SHA as
seed. Retain failure evidence and document the 12 existing failures,
pending human review and remaining coverage gaps.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

3 files changed, 24 insertions, 6 deletions

diff --git a/AGENTS/ARCH.md b/AGENTS/ARCH.md
index ffde252..04edd24 100644
--- a/AGENTS/ARCH.md
+++ b/AGENTS/ARCH.md
@@ -245,8 +245,10 @@ Windows system directories, excluding MSYS2 and LLVM. The macOS check also
 verifies arm64 and rejects non-system dynamic libraries. The plugin uses
 Node.js 22, `npm ci --ignore-scripts` and
 `npm run package`, then checks VSIX contents. Each job uploads directly after
-its own checks pass; failure in one does not block the others. No Actions
-artifacts are uploaded.
+its own checks pass; failure in one does not block the others. Compiler jobs enable
+BUILD_TESTING, install Perl and run all CTest cases with the push SHA as seed before
+their package smoke checks and uploads. Failed tests retain inputs and logs as
+Actions artifacts for 14 days; test summaries appear even on failure. See TESTING.md.
 Cleanup and uploads check main's current SHA to avoid modifying nightly for a
 superseded push. The workflow uses `GITHUB_TOKEN` with `contents: write`.
 Compiler package names retain the existing commit-based version; the VSIX uses
diff --git a/AGENTS/PITFALLS.md b/AGENTS/PITFALLS.md
index 58934ce..16fa16d 100644
--- a/AGENTS/PITFALLS.md
+++ b/AGENTS/PITFALLS.md
@@ -108,7 +108,8 @@ random files (OPENFILE...FOR RANDOM/SEEK/GETRECORD/PUTRECORD — TYPE is done, s
 are unblocked now; the RANDOM keyword is reserved and gives a dedicated parse error),
 DATE, INPUT into array elements, string escape sequences,
 array-typed record fields, whole-array RETURN, class methods/constructors/destructors,
-NEW and class inheritance. There is no dedicated compiler test suite; nightly CI
-smoke-tests the packaged EMBEDDED compiler and runs the existing plugin tests.
-Verify compiler changes by compiling sample programs end-to-end (command
-sequence in `ARCH.md`).
+NEW and class inheritance. Compiler fixtures now live in `tests/` (see TESTING.md).
+Nightly CI runs the compiler suites, packaged EMBEDDED smoke checks and existing
+plugin tests. The initial compiler suite intentionally exposes existing standard
+conflicts and missing diagnostic locations; see `tests/FAILURES.md` before interpreting
+a failing baseline. Keep those failures visible until their implementations are fixed.
diff --git a/AGENTS/TESTING.md b/AGENTS/TESTING.md
new file mode 100644
index 0000000..7403aa1
--- /dev/null
+++ b/AGENTS/TESTING.md
@@ -0,0 +1,15 @@
+# Compiler testing
+
+Compiler tests live in `tests/`; run instructions and findings are in its README and FAILURES files. `BUILD_TESTING` defaults to OFF and adds a Perl dependency only when enabled. Python is already a compiler build dependency. CMake builds independent C++ reference executables with the same host ABI and registers one CTest entry per `case.json`, plus harness and generator checks.
+
+Each case directory has `program.cps` and `case.json`. Suite i adds `expected.txt`, optional `input.txt`, input files under `files/`, expected text-file contents under `expected-files/`, and independently written trace notes with human-review status. Suite ii adds `reference.cc` and `generate.pl`; the generator accepts seed and random count, emits the total record count followed by one ASCII record per line, and each program emits one result line per record. Suite iii stores an `errors` list of `{reason: regex, line: integer}` in its manifest. A cause and its line must appear on the same error diagnostic. EOF after a final newline is the following source line.
+
+Optional manifest fields are `exit_code` for expected program status (default 0) and `timeout` for execution/reference seconds (default 30; file-intensive cases use 180). Compilation has a 120-second timeout. Do not make a known compiler defect pass by changing the expectation, adding an exemption or comparing only a permissive substring of program output.
+
+The Python runner accepts `--case`, `--cpsc`, `--run-mode`, `--perl`, `--reference`, `--seed`, `--count` and `--results`. CMake provides paths, including executable suffixes, and caches `CPSC_TEST_SEED` and `CPSC_TEST_COUNT`. The default seed is HEAD read at execution, not configure time. CI overrides it with the checked-out push SHA. Both native and enabled runner paths are tested independently; NONE skips only runner execution.
+
+All commands run without a shell and retain their exact argv, working directory, input, status and output under the case result directory. Each execution path has its own files. Existing results for that case are replaced on re-run. Avoid concurrent runs of the same case into the same results directory. Tests compare CRLF-normalized bytes without stripping whitespace. Runtime diagnostics on stdout are part of fixed expectations; successful compilation and ordinary programs require empty stderr.
+
+Perl generators use Random.pm's SHA-256 sequence with explicit big-endian extraction, fixed group identity and decimal counter. `generator-vectors.json` pins 100 records plus boundaries using the all-zero 40-character seed; update a vector only when deliberately changing a generator, and verify repeatability and a different seed. C++ references must avoid undefined behavior and implement the specification independently. Use arithmetic closed forms or standard containers where they provide an independent oracle. Record platform-dependent C behavior as such; never call it CIE behavior.
+
+Before committing a test change, build the affected references and run the affected CTest cases. Compiler defects may fail: inspect retained diagnostics and document the failure accurately. Before committing workflow changes, run actionlint and retain the existing release API simulation/package smoke checks without contacting the real release. No compiler implementation changes belong in a tests-only task.