|
| 1 | +# Replay-compat corpus |
| 2 | + |
| 3 | +`.ad` scripts as **released** versions of agent-device wrote them, each paired with the verdict |
| 4 | +today's parser owes it: `parses`, or `fails` with an error code and the migration hint substring. |
| 5 | + |
| 6 | +`corpus.test.ts` runs in `unit-core` and asserts every verdict through `parseReplayInput` — the same |
| 7 | +composition `replay`/`test` use — so a grammar change cannot quietly alter which historical script |
| 8 | +surface still loads. #1417. |
| 9 | + |
| 10 | +## The freeze rule |
| 11 | + |
| 12 | +**Never edit a script under `scripts/` to make a parser change pass.** |
| 13 | + |
| 14 | +A corpus script is a recording that already exists in the wild; editing it deletes the evidence the |
| 15 | +gate is here to keep, and the pinned identities below make an edited script fail rather than pass. |
| 16 | +When a grammar change flips a verdict: |
| 17 | + |
| 18 | +1. Leave the script byte-identical. |
| 19 | +2. Update its `verdict` in `manifest.ts` in the same PR. |
| 20 | +3. Say in the PR description which historical surface broke, and why the new refusal is the intended |
| 21 | + migration rather than a regression. |
| 22 | + |
| 23 | +A verdict that changes from `parses` to `fails` is a compatibility break by definition. That is |
| 24 | +allowed — the corpus only insists it be deliberate, reviewed, and paired with a migration hint good |
| 25 | +enough to put in front of whoever recorded the script. |
| 26 | + |
| 27 | +## What is in the corpus |
| 28 | + |
| 29 | +- `scripts/integration/` and `scripts/examples/` — mined from the git history of |
| 30 | + `test/integration/replays` and `examples/test-app/replays`. |
| 31 | +- `scripts/docs/` — surfaces the released grammar wrote that those two suites never exercised |
| 32 | + (`${VAR}` parameterization, reserved/ordering `env` refusals, `wait stable` and landmark waits, |
| 33 | + `target-v1` annotations, retired `gesture rotate` velocity and `gesture swipe` durationMs). |
| 34 | + |
| 35 | +## What earns an entry |
| 36 | + |
| 37 | +This is a **parser**-compatibility corpus, not a device matrix. An entry has to be the only witness |
| 38 | +of something: |
| 39 | + |
| 40 | +- a shipped syntactic form no other entry's bytes contain, or |
| 41 | +- a distinct migration refusal (error code + hint) no other entry already provokes. |
| 42 | + |
| 43 | +So the same flow recorded on another platform, or re-recorded at an adjacent release with only |
| 44 | +coordinates and labels changed, does **not** get an entry: it costs maintenance and adds no parser |
| 45 | +leverage. Each entry's `note` says which form or refusal it is the sole witness of; if a new entry |
| 46 | +cannot claim one, it does not belong here. Keep the corpus in the tens of entries, not a mirror of |
| 47 | +the replay-fixture tree. |
| 48 | + |
| 49 | +## How "frozen" and "released" are enforced, not asserted |
| 50 | + |
| 51 | +Provenance lives in `manifest.ts`. `recordedBy` is the released tag whose grammar produced the |
| 52 | +surface, and `provenance` pins the bytes: |
| 53 | + |
| 54 | +- `mined` entries carry the **git object id of the historical blob**. `corpus.test.ts` hashes the |
| 55 | + checked-in bytes and must reproduce that id, so a rewritten script cannot be made green by editing |
| 56 | + the manifest to match — the id is only obtainable from the released content. |
| 57 | +- `derived` entries (`scripts/docs/`) have no historical blob to point at, so their bytes are pinned |
| 58 | + by SHA-256. Only the bytes are machine-checked: the `from` citation naming the released grammar or |
| 59 | + doc that emitted the surface is **reviewer-verified**, so a derived entry has to be reviewed against |
| 60 | + its cited source at that tag (`git show <tag>:<path>`) before it lands. |
| 61 | + |
| 62 | +**Two sources, never invention.** The _form_ comes from the cited release — the script bytes are what |
| 63 | +that tag's grammar or docs emitted. The _verdict_ comes from today's parser, because a migration |
| 64 | +refusal for a retired form did not exist at the tag that emitted it (v0.16.8 wrote |
| 65 | +`gesture rotate <deg> <x> <y> <velocity>`; the hint telling you to drop the velocity is current |
| 66 | +output). So copy the form from the release and paste the code/hint from an observed run — never |
| 67 | +paraphrase either. |
| 68 | + |
| 69 | +The kind is not a free choice: `provenance-rules.ts` fixes it by corpus area (`scripts/integration/` |
| 70 | +and `scripts/examples/` must stay `mined`, `scripts/docs/` is `derived`), so an edited script cannot |
| 71 | +be relabelled `derived`, re-pinned by digest, and thereby skipped by the history check. Both the unit |
| 72 | +test and the verifier enforce that rule, and a new area must declare its kind before entries can live |
| 73 | +under it. |
| 74 | + |
| 75 | +`pnpm check:replay-compat` closes the other half: it re-derives every mined id from git history |
| 76 | +(`git rev-parse <recordedBy>:<path>`) and checks every cited tag against `git tag --list`, so an |
| 77 | +entry cannot claim a version that was never cut. It needs full history and tags, so it runs in its |
| 78 | +own `Replay-Compat Provenance` CI job (`fetch-depth: 0`) rather than inside the shallow-clone-safe |
| 79 | +unit lane, and any change under `test/replay-compat/` selects it in `pnpm check:affected`. |
| 80 | + |
| 81 | +**Released surfaces only.** A grammar state that only ever existed between commits is not compat |
| 82 | +surface — `git tag --contains <commit>` decides, and unreleased shapes stay out (AGENTS.md, |
| 83 | +"Unreleased API surface dies free"). The corpus is baselined against tags up to v0.20.0. |
| 84 | + |
| 85 | +## Adding an entry |
| 86 | + |
| 87 | +1. Find the surface at a release tag (`git show <tag>:<path>`, or the released grammar/docs that |
| 88 | + emitted it) and copy it verbatim into `scripts/<area>/<name>.<tag>.ad`. |
| 89 | +2. Add a `manifest.ts` entry with `recordedBy`, `provenance`, `covers`, and a `note` naming the form |
| 90 | + or refusal it is the sole witness of. For a mined entry the blob id is `git rev-parse <tag>:<path>`; |
| 91 | + for a derived one it is `shasum -a 256 <file>`. Add the tag to `REPLAY_COMPAT_RELEASED_TAGS` if it |
| 92 | + is new. |
| 93 | +3. Take the `verdict` from what the current parser does with those bytes: start from |
| 94 | + `verdict: { kind: 'parses' }`, run the suite, and if it refuses, paste the reported code and hint |
| 95 | + substring verbatim. Do not transcribe a hint from a release, an issue, or memory — the refusal is |
| 96 | + produced by today's parser, not by the tag that emitted the form. |
| 97 | +4. Run `pnpm exec vitest run --project unit-core test/replay-compat` and `pnpm check:replay-compat`. |
| 98 | + |
| 99 | +Add an entry when a change retires, renames, or narrows a `.ad` form — the corpus is the record of |
| 100 | +what that costs someone with a saved recording. |
0 commit comments