Skip to content

Commit d747ef6

Browse files
test: frozen replay-compat corpus with expected verdicts (#1417) (#1436)
* test: frozen replay-compat corpus with expected verdicts (#1417) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: pin replay-compat corpus bytes to released blobs and assert via parseReplayInput Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: lock replay-compat provenance kind by corpus area and verify it in CI Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs: describe corpus provenance-kind lock and CI job Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: prune replay-compat corpus to minimal witnesses per shipped form Reviewer feedback on #1436: the mechanism earns its place, the dataset did not. Drop the 30 corpus entries whose bytes repeat a syntactic form or a migration refusal another entry already witnesses (platform twins and adjacent-release re-recordings), leaving 22 deliberate entries; make note required and state per entry which form or refusal it is the sole witness of. Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * test: address corpus review nits (typed coverage list, cap rationale, derived-citation note) Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs: split corpus rule — form from the release, verdict from today's parser Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> * docs: format corpus README emphasis markers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --------- Co-authored-by: Michał Pierzchała <[email protected]> Co-authored-by: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
1 parent 4c02b6a commit d747ef6

35 files changed

Lines changed: 1186 additions & 1 deletion

‎.github/workflows/ci.yml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -216,6 +216,26 @@ jobs:
216216
- name: Check for production-unused exports
217217
run: pnpm check:production-exports
218218

219+
replay-compat-provenance:
220+
# The frozen replay-compat corpus (#1417) claims each entry was published by
221+
# a released tag. Only a full-history checkout can re-derive that claim, so
222+
# this job exists separately from the shallow-clone-safe unit lane.
223+
name: Replay-Compat Provenance
224+
runs-on: ubuntu-latest
225+
timeout-minutes: 10
226+
steps:
227+
- name: Checkout
228+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
229+
with:
230+
fetch-depth: 0
231+
fetch-tags: true
232+
233+
- name: Setup toolchain
234+
uses: ./.github/actions/setup-node-pnpm
235+
236+
- name: Verify corpus entries against their released blobs
237+
run: pnpm check:replay-compat
238+
219239
coverage:
220240
# Runs the full unit + provider-integration suites under coverage with
221241
# thresholds, so a separate unit-tests job would rerun the same tests.

‎docs/agents/testing.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ The mapping it encodes, for when you need to run a gate directly or reason about
1919
| CLI help/guidance (`src/cli/parser/cli-help.ts`, `src/cli-schema/`) | `pnpm exec vitest run src/cli/parser/__tests__ src/cli-schema/command-schema-guards.test.ts scripts/__tests__` — the `scripts/__tests__` gates enforce help-topic benchmark coverage and pin the bench's quoted CLI samples to the real renderers |
2020
| Help benchmark cases (`scripts/help-conformance-*.mjs`) | `pnpm exec vitest run scripts/__tests__` (deterministic gates); model-backed: `pnpm bench:help-conformance` (paid LLM calls, local only) |
2121
| SkillGym prompts/assertions | `pnpm test:skillgym:case <case-id>` (broad: `pnpm test:skillgym`, filter with `-- --tag fixture-smoke` or `-- --tag skill-guidance`) — agentic routing + local-help-consumption proof only; command-planning knowledge checks belong in the help bench |
22+
| `.ad` grammar (`src/replay/script.ts`, gesture arity, replay vars) | `pnpm exec vitest run --project unit-core test/replay-compat` — the frozen replay-compat corpus asserts which released script surfaces still parse; a flipped verdict is edited in `test/replay-compat/manifest.ts`, never in the script. Adding or re-pinning a corpus entry also runs `pnpm check:replay-compat`, which re-derives each entry from its release tag in git history |
2223
| Anything in `src/`, `test/`, `skills/` | `pnpm format` |
2324

2425
Two traps worth naming:

‎package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -122,6 +122,7 @@
122122
"depgraph:test": "node --experimental-strip-types --test scripts/depgraph/model.test.ts",
123123
"check:production-exports": "fallow dead-code --config fallow-production-exports.json --production --unused-exports --fail-on-issues",
124124
"check:bundle-owner-files": "node --experimental-strip-types scripts/check-bundle-owner-files.ts",
125+
"check:replay-compat": "node --experimental-strip-types scripts/check-replay-compat-provenance.ts",
125126
"check:freerange": "fr",
126127
"check:quick": "pnpm lint && pnpm typecheck",
127128
"sync:mcp-metadata": "node scripts/sync-mcp-metadata.mjs",

‎scripts/check-affected/checks.ts‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,15 @@ export const CHECK_CATALOG: readonly CheckSpec[] = [
152152
ciJobs: [],
153153
localRunnable: true,
154154
},
155+
{
156+
id: 'replay-compat',
157+
label: 'Replay-compat corpus provenance (released blobs)',
158+
kind: { type: 'script', script: 'check:replay-compat' },
159+
// Needs full history and tags, so it runs in its own fetch-depth: 0 job
160+
// rather than inside the shallow-clone-safe unit lane.
161+
ciJobs: ['Replay-Compat Provenance'],
162+
localRunnable: true,
163+
},
155164
];
156165

157166
export function getCheckSpec(id: CheckId): CheckSpec {

‎scripts/check-affected/model.test.ts‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,19 @@ test('a non-.ts fixture under an owned root fails open (format alone is not owne
114114
assert.equal(result.failOpenReasons[0]?.rule, 'ambiguous-path');
115115
});
116116

117+
test('a frozen replay-compat corpus script selects the unit lane and the provenance verifier', () => {
118+
const result = plan(['test/replay-compat/scripts/examples/gesture-lab.v0.16.8.ad']);
119+
assert.equal(result.failOpen, false);
120+
assert.ok(result.checks.includes('unit'));
121+
assert.ok(result.checks.includes('replay-compat'));
122+
});
123+
124+
test('a replay-compat manifest edit selects the provenance verifier', () => {
125+
const result = plan(['test/replay-compat/manifest.ts']);
126+
assert.equal(result.failOpen, false);
127+
assert.ok(result.checks.includes('replay-compat'));
128+
});
129+
117130
test('skills guidance change selects format + skillgym, not docs-only', () => {
118131
const result = plan(['skills/agent-device/SKILL.md']);
119132
assert.equal(result.failOpen, false);
@@ -175,6 +188,7 @@ test('every catalog command resolves against package scripts', () => {
175188
'build:macos-helper': 'x',
176189
'test:smoke:web': 'x',
177190
'test:skillgym': 'x',
191+
'check:replay-compat': 'x',
178192
};
179193
for (const spec of CHECK_CATALOG) {
180194
const command = resolveCommand(spec, scripts, 'origin/main');

‎scripts/check-affected/model.ts‎

Lines changed: 32 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,8 @@ export type CheckId =
4040
| 'android-helpers'
4141
| 'macos-helper'
4242
| 'web-smoke'
43-
| 'skillgym';
43+
| 'skillgym'
44+
| 'replay-compat';
4445

4546
// The complete local check universe. A fail-open plan selects all of these;
4647
// keep it in sync with the catalog in checks.ts (asserted by the self-test).
@@ -63,6 +64,7 @@ export const ALL_CHECKS: readonly CheckId[] = [
6364
'macos-helper',
6465
'web-smoke',
6566
'skillgym',
67+
'replay-compat',
6668
];
6769

6870
export type SelectionReason = {
@@ -228,6 +230,34 @@ const nodeIntegrationOwnership: OwnershipRule = ({ file }) =>
228230
? [reason('integration-node', file, 'node-integration', 'node --test integration smoke')]
229231
: [];
230232

233+
// The frozen replay-compat corpus (#1417). `.ad` fixture data would otherwise
234+
// fail open on its extension: its only consumer is the unit-lane corpus test.
235+
// Any corpus change — script or manifest — also runs the history-backed
236+
// provenance verifier, which is the only gate that can prove an entry's blob
237+
// really came from the release tag it names.
238+
const replayCompatOwnership: OwnershipRule = ({ file }) => {
239+
if (!file.startsWith('test/replay-compat/')) return [];
240+
const selections = [
241+
reason(
242+
'replay-compat',
243+
file,
244+
'own:replay-compat-provenance',
245+
'corpus provenance is re-derived from released git blobs',
246+
),
247+
];
248+
if (file.endsWith('.ad')) {
249+
selections.push(
250+
reason(
251+
'unit',
252+
file,
253+
'own:replay-compat',
254+
'frozen replay-compat corpus is asserted by the unit-lane corpus test',
255+
),
256+
);
257+
}
258+
return selections;
259+
};
260+
231261
// SkillGym validates skill guidance (`skills/`) and owns its harness
232262
// (`test/skillgym/`); the Testing Matrix in docs/agents/testing.md routes
233263
// skill-prompt/assertion changes here.
@@ -294,6 +324,7 @@ const OWNERSHIP_RULES: readonly OwnershipRule[] = [
294324
srcProdGate,
295325
vitestRelatedOwnership,
296326
nodeIntegrationOwnership,
327+
replayCompatOwnership,
297328
skillgymOwnership,
298329
buildOwnership,
299330
];

‎scripts/check-affected/run.test.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -101,6 +101,7 @@ const ALL_SCRIPTS: Record<string, string> = {
101101
'test:integration:node': 'x',
102102
'test:integration:progress:check': 'x',
103103
'test:skillgym': 'x',
104+
'check:replay-compat': 'x',
104105
};
105106

106107
const ARGS = { base: 'origin/main', head: 'HEAD', json: false, run: true };
Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
/**
2+
* Re-derives the frozen replay-compat corpus (#1417) from git history: every
3+
* mined entry must still be the exact blob a RELEASED tag published at the path
4+
* it claims, and every tag the corpus cites must be a tag that was actually cut.
5+
*
6+
* The unit gate (`test/replay-compat/corpus.test.ts`) hashes the checked-in
7+
* bytes against the same object ids, which catches a rewritten script; only this
8+
* check can say those object ids came from a release rather than from whoever
9+
* wrote the manifest line.
10+
*/
11+
12+
import fs from 'node:fs';
13+
import path from 'node:path';
14+
import {
15+
REPLAY_COMPAT_CORPUS,
16+
REPLAY_COMPAT_RELEASED_TAGS,
17+
} from '../test/replay-compat/manifest.ts';
18+
import { findProvenanceKindViolations } from '../test/replay-compat/provenance-rules.ts';
19+
import { runCmdSync } from '../src/utils/exec.ts';
20+
21+
const repoRoot = path.resolve(import.meta.dirname, '..');
22+
const corpusRoot = path.join(repoRoot, 'test', 'replay-compat');
23+
24+
function git(args: string[]): string {
25+
return runCmdSync('git', args, { cwd: repoRoot, allowFailure: true }).stdout.trim();
26+
}
27+
28+
if (git(['rev-parse', '--is-shallow-repository']) === 'true') {
29+
throw new Error(
30+
'Corpus provenance needs full history and tags. Run `git fetch --unshallow --tags` first.',
31+
);
32+
}
33+
34+
// An entry that escaped `mined` would also escape the history check below, so
35+
// the area rule is enforced here as well as in the unit lane.
36+
const failures: string[] = [...findProvenanceKindViolations(REPLAY_COMPAT_CORPUS)];
37+
38+
const existingTags = new Set(git(['tag', '--list']).split('\n').filter(Boolean));
39+
if (existingTags.size === 0) {
40+
throw new Error('No release tags found. Run `git fetch --tags` first.');
41+
}
42+
for (const tag of REPLAY_COMPAT_RELEASED_TAGS) {
43+
if (!existingTags.has(tag)) {
44+
failures.push(`${tag} is cited as a released version but is not a tag in this repository`);
45+
}
46+
}
47+
48+
for (const entry of REPLAY_COMPAT_CORPUS) {
49+
if (!REPLAY_COMPAT_RELEASED_TAGS.includes(entry.recordedBy)) {
50+
failures.push(`${entry.id}: recordedBy ${entry.recordedBy} is not in REPLAY_COMPAT_RELEASED_TAGS`);
51+
continue;
52+
}
53+
if (entry.provenance.kind !== 'mined') continue;
54+
55+
const { path: sourcePath, blob } = entry.provenance;
56+
const historical = git(['rev-parse', `${entry.recordedBy}:${sourcePath}`]);
57+
if (historical !== blob) {
58+
failures.push(
59+
`${entry.id}: ${entry.recordedBy}:${sourcePath} is ${historical || '<missing>'}, manifest pins ${blob}`,
60+
);
61+
continue;
62+
}
63+
const frozen = fs.readFileSync(path.join(corpusRoot, entry.file));
64+
const released = runCmdSync('git', ['cat-file', 'blob', blob], {
65+
cwd: repoRoot,
66+
binaryStdout: true,
67+
}).stdoutBuffer;
68+
if (!released?.equals(frozen)) {
69+
failures.push(`${entry.id}: ${entry.file} no longer matches the released blob ${blob}`);
70+
}
71+
}
72+
73+
if (failures.length > 0) {
74+
throw new Error(
75+
`Replay-compat corpus provenance is broken (see test/replay-compat/README.md):\n${failures
76+
.map((failure) => `- ${failure}`)
77+
.join('\n')}`,
78+
);
79+
}
80+
81+
const mined = REPLAY_COMPAT_CORPUS.filter((entry) => entry.provenance.kind === 'mined').length;
82+
process.stdout.write(
83+
`Verified ${mined} mined replay-compat scripts against their released blobs across ${REPLAY_COMPAT_RELEASED_TAGS.length} tags (${REPLAY_COMPAT_CORPUS.length - mined} derived entries are digest-pinned).\n`,
84+
);

‎test/replay-compat/README.md‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# Replay-compat corpus
2+
3+
`.ad` scripts as **released** versions of agent-device wrote them, each paired with the verdict
4+
today's parser owes it: `parses`, or `fails` with an error code and the migration hint substring.
5+
6+
`corpus.test.ts` runs in `unit-core` and asserts every verdict through `parseReplayInput` — the same
7+
composition `replay`/`test` use — so a grammar change cannot quietly alter which historical script
8+
surface still loads. #1417.
9+
10+
## The freeze rule
11+
12+
**Never edit a script under `scripts/` to make a parser change pass.**
13+
14+
A corpus script is a recording that already exists in the wild; editing it deletes the evidence the
15+
gate is here to keep, and the pinned identities below make an edited script fail rather than pass.
16+
When a grammar change flips a verdict:
17+
18+
1. Leave the script byte-identical.
19+
2. Update its `verdict` in `manifest.ts` in the same PR.
20+
3. Say in the PR description which historical surface broke, and why the new refusal is the intended
21+
migration rather than a regression.
22+
23+
A verdict that changes from `parses` to `fails` is a compatibility break by definition. That is
24+
allowed — the corpus only insists it be deliberate, reviewed, and paired with a migration hint good
25+
enough to put in front of whoever recorded the script.
26+
27+
## What is in the corpus
28+
29+
- `scripts/integration/` and `scripts/examples/` — mined from the git history of
30+
`test/integration/replays` and `examples/test-app/replays`.
31+
- `scripts/docs/` — surfaces the released grammar wrote that those two suites never exercised
32+
(`${VAR}` parameterization, reserved/ordering `env` refusals, `wait stable` and landmark waits,
33+
`target-v1` annotations, retired `gesture rotate` velocity and `gesture swipe` durationMs).
34+
35+
## What earns an entry
36+
37+
This is a **parser**-compatibility corpus, not a device matrix. An entry has to be the only witness
38+
of something:
39+
40+
- a shipped syntactic form no other entry's bytes contain, or
41+
- a distinct migration refusal (error code + hint) no other entry already provokes.
42+
43+
So the same flow recorded on another platform, or re-recorded at an adjacent release with only
44+
coordinates and labels changed, does **not** get an entry: it costs maintenance and adds no parser
45+
leverage. Each entry's `note` says which form or refusal it is the sole witness of; if a new entry
46+
cannot claim one, it does not belong here. Keep the corpus in the tens of entries, not a mirror of
47+
the replay-fixture tree.
48+
49+
## How "frozen" and "released" are enforced, not asserted
50+
51+
Provenance lives in `manifest.ts`. `recordedBy` is the released tag whose grammar produced the
52+
surface, and `provenance` pins the bytes:
53+
54+
- `mined` entries carry the **git object id of the historical blob**. `corpus.test.ts` hashes the
55+
checked-in bytes and must reproduce that id, so a rewritten script cannot be made green by editing
56+
the manifest to match — the id is only obtainable from the released content.
57+
- `derived` entries (`scripts/docs/`) have no historical blob to point at, so their bytes are pinned
58+
by SHA-256. Only the bytes are machine-checked: the `from` citation naming the released grammar or
59+
doc that emitted the surface is **reviewer-verified**, so a derived entry has to be reviewed against
60+
its cited source at that tag (`git show <tag>:<path>`) before it lands.
61+
62+
**Two sources, never invention.** The _form_ comes from the cited release — the script bytes are what
63+
that tag's grammar or docs emitted. The _verdict_ comes from today's parser, because a migration
64+
refusal for a retired form did not exist at the tag that emitted it (v0.16.8 wrote
65+
`gesture rotate <deg> <x> <y> <velocity>`; the hint telling you to drop the velocity is current
66+
output). So copy the form from the release and paste the code/hint from an observed run — never
67+
paraphrase either.
68+
69+
The kind is not a free choice: `provenance-rules.ts` fixes it by corpus area (`scripts/integration/`
70+
and `scripts/examples/` must stay `mined`, `scripts/docs/` is `derived`), so an edited script cannot
71+
be relabelled `derived`, re-pinned by digest, and thereby skipped by the history check. Both the unit
72+
test and the verifier enforce that rule, and a new area must declare its kind before entries can live
73+
under it.
74+
75+
`pnpm check:replay-compat` closes the other half: it re-derives every mined id from git history
76+
(`git rev-parse <recordedBy>:<path>`) and checks every cited tag against `git tag --list`, so an
77+
entry cannot claim a version that was never cut. It needs full history and tags, so it runs in its
78+
own `Replay-Compat Provenance` CI job (`fetch-depth: 0`) rather than inside the shallow-clone-safe
79+
unit lane, and any change under `test/replay-compat/` selects it in `pnpm check:affected`.
80+
81+
**Released surfaces only.** A grammar state that only ever existed between commits is not compat
82+
surface — `git tag --contains <commit>` decides, and unreleased shapes stay out (AGENTS.md,
83+
"Unreleased API surface dies free"). The corpus is baselined against tags up to v0.20.0.
84+
85+
## Adding an entry
86+
87+
1. Find the surface at a release tag (`git show <tag>:<path>`, or the released grammar/docs that
88+
emitted it) and copy it verbatim into `scripts/<area>/<name>.<tag>.ad`.
89+
2. Add a `manifest.ts` entry with `recordedBy`, `provenance`, `covers`, and a `note` naming the form
90+
or refusal it is the sole witness of. For a mined entry the blob id is `git rev-parse <tag>:<path>`;
91+
for a derived one it is `shasum -a 256 <file>`. Add the tag to `REPLAY_COMPAT_RELEASED_TAGS` if it
92+
is new.
93+
3. Take the `verdict` from what the current parser does with those bytes: start from
94+
`verdict: { kind: 'parses' }`, run the suite, and if it refuses, paste the reported code and hint
95+
substring verbatim. Do not transcribe a hint from a release, an issue, or memory — the refusal is
96+
produced by today's parser, not by the tag that emitted the form.
97+
4. Run `pnpm exec vitest run --project unit-core test/replay-compat` and `pnpm check:replay-compat`.
98+
99+
Add an entry when a change retires, renames, or narrows a `.ad` form — the corpus is the record of
100+
what that costs someone with a saved recording.

0 commit comments

Comments
 (0)