Skip to content

Commit fadefcc

Browse files
test: frozen replay-compat corpus with expected verdicts (#1417)
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
1 parent e8b779c commit fadefcc

59 files changed

Lines changed: 1458 additions & 0 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎docs/agents/testing.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ The mapping it encodes, for when you need to run a gate directly or reason about
1919
| CLI help/guidance (`src/cli/parser/cli-help.ts`, `src/cli-schema/`) | `pnpm exec vitest run src/cli/parser/__tests__ src/cli-schema/command-schema-guards.test.ts scripts/__tests__` — the `scripts/__tests__` gates enforce help-topic benchmark coverage and pin the bench's quoted CLI samples to the real renderers |
2020
| Help benchmark cases (`scripts/help-conformance-*.mjs`) | `pnpm exec vitest run scripts/__tests__` (deterministic gates); model-backed: `pnpm bench:help-conformance` (paid LLM calls, local only) |
2121
| SkillGym prompts/assertions | `pnpm test:skillgym:case <case-id>` (broad: `pnpm test:skillgym`, filter with `-- --tag fixture-smoke` or `-- --tag skill-guidance`) — agentic routing + local-help-consumption proof only; command-planning knowledge checks belong in the help bench |
22+
| `.ad` grammar (`src/replay/script.ts`, gesture arity, replay vars) | `pnpm exec vitest run --project unit-core test/replay-compat` — the frozen replay-compat corpus asserts which released script surfaces still parse; a flipped verdict is edited in `test/replay-compat/manifest.ts`, never in the script |
2223
| Anything in `src/`, `test/`, `skills/` | `pnpm format` |
2324

2425
Two traps worth naming:

‎scripts/check-affected/model.test.ts‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,6 +114,12 @@ test('a non-.ts fixture under an owned root fails open (format alone is not owne
114114
assert.equal(result.failOpenReasons[0]?.rule, 'ambiguous-path');
115115
});
116116

117+
test('a frozen replay-compat corpus script selects the unit lane that asserts it', () => {
118+
const result = plan(['test/replay-compat/scripts/examples/gesture-lab.v0.16.8.ad']);
119+
assert.equal(result.failOpen, false);
120+
assert.ok(result.checks.includes('unit'));
121+
});
122+
117123
test('skills guidance change selects format + skillgym, not docs-only', () => {
118124
const result = plan(['skills/agent-device/SKILL.md']);
119125
assert.equal(result.failOpen, false);

‎scripts/check-affected/model.ts‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -228,6 +228,21 @@ const nodeIntegrationOwnership: OwnershipRule = ({ file }) =>
228228
? [reason('integration-node', file, 'node-integration', 'node --test integration smoke')]
229229
: [];
230230

231+
// The frozen replay-compat corpus (#1417) is `.ad` fixture data whose only
232+
// consumer is `test/replay-compat/corpus.test.ts` in the unit lane, so a corpus
233+
// addition has a derivable owner instead of failing open on its extension.
234+
const replayCompatOwnership: OwnershipRule = ({ file }) =>
235+
file.startsWith('test/replay-compat/') && file.endsWith('.ad')
236+
? [
237+
reason(
238+
'unit',
239+
file,
240+
'own:replay-compat',
241+
'frozen replay-compat corpus is asserted by the unit-lane corpus test',
242+
),
243+
]
244+
: [];
245+
231246
// SkillGym validates skill guidance (`skills/`) and owns its harness
232247
// (`test/skillgym/`); the Testing Matrix in docs/agents/testing.md routes
233248
// skill-prompt/assertion changes here.
@@ -294,6 +309,7 @@ const OWNERSHIP_RULES: readonly OwnershipRule[] = [
294309
srcProdGate,
295310
vitestRelatedOwnership,
296311
nodeIntegrationOwnership,
312+
replayCompatOwnership,
297313
skillgymOwnership,
298314
buildOwnership,
299315
];

‎test/replay-compat/README.md‎

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
# Replay-compat corpus
2+
3+
`.ad` scripts as **released** versions of agent-device wrote them, each paired with the verdict
4+
today's parser owes it: `parses`, or `fails` with an error code and the migration hint substring.
5+
6+
`corpus.test.ts` runs in `unit-core` and asserts every verdict, so a grammar change cannot quietly
7+
alter which historical script surface still loads. #1417.
8+
9+
## The freeze rule
10+
11+
**Never edit a script under `scripts/` to make a parser change pass.**
12+
13+
A corpus script is a recording that already exists in the wild; editing it deletes the evidence the
14+
gate is here to keep. When a grammar change flips a verdict:
15+
16+
1. Leave the script byte-identical.
17+
2. Update its `verdict` in `manifest.ts` in the same PR.
18+
3. Say in the PR description which historical surface broke, and why the new refusal is the intended
19+
migration rather than a regression.
20+
21+
A verdict that changes from `parses` to `fails` is a compatibility break by definition. That is
22+
allowed — the corpus only insists it be deliberate, reviewed, and paired with a migration hint good
23+
enough to put in front of whoever recorded the script.
24+
25+
## What is in the corpus
26+
27+
- `scripts/integration/` and `scripts/examples/` — mined from the git history of
28+
`test/integration/replays` and `examples/test-app/replays`: one entry per distinct file content at
29+
a release tag.
30+
- `scripts/docs/` — surfaces the released grammar wrote that those two suites never exercised
31+
(`${VAR}` parameterization, reserved/ordering `env` refusals, `wait stable` and landmark waits,
32+
`target-v1` annotations, retired `gesture rotate` velocity and `gesture swipe` durationMs).
33+
34+
Provenance lives in `manifest.ts`: `recordedBy` is the released tag whose grammar produced the
35+
surface, and `source` is where it was mined from at that tag.
36+
37+
**Released surfaces only.** A grammar state that only ever existed between commits is not compat
38+
surface — `git tag --contains <commit>` decides, and unreleased shapes stay out (AGENTS.md,
39+
"Unreleased API surface dies free"). The corpus is baselined against tags up to v0.20.0.
40+
41+
## Adding an entry
42+
43+
1. Find the surface at a release tag (`git show <tag>:<path>`, or the released grammar/docs that
44+
emitted it) and copy it verbatim into `scripts/<area>/<name>.<tag>.ad`.
45+
2. Add a `manifest.ts` entry with `recordedBy`, `source`, `covers`, and the observed verdict.
46+
3. Run `pnpm exec vitest run --project unit-core test/replay-compat`.
47+
48+
Add an entry when a change retires, renames, or narrows a `.ad` form — the corpus is the record of
49+
what that costs someone with a saved recording.

‎test/replay-compat/corpus.test.ts‎

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
import { describe, expect, test } from 'vitest';
2+
import { readdirSync, readFileSync } from 'node:fs';
3+
import { join } from 'node:path';
4+
import { fileURLToPath } from 'node:url';
5+
import { AppError } from '../../src/kernel/errors.ts';
6+
import { parseReplayScriptDetailed, readReplayScriptMetadata } from '../../src/replay/script.ts';
7+
import {
8+
REPLAY_COMPAT_CORPUS,
9+
type ReplayCompatCoverage,
10+
type ReplayCompatEntry,
11+
} from './manifest.ts';
12+
13+
const CORPUS_DIR = fileURLToPath(new URL('.', import.meta.url));
14+
15+
/**
16+
* Every historical surface the corpus exists to hold still. A coverage tag with
17+
* no entry means the corpus stopped covering a compat surface #1417 named.
18+
*/
19+
const REQUIRED_COVERAGE: ReplayCompatCoverage[] = [
20+
'context-header',
21+
'env-vars',
22+
'quoting',
23+
'retired-gesture',
24+
'target-annotation',
25+
'wait-landmark',
26+
];
27+
28+
function readCorpusScript(entry: ReplayCompatEntry): string {
29+
return readFileSync(join(CORPUS_DIR, entry.file), 'utf8');
30+
}
31+
32+
/** The whole parse surface a `.ad` file meets before replay executes anything. */
33+
function parseCorpusScript(script: string): void {
34+
readReplayScriptMetadata(script);
35+
parseReplayScriptDetailed(script);
36+
}
37+
38+
function listCorpusScriptFiles(): string[] {
39+
const areas = readdirSync(join(CORPUS_DIR, 'scripts'), { withFileTypes: true });
40+
return areas.flatMap((area) =>
41+
readdirSync(join(CORPUS_DIR, 'scripts', area.name))
42+
.filter((name) => name.endsWith('.ad'))
43+
.map((name) => `scripts/${area.name}/${name}`),
44+
);
45+
}
46+
47+
describe('replay-compat corpus', () => {
48+
test.each(REPLAY_COMPAT_CORPUS.map((entry) => [entry.id, entry] as const))('%s', (_id, entry) => {
49+
const script = readCorpusScript(entry);
50+
if (entry.verdict.kind === 'parses') {
51+
expect(() => {
52+
parseCorpusScript(script);
53+
}).not.toThrow();
54+
return;
55+
}
56+
let thrown: unknown;
57+
try {
58+
parseCorpusScript(script);
59+
} catch (error) {
60+
thrown = error;
61+
}
62+
expect(thrown, `expected ${entry.id} to fail parsing`).toBeInstanceOf(AppError);
63+
const error = thrown as AppError;
64+
expect(error.code).toBe(entry.verdict.code);
65+
expect(error.message).toContain(entry.verdict.hint);
66+
});
67+
68+
test('every corpus script is claimed by exactly one manifest entry', () => {
69+
const claimed = REPLAY_COMPAT_CORPUS.map((entry) => entry.file);
70+
expect([...claimed].sort()).toEqual(listCorpusScriptFiles().sort());
71+
expect(new Set(claimed).size).toBe(claimed.length);
72+
});
73+
74+
test('entry ids are unique and name the producing released version', () => {
75+
const ids = REPLAY_COMPAT_CORPUS.map((entry) => entry.id);
76+
expect(new Set(ids).size).toBe(ids.length);
77+
for (const entry of REPLAY_COMPAT_CORPUS) {
78+
expect(entry.recordedBy, entry.id).toMatch(/^v\d+\.\d+\.\d+$/);
79+
expect(entry.source, entry.id).toContain(entry.recordedBy);
80+
expect(entry.file, entry.id).toContain(`.${entry.recordedBy}.ad`);
81+
}
82+
});
83+
84+
test('the corpus covers every compat surface it is required to freeze', () => {
85+
const covered = new Set(REPLAY_COMPAT_CORPUS.flatMap((entry) => entry.covers));
86+
expect([...covered].sort()).toEqual([...REQUIRED_COVERAGE].sort());
87+
});
88+
});

0 commit comments

Comments
 (0)