Skip to content

Commit aec9df2

Browse files
committed
test(ci): govern gates that drive their own runner
The previous commit claimed `fuzz:parsers` was pulled into the suite universe. It was not. Membership followed what a script RUNS, but "runs" was recognised only as a Vitest project or a `node --test` file, and `fuzz:parsers` is `node scripts/fuzz/run.ts` — an `exec:` terminal, indistinguishable by shape from `size`'s. So the widening admitted five scripts, none of them the one the commit message named, and deleting the nightly-parser-fuzz job still would not have failed this gate. Two more real gates were missed the same way: `mutation:run` and `maestro:conformance:differential`. A gate that spawns its own runner cannot be recognised statically — the walk sees a script path, and `scripts/fuzz/run.ts` and `scripts/size-report.mjs` look alike. So those executables are declared (GATE_RUNNERS), the way opaque runners and argv forwarders already are, and everything else stays out. One entry governs every script that runs it, so `mutation:run`, `mutation:check`, `mutation:affected` and `mutation:baseline` come in together — the same executable under different flags. The universe grows 52 -> 58 and every new member is already owned, so this needs no new wiring and no new waivers. Declaring rather than deriving has a cost worth stating: unlike a waiver, deleting a GATE_RUNNERS entry makes this gate quieter, not louder, and a gate added tomorrow outside the convention driving its own runner stays invisible until listed. Two assertions keep the list honest — the file must exist, and the entry must ALONE admit a suite the undeclared universe lacks (weighed alone, so two entries reached by one script do not cancel each other out). The remaining pressure to derive is noted against #1429. Checked non-vacuously against the live tree, not fixtures: deleting nightly-parser-fuzz, the differential-ios job, and both mutation workflows each fails with the gate named; pointing an entry at a tracked file no script runs reports it inert. The regression pins the red side too — the fuzz script is not in the universe without its declaration. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ
1 parent d323b48 commit aec9df2

6 files changed

Lines changed: 188 additions & 7 deletions

File tree

‎scripts/gate-manifest/check.ts‎

Lines changed: 32 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,7 @@ import {
3535
DECLARED_EDGES,
3636
DOCS_LANE_OWNERS,
3737
FORWARDED_SELECTOR_RULES,
38+
GATE_RUNNERS,
3839
LOCAL_ONLY,
3940
TRANSPARENT_WRAPPERS,
4041
type DocsLaneOwner,
@@ -97,6 +98,7 @@ function context(overrides: Partial<ResolveContext> = {}): ResolveContext {
9798
expandTestPaths,
9899
transparentWrappers: new Set(TRANSPARENT_WRAPPERS.map((entry) => entry.file)),
99100
declaredTerminals: new Map(DECLARED_EDGES.map((entry) => [entry.file, entry.reaches])),
101+
gateRunners: new Set(GATE_RUNNERS.map((entry) => entry.file)),
100102
...overrides,
101103
};
102104
}
@@ -113,12 +115,17 @@ function resolvedTerminals(candidate: ResolveContext): Set<Terminal> {
113115
]);
114116
}
115117

116-
function sameTerminals(left: ReadonlySet<Terminal>, right: ReadonlySet<Terminal>): boolean {
117-
return left.size === right.size && [...left].every((terminal) => right.has(terminal));
118+
function sameSet(left: ReadonlySet<string>, right: ReadonlySet<string>): boolean {
119+
return left.size === right.size && [...left].every((member) => right.has(member));
118120
}
119121

120122
const baselineTerminals = resolvedTerminals(ctx);
121123

124+
/** The universe under `candidate`, for differencing a declaration against its absence. */
125+
function suiteIds(candidate: ResolveContext): Set<string> {
126+
return new Set(suiteUniverse(candidate).map((suite) => suite.id));
127+
}
128+
122129
/**
123130
* How much the gate would report under `candidate`. The two waiver kinds are effective in
124131
* different ways, so they need different differentials: a TRANSPARENT_WRAPPERS entry changes
@@ -191,6 +198,27 @@ const staleWaivers = [
191198
...DECLARED_EDGES.filter((entry) => !trackedSet.has(entry.file)).map(
192199
(entry) => `DECLARED_EDGES "${entry.file}" does not exist`,
193200
),
201+
...GATE_RUNNERS.filter((entry) => !trackedSet.has(entry.file)).map(
202+
(entry) => `GATE_RUNNERS "${entry.file}" does not exist`,
203+
),
204+
// A gate runner earns its place by putting a suite in the universe that would not be there
205+
// without it. Matching nothing means the script that ran it was renamed or deleted — and this
206+
// list is the only thing holding those suites in, so an entry that has quietly stopped
207+
// applying takes the gate with it.
208+
//
209+
// Each entry is weighed ALONE against the undeclared universe rather than by removing it from
210+
// the list: two entries reached by one script would otherwise each look redundant because of
211+
// the other, and both would be reported inert while both were load-bearing.
212+
...GATE_RUNNERS.filter((entry) =>
213+
sameSet(
214+
suiteIds(context({ gateRunners: new Set() })),
215+
suiteIds(context({ gateRunners: new Set([entry.file]) })),
216+
),
217+
).map(
218+
(entry) =>
219+
`GATE_RUNNERS "${entry.file}" changes nothing — no package script outside the ` +
220+
`test:/check: convention runs it, so the declaration is inert`,
221+
),
194222
...DECLARED_EDGES.flatMap((entry) =>
195223
entry.reaches
196224
.filter((terminal) => terminal.startsWith('vitest:'))
@@ -199,7 +227,7 @@ const staleWaivers = [
199227
),
200228
// Applied-reachability: drop the waiver and the resolved terminal set must move.
201229
...TRANSPARENT_WRAPPERS.filter((entry) =>
202-
sameTerminals(
230+
sameSet(
203231
baselineTerminals,
204232
resolvedTerminals(
205233
context({
@@ -458,6 +486,7 @@ console.log(
458486
`${
459487
LOCAL_ONLY.length +
460488
DECLARED_EDGES.length +
489+
GATE_RUNNERS.length +
461490
TRANSPARENT_WRAPPERS.length +
462491
FORWARDED_SELECTOR_RULES.length
463492
} owned waivers.`,

‎scripts/gate-manifest/execution-terminals.ts‎

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -63,6 +63,12 @@ export type ResolveContext = {
6363
* mapped to the terminals they are declared to reach.
6464
*/
6565
readonly declaredTerminals: ReadonlyMap<string, readonly Terminal[]>;
66+
/**
67+
* Executables that ARE a gate — they verify something and exit non-zero — while driving their
68+
* own runner rather than Vitest or `node --test`. Declared, because "does this executable
69+
* verify anything?" is not visible to a static walk. See GATE_RUNNERS in waivers.ts.
70+
*/
71+
readonly gateRunners: ReadonlySet<string>;
6672
};
6773

6874
export type Sink = {
@@ -76,6 +82,17 @@ export function report(sink: Sink, kind: UnresolvedEdge['kind'], detail: string)
7682
sink.unresolved.push({ source: sink.source, step: sink.step, kind, detail });
7783
}
7884

85+
/**
86+
* The file an `exec:` terminal runs, without its positional arguments — null for every other
87+
* kind. `exec:src/bin.ts test test/integration/replays/ios` and `exec:src/bin.ts replay …` are
88+
* different units of work but the same executable, which is the question a caller asks when it
89+
* wants to know WHAT ran rather than with which arguments.
90+
*/
91+
export function execTarget(terminal: Terminal): string | null {
92+
if (!terminal.startsWith('exec:')) return null;
93+
return terminal.slice('exec:'.length).split(' ')[0] ?? null;
94+
}
95+
7996
/** Which Vitest projects an invocation runs. */
8097
function addVitestTerminals(tokens: readonly string[], ctx: ResolveContext, sink: Sink): void {
8198
const projects: string[] = [];

‎scripts/gate-manifest/suite-ownership.test.ts‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -111,3 +111,61 @@ test('a gate named outside the test:/check: convention is still in the universe'
111111
assert.ok(ids.includes('maestro:conformance'), 'a test-running script must be governed');
112112
assert.ok(!ids.includes('build'), 'an ordinary script must stay out of the universe');
113113
});
114+
115+
// `fuzz:parsers` verbatim, because reaching a test RUNNER is not the same as being a gate: the
116+
// case above passes on a `node --test` script and says nothing about this one, which is how the
117+
// fuzz lane stayed outside the universe while the convention hole was reported closed.
118+
const FUZZ_SCRIPTS = new Map([
119+
['fuzz:parsers', 'node --experimental-strip-types scripts/fuzz/run.ts'],
120+
['size', 'node scripts/size-report.mjs'],
121+
]);
122+
123+
test('a gate driving its own runner is in the universe only once it is declared', () => {
124+
// Red: the executable is the same shape as `size`'s, so nothing in the walk separates them.
125+
const undeclared = suiteUniverse(context({ packageScripts: FUZZ_SCRIPTS, vitestProjects: [] }));
126+
assert.deepEqual(
127+
undeclared.map((suite) => suite.id),
128+
[],
129+
'an exec: terminal must not qualify as a gate on shape alone',
130+
);
131+
132+
// Green: declaring the executable admits it, and only it.
133+
const declared = suiteUniverse(
134+
context({
135+
packageScripts: FUZZ_SCRIPTS,
136+
vitestProjects: [],
137+
gateRunners: new Set(['scripts/fuzz/run.ts']),
138+
}),
139+
);
140+
assert.deepEqual(
141+
declared.map((suite) => [suite.id, [...suite.terminals]]),
142+
[['fuzz:parsers', ['exec:scripts/fuzz/run.ts']]],
143+
'the declared gate is governed; the report script beside it is not',
144+
);
145+
});
146+
147+
test('deleting the scheduled job that runs a declared gate fails the manifest', () => {
148+
// The whole point of membership: #1414 runs nightly and nowhere else, so if the universe does
149+
// not hold `fuzz:parsers`, deleting nightly-parser-fuzz drops the lane silently.
150+
const ctx = context({
151+
packageScripts: FUZZ_SCRIPTS,
152+
vitestProjects: [],
153+
gateRunners: new Set(['scripts/fuzz/run.ts']),
154+
});
155+
const nightly = 'name: Replay Nightly\non:\n schedule:\n - cron: "0 3 * * *"\njobs:\n';
156+
const fuzzJob =
157+
' nightly-parser-fuzz:\n steps:\n' +
158+
' - run: pnpm fuzz:parsers --self-check --artifact-dir .tmp/fuzz/self-check\n';
159+
const otherJob = ' nightly-ios:\n steps:\n - run: pnpm test:replay:ios\n';
160+
161+
assert.deepEqual(
162+
unownedTerminals(suiteUniverse(ctx), lanesFor({ 'n.yml': nightly + fuzzJob }, ctx), new Set()),
163+
[],
164+
'the lane owns the gate while the job exists',
165+
);
166+
assert.deepEqual(
167+
unownedTerminals(suiteUniverse(ctx), lanesFor({ 'n.yml': nightly + otherJob }, ctx), new Set()),
168+
[{ terminal: 'exec:scripts/fuzz/run.ts', suites: ['fuzz:parsers'] }],
169+
'deleting the job must name the gate that stopped running',
170+
);
171+
});

‎scripts/gate-manifest/suite-ownership.ts‎

Lines changed: 23 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
// single job runs all of it.
66

77
import {
8+
execTarget,
89
report,
910
resolveScript,
1011
type ResolveContext,
@@ -39,18 +40,23 @@ export function suiteUniverse(ctx: ResolveContext): Suite[] {
3940
// gates in real lanes under other names, and were invisible here. Membership is therefore
4041
// decided by what a script RUNS, not what it is called, so the universe cannot be dodged —
4142
// by accident or otherwise — by naming a gate something else.
42-
if (!/^(?:test|check):/.test(name) && !runsTestRunner(name, command, ctx)) continue;
43+
if (!/^(?:test|check):/.test(name) && !runsGate(name, command, ctx)) continue;
4344
suites.push(packageScriptSuite(name, command, ctx));
4445
}
4546
return suites.sort((left, right) => left.id.localeCompare(right.id));
4647
}
4748

4849
/**
49-
* Whether a script reaches a test runner. Probed WITHOUT reporting, because most scripts here are
50+
* Whether a script reaches a gate. Probed WITHOUT reporting, because most scripts here are
5051
* ordinary (`build`, `dev`) and resolve to nothing — reporting during the probe would file every
5152
* one of them as an unresolved edge.
53+
*
54+
* Two ways to qualify, because "runs a test runner" is not the same question as "is a gate".
55+
* A Vitest or `node --test` terminal proves itself. A gate that drives its OWN runner does not:
56+
* `fuzz:parsers` and `mutation:run` are `exec:` terminals indistinguishable from `build`'s by
57+
* shape, so the executables behind them are declared (GATE_RUNNERS) rather than guessed at.
5258
*/
53-
function runsTestRunner(name: string, command: string, ctx: ResolveContext): boolean {
59+
function runsGate(name: string, command: string, ctx: ResolveContext): boolean {
5460
const probe: Sink = {
5561
terminals: new Set(),
5662
unresolved: [],
@@ -59,10 +65,23 @@ function runsTestRunner(name: string, command: string, ctx: ResolveContext): boo
5965
};
6066
resolveScript(name, command, ctx, probe);
6167
return [...probe.terminals].some(
62-
(terminal) => terminal.startsWith('vitest:') || terminal.startsWith('node-test:'),
68+
(terminal) =>
69+
terminal.startsWith('vitest:') ||
70+
terminal.startsWith('node-test:') ||
71+
isDeclaredGateRunner(terminal, ctx),
6372
);
6473
}
6574

75+
/**
76+
* Whether a terminal runs a declared gate executable, matched on the executable rather than the
77+
* whole terminal so one declaration governs every script that runs it: `mutation:run`,
78+
* `mutation:check` and `mutation:affected` differ only in flags.
79+
*/
80+
function isDeclaredGateRunner(terminal: Terminal, ctx: ResolveContext): boolean {
81+
const target = execTarget(terminal);
82+
return target !== null && ctx.gateRunners.has(target);
83+
}
84+
6685
export function packageScriptSuite(name: string, command: string, ctx: ResolveContext): Suite {
6786
const sink: Sink = {
6887
terminals: new Set(),

‎scripts/gate-manifest/test-context.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ export function context(overrides: Partial<ResolveContext> = {}): ResolveContext
1616
expandTestPaths: (pattern) => [pattern],
1717
transparentWrappers: new Set(),
1818
declaredTerminals: new Map(),
19+
gateRunners: new Set(),
1920
...overrides,
2021
};
2122
}

‎scripts/gate-manifest/waivers.ts‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,63 @@ export const FORWARDED_SELECTOR_RULES: readonly ForwardedRule[] = [
7878
},
7979
];
8080

81+
/**
82+
* An executable that IS a gate — it verifies something and exits non-zero when the verification
83+
* fails — but drives its own runner instead of Vitest or `node --test`.
84+
*
85+
* The suite universe follows what a script RUNS rather than what it is called, which is only as
86+
* good as the walk's ability to recognise a gate. It recognises a Vitest project and a
87+
* `node --test` file; it cannot recognise one from an `exec:` terminal, because
88+
* `scripts/fuzz/run.ts` and `scripts/size-report.mjs` are the same shape and only one of them
89+
* can fail a build. So the gates are named here, and everything else stays out. Declared, never
90+
* guessed — the same rule TRANSPARENT_WRAPPERS follows.
91+
*
92+
* This list is the manifest's weakest link and should stay short: unlike a waiver, deleting an
93+
* entry makes the gate QUIETER rather than louder, so only check.ts's staleness assertions keep
94+
* it honest (the file must exist, and some package script must actually run it). A gate added
95+
* tomorrow under a name outside the convention, driving its own runner, is invisible until it
96+
* is listed here. The candidates are `exec:` terminals that qualifying lanes run but no suite
97+
* claims; #1429 tracks deriving that pressure instead of relying on this comment.
98+
*/
99+
export type GateRunner = {
100+
readonly file: string;
101+
readonly reason: string;
102+
readonly issue: string;
103+
};
104+
105+
export const GATE_RUNNERS: readonly GateRunner[] = [
106+
{
107+
file: 'scripts/fuzz/run.ts',
108+
reason:
109+
'The parser fuzz lane (#1414) generates hostile input in worker processes it spawns ' +
110+
'itself and asserts one invariant over the results — every rejection a typed AppError ' +
111+
'with a non-empty hint, no case hanging — then exits non-zero on a violation ' +
112+
'(scripts/fuzz/run.ts:219). No Vitest project and no `node --test` file is involved, so ' +
113+
'without this entry `fuzz:parsers` is not in the universe and deleting its nightly job ' +
114+
'would not fail this gate.',
115+
issue: '#1414',
116+
},
117+
{
118+
file: 'scripts/mutation/run.ts',
119+
reason:
120+
'The decision-kernel mutation lane (#1415) drives Stryker, which spawns the test runner ' +
121+
'itself, and ratchets the surviving-mutant score against the committed baseline — ' +
122+
'non-zero once that baseline reports `gating: true` (scripts/mutation/run.ts:553). One ' +
123+
'entry governs `mutation:run`, `mutation:baseline`, `mutation:check` and ' +
124+
'`mutation:affected`: they are the same executable under different flags.',
125+
issue: '#1415',
126+
},
127+
{
128+
file: 'packages/maestro/test/conformance/differential/run.ts',
129+
reason:
130+
'The differential conformance lane runs each scenario through both engines and fails on ' +
131+
'an UNDECLARED divergence (differential/run.ts:267). It shells out to the two engines ' +
132+
'rather than running as a test file, so the `node --test` half of the same suite ' +
133+
'(`maestro:conformance`) does not cover it.',
134+
issue: '#1429',
135+
},
136+
];
137+
81138
export const DECLARED_EDGES: readonly DeclaredEdge[] = [
82139
{
83140
file: 'scripts/lib/contention-retry-run.ts',

0 commit comments

Comments
 (0)