Skip to content

Commit c859a11

Browse files
committed
fix(ci): a file's non-gate steps are counted, so --update cannot bless one
Rounds 8, 9 and 10 each asked for per-edge ownership of the generated baseline, and I answered each time by explaining why the baseline is generated rather than by testing the claim. The claim was right, and the defect behind it is concrete: `REASONS` was keyed per declaring file with nothing checking arity, so a NEW step in an already-described file inherited that file's reason, and a NEW file had no reason at all. Both reproduce. Adding run: node -e 'import("./scripts/layering/check.ts")' to ios.yml, running `--update`, then auditing printed `ok — 47 checks wired`. So did a whole new workflow running the same thing. The branch already had three live instances: `build-docs/action.yml` (1 step) and `setup-apple-runner-build/action.yml` (4 steps) were in the baseline with no description at all, and `macos.yml` still carried a reason after round 7 moved its build step into the Apple runner action. Each entry now pins how many non-gate steps its file is allowed. The count is hand-written, which is the point: `--update` regenerates digests but cannot raise the number that admits them, so an unclassified step fails closed with a message naming the file, both counts, and which direction moved. Undescribed files fail as `census`, dead descriptions as `inert`. Deliberately arity rather than per-entry prose: the same review asked for the census to be compact data instead of 685 lines of declarations, and per-edge reasons for 90 entries is that file again. This is the cheapest thing that makes an unclassified step fail closed. Both attacks re-run after the fix and now fail with the count mismatch. Three checked-in regressions, all of which regenerate the baseline first so they test what survives `--update` rather than what the shipped baseline happens to reject. 73 gate tests. Manifest, lint, typecheck, format, layering, depgraph, production-exports, fallow and trusted-fixture-artifact green locally.
1 parent ede5b60 commit c859a11

4 files changed

Lines changed: 191 additions & 31 deletions

File tree

‎scripts/gate/audit.test.ts‎

Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -434,6 +434,47 @@ test('every declared gate condition that credits is load-bearing', () => {
434434
}
435435
});
436436

437+
// Rounds 8, 9 and 10 each asked for per-edge ownership of the generated baseline. The
438+
// concrete defect behind that objection — never stated in the reviews, found by reproducing
439+
// it — is that reasons were keyed per FILE with no arity, so `--update` blessed both a new
440+
// step in an already-described file and an entirely new undescribed file. Both printed `ok`.
441+
const regenerate = (model: Model) => census(model.lanes, (step) => plainGateStep(step) !== null);
442+
443+
test('r8-10: `--update` cannot bless a new step in an already-described file', () => {
444+
const added = plantStep(`node -e 'import("./scripts/layering/check.ts")'`);
445+
const found = audit(added, regenerate(added));
446+
assert.deepEqual(
447+
found.filter((f) => f.assertion === 'bypass'),
448+
[],
449+
'regeneration silences the bypass — which is why the count has to be hand-written',
450+
);
451+
const census_ = found.filter((f) => f.assertion === 'census');
452+
assert.equal(census_.length, 1);
453+
assert.match(census_[0]?.message ?? '', /ci\.yml declares 7 non-gate step\(s\).*has 8/s);
454+
});
455+
456+
test('r8-10: `--update` cannot bless a file nobody has described', () => {
457+
const planted = plantWorkflow(
458+
workflow(` - name: Do something\n run: node -e 'import("./x.ts")'\n`),
459+
);
460+
const found = audit(planted, regenerate(planted)).filter((f) => f.assertion === 'census');
461+
assert.equal(found.length, 1);
462+
assert.match(found[0]?.message ?? '', /planted\.yml has 1 step\(s\).*no REASONS entry/s);
463+
});
464+
465+
test('r8-10: a REASONS entry describing no live step is inert', () => {
466+
// The other direction, and it was live on this branch: `macos.yml` still carried a reason
467+
// after round 7 moved its build step into the Apple runner action.
468+
const without: Model = {
469+
...base,
470+
lanes: base.lanes.filter((lane) => lane.workflow !== 'size.yml'),
471+
};
472+
const found = audit(without, regenerate(without)).filter(
473+
(f) => f.assertion === 'inert' && /REASONS entry "size\.yml"/.test(f.message),
474+
);
475+
assert.equal(found.length, 1, 'a reason describing nothing must be deleted');
476+
});
477+
437478
test('a baseline entry describing no live step is inert', () => {
438479
const invented = [
439480
...baseline,

‎scripts/gate/audit.ts‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ import {
1313
EXTERNAL_ACTIONS,
1414
GATE_ACTIONS,
1515
GATE_CONDITIONS,
16+
REASONS,
1617
UNPROVABLE_OWNERS,
1718
type ExternalAction,
1819
} from './declarations.ts';
@@ -30,6 +31,7 @@ const HEADINGS: Readonly<Record<string, string>> = {
3031
external: 'Third-party actions with no declaration',
3132
surface: 'Execution surfaces the manifest does not model',
3233
'path-coverage': 'Paths whose selected checks no triggered lane runs',
34+
census: 'Files running shell outside the runner that nobody has described',
3335
condition: 'Gate steps behind an `if:` nobody has ruled on',
3436
inert: 'Declarations that no longer apply',
3537
registered: 'Suites and projects no registered check covers',
@@ -390,6 +392,50 @@ function inertOpaque(model: Model): Failure[] {
390392
});
391393
}
392394

395+
// 4b. Every file in the baseline is DESCRIBED, and described with the right arity.
396+
//
397+
// Rounds 8, 9 and 10 all asked for per-edge ownership of the baseline, and the objection
398+
// was right: with reasons keyed per file and nothing checking how many steps each file
399+
// was allowed, `--update` blessed a new step in an already-described file — and an
400+
// entirely new file with no description at all. Both reproduce on this branch: adding
401+
// `run: node -e 'import("./scripts/layering/check.ts")'` to ios.yml, then regenerating,
402+
// printed `ok`.
403+
//
404+
// The count is hand-written, which is the whole point — `--update` regenerates the
405+
// digests but cannot raise the number that admits them.
406+
function censusFiles(declared: readonly BaselineEntry[]): Failure[] {
407+
const counted = new Map<string, number>();
408+
for (const entry of declared) counted.set(entry.source, (counted.get(entry.source) ?? 0) + 1);
409+
const undeclared = [...counted.entries()]
410+
.filter(([source]) => !(source in REASONS))
411+
.map(([source, count]) =>
412+
fail(
413+
'census',
414+
`${source} has ${count} step(s) in the baseline but no REASONS entry: nothing says why ` +
415+
`this file runs shell outside the runner. Add one with \`steps: ${count}\`.`,
416+
),
417+
);
418+
const miscounted = [...counted.entries()].flatMap(([source, count]) => {
419+
const declaredFile = REASONS[source];
420+
if (declaredFile === undefined || declaredFile.steps === count) return [];
421+
const verb = count > declaredFile.steps ? 'added' : 'removed';
422+
return [
423+
fail(
424+
'census',
425+
`${source} declares ${declaredFile.steps} non-gate step(s) and the baseline has ` +
426+
`${count}: a step was ${verb}. Confirm it belongs outside \`pnpm gate\`, update the ` +
427+
`reason if it no longer describes the file, and set \`steps: ${count}\`.`,
428+
),
429+
];
430+
});
431+
const dead = Object.keys(REASONS)
432+
.filter((source) => !counted.has(source))
433+
.map((source) =>
434+
fail('inert', `REASONS entry "${source}" describes no step any qualifying lane reaches.`),
435+
);
436+
return [...undeclared, ...miscounted, ...dead];
437+
}
438+
393439
function seenConditions(model: Model): Set<string> {
394440
return new Set(
395441
model.lanes
@@ -498,6 +544,7 @@ export function audit(
498544
...laneSurfaces(model),
499545
...externalActions(model, externals),
500546
...pathCoverage(model),
547+
...censusFiles(declared),
501548
...inert(model, declared),
502549
...unregisteredSuites(model),
503550
...orphanProjects(model),

‎scripts/gate/baseline.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -32,7 +32,7 @@ export function loadBaseline(file = FILE): BaselineEntry[] {
3232

3333
/** Why the steps in a given file run outside the runner, for a reviewer reading the diff. */
3434
export function reasonFor(source: string): string {
35-
return REASONS[source] ?? '(no reason recorded — add one to REASONS)';
35+
return REASONS[source]?.reason ?? '(no reason recorded — add one to REASONS)';
3636
}
3737

3838
const key = (entry: BaselineEntry) => JSON.stringify([entry.source, entry.digest]);

‎scripts/gate/declarations.ts‎

Lines changed: 102 additions & 30 deletions
Original file line numberDiff line numberDiff line change
@@ -33,42 +33,114 @@ export const OPAQUE_RUNNERS: Readonly<Record<string, readonly string[]>> = {
3333
};
3434

3535
/**
36-
* Why each file runs shell outside `pnpm gate`, once per declaring file.
36+
* Why each file runs shell outside `pnpm gate`, and HOW MANY such steps it is allowed.
3737
*
3838
* The step identities and digests live in the generated `baseline.json`; this is the part a
3939
* human writes. One reason per file rather than per step: "what is this workflow doing
4040
* outside the runner" is the question a reviewer actually asks, and answering it 109 times
4141
* produced prose nobody read.
42+
*
43+
* `steps` is what stops that from being an unowned census. Review rounds 8–10 asked three
44+
* times for per-edge ownership, and the objection was right for a reason none of us stated
45+
* concretely: because the reason was keyed per FILE and nothing checked arity, `--update`
46+
* would bless a brand-new step in an already-described file, and an entirely new file with no
47+
* description at all. Both were reproduced on this branch —
48+
* `run: node -e 'import("./scripts/layering/check.ts")'` added to `ios.yml`, regenerated, and
49+
* the audit still printed `ok`.
50+
*
51+
* Pinning the count is what makes `--update` unable to launder an addition: the number lives
52+
* here, in a hand-written file, so a new step fails until a human raises it and re-reads the
53+
* reason it is being admitted under. It fails in both directions — a deleted step drops the
54+
* count, and a file with no live steps is an inert declaration.
55+
*
56+
* It is deliberately NOT per-entry prose. The same review asked for the census to be compact
57+
* data rather than 685 lines of declarations, and per-edge reasons for 90 entries is that
58+
* file again. Arity is the cheapest thing that makes an unclassified step fail closed.
4259
*/
43-
export const REASONS: Readonly<Record<string, string>> = {
44-
'.github/actions/boot-ios-test-simulator/action.yml': 'boots and settles the iOS simulator',
45-
'.github/actions/setup-android-replay-host/action.yml':
46-
'Android SDK, KVM and emulator-host setup',
47-
'.github/actions/setup-fixture-app/action.yml':
48-
'fixture-app cache lookup, download and staging for the device lanes',
49-
'.github/actions/setup-node-pnpm/action.yml':
50-
'toolchain setup: pins pnpm to packageManager and installs dependencies',
51-
'.github/actions/setup-test-app-dependencies/action.yml': 'Expo test-app dependency setup',
52-
'android.yml': 'Android fixture APK restore, staged without installing',
53-
'ci.yml':
54-
'macOS runner build for the Swift unit-test surface, gated and carrying the unit-test opt-in',
55-
'conformance-differential.yml': 'iOS runner build, simulator boot and fixture app',
56-
'conformance-regenerate.yml': 'regeneration diff assertion and the fixture-seal verification',
57-
'ios.yml': 'iOS runner build, simulator boot and fixture app',
58-
'linux.yml': 'Linux desktop session setup (Xvfb, D-Bus, AT-SPI) and the replay smoke',
59-
'macos.yml': 'macOS runner build for the replay lane',
60-
'mutation-affected.yml':
61-
'shard-matrix derivation and the failure-path envelope recorders (#1430)',
62-
'mutation-weekly.yml': 'shard-matrix derivation and the failure-path envelope recorders (#1430)',
63-
'perf-nightly.yml': 'iOS runner build and simulator boot for the benchmark lane',
64-
'pr-preview-cleanup.yml':
65-
'website preview teardown: the same call with the same values, from the cleanup lane',
66-
'pr-preview.yml':
67-
'website preview deploy: the values pr-preview-action interpolates into its own `run:` blocks',
68-
'replays-nightly.yml': 'the nightly Android replay sweep, run inside the emulator action',
69-
'size.yml':
70-
'bundle-size measurement, which reports rather than gates, and runs at the PR base commit',
71-
'test-app-build-cache.yml': 'Expo release app build and artifact staging',
60+
export type NonGateFile = { readonly steps: number; readonly reason: string };
61+
62+
export const REASONS: Readonly<Record<string, NonGateFile>> = {
63+
'.github/actions/boot-ios-test-simulator/action.yml': {
64+
steps: 1,
65+
reason: 'boots and settles the iOS simulator',
66+
},
67+
'.github/actions/build-docs/action.yml': {
68+
steps: 1,
69+
reason: 'builds the docs site the website preview lane publishes',
70+
},
71+
'.github/actions/setup-android-replay-host/action.yml': {
72+
steps: 6,
73+
reason: 'Android SDK, KVM and emulator-host setup',
74+
},
75+
'.github/actions/setup-apple-runner-build/action.yml': {
76+
steps: 4,
77+
reason:
78+
'Xcode and source-hash cache identity, plus the build itself — which IS `pnpm gate ' +
79+
'"$INPUT_GATE"`, and is listed here rather than shape-approved only because its ' +
80+
'`if:` makes it conditional (see GATE_CONDITIONS)',
81+
},
82+
'.github/actions/setup-fixture-app/action.yml': {
83+
steps: 9,
84+
reason: 'fixture-app cache lookup, download and staging for the device lanes',
85+
},
86+
'.github/actions/setup-node-pnpm/action.yml': {
87+
steps: 3,
88+
reason: 'toolchain setup: pins pnpm to packageManager and installs dependencies',
89+
},
90+
'.github/actions/setup-test-app-dependencies/action.yml': {
91+
steps: 2,
92+
reason: 'Expo test-app dependency setup',
93+
},
94+
'android.yml': { steps: 3, reason: 'Android fixture APK restore, staged without installing' },
95+
'ci.yml': {
96+
steps: 7,
97+
reason:
98+
'macOS runner build for the Swift unit-test surface, gated and carrying the unit-test opt-in',
99+
},
100+
'conformance-differential.yml': {
101+
steps: 3,
102+
reason: 'iOS runner build, simulator boot and fixture app',
103+
},
104+
'conformance-regenerate.yml': {
105+
steps: 2,
106+
reason: 'regeneration diff assertion and the fixture-seal verification',
107+
},
108+
'ios.yml': { steps: 9, reason: 'iOS runner build, simulator boot and fixture app' },
109+
'linux.yml': {
110+
steps: 5,
111+
reason: 'Linux desktop session setup (Xvfb, D-Bus, AT-SPI) and the replay smoke',
112+
},
113+
'mutation-affected.yml': {
114+
steps: 6,
115+
reason: 'shard-matrix derivation and the failure-path envelope recorders (#1430)',
116+
},
117+
'mutation-weekly.yml': {
118+
steps: 5,
119+
reason: 'shard-matrix derivation and the failure-path envelope recorders (#1430)',
120+
},
121+
'perf-nightly.yml': {
122+
steps: 3,
123+
reason: 'iOS runner build and simulator boot for the benchmark lane',
124+
},
125+
'pr-preview-cleanup.yml': {
126+
steps: 1,
127+
reason: 'website preview teardown: the same call with the same values, from the cleanup lane',
128+
},
129+
'pr-preview.yml': {
130+
steps: 1,
131+
reason:
132+
'website preview deploy: the values pr-preview-action interpolates into its own `run:` blocks',
133+
},
134+
'replays-nightly.yml': {
135+
steps: 9,
136+
reason: 'the nightly Android replay sweep, run inside the emulator action',
137+
},
138+
'size.yml': {
139+
steps: 5,
140+
reason:
141+
'bundle-size measurement, which reports rather than gates, and runs at the PR base commit',
142+
},
143+
'test-app-build-cache.yml': { steps: 5, reason: 'Expo release app build and artifact staging' },
72144
};
73145

74146
/**

0 commit comments

Comments
 (0)