Skip to content

Commit 5a72c28

Browse files
committed
fix(ci): credit gates by execution shape, and keep every guard
Three ways the manifest could report a gate as owned when it does not run. 1. Crediting was a substring scan over `run:`, which #1429 explicitly rules out — "do not infer reachability from a command name merely appearing in workflow text". `false && pnpm gate x`, a gate inside `if false; then … fi`, one named in a heredoc, and `echo pnpm gate x` all credited it. There is a live instance: conformance-regenerate.yml's "Fail if regeneration changed anything" step names `pnpm gate maestro-regenerate` inside an error message telling a human to run it, and that credited the gate. A gate now counts only as the first command segment of a line, and a body carrying shell structure earns nothing. Reachability inside a script is not decidable, so this does not try: unrecognised shape means no credit and the check reports unowned. `VAR=$(pnpm gate x …)` is read, since the assignment form is unambiguous and the gate runs. 2. Job-level `if:` was not modelled at all, though six live jobs carry one, so a job that cannot run still credited every gate inside it. Two conditions on the mutation lanes are now declared. 3. A caller's `if:` REPLACED the guard on a nested composite-action step (`guard[0] ?? step.condition`), so an outer `always()` erased an inner `if: false`. Steps carry every guard between the lane and the step. Also corrects two source comments that still claimed project code run outside the runner fails the manifest. It does not: such a step earns no credit. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ
1 parent e26f7ae commit 5a72c28

5 files changed

Lines changed: 194 additions & 19 deletions

File tree

‎scripts/check-affected/model.ts‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,8 @@ import { WIRE_SURFACE_FILES } from '../../test/wire-compat/surface.ts';
2626
// the command-reference docs gate, which used to be reachable only as a workflow
2727
// job nothing in this repo could name. `pnpm gate <id>` is the only way a CI lane
2828
// may invoke one, so a gate that is not listed here cannot silently start (or stop)
29-
// running: `check:gate-manifest` fails on project code run outside the runner.
29+
// running: `check:gate-manifest` fails when a registered check is run by no lane. Work a
30+
// lane performs outside the runner earns no credit rather than failing in its own right.
3031
export type CheckId =
3132
| 'format'
3233
| 'lint'

‎scripts/gate/audit.test.ts‎

Lines changed: 91 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -17,7 +17,7 @@ import test from 'node:test';
1717
import { audit, formatFailures } from './audit.ts';
1818
import { GATE_CONDITIONS } from './declarations.ts';
1919
import { loadModel, type Model } from './model.ts';
20-
import { creditsUnder, loadLanes } from './workflows.ts';
20+
import { creditsUnder, gateIdsIn, loadLanes } from './workflows.ts';
2121

2222
const repoRoot = path.resolve(import.meta.dirname, '../..');
2323
const tracked = execFileSync('git', ['ls-files'], { cwd: repoRoot, encoding: 'utf8' })
@@ -242,3 +242,93 @@ test('planted failures are reported under the assertion that owns them', () => {
242242
);
243243
assert.deepEqual(kinds(model), ['condition']);
244244
});
245+
246+
// --- Crediting is by execution shape, not by text appearing in a body -----------------
247+
//
248+
// #1429: "Do not infer reachability from a command name merely appearing in workflow text."
249+
// A substring scan did exactly that. Each row is a body that NAMES a gate without running
250+
// it, and must therefore leave the check unowned.
251+
252+
const DEAD_TEXT: [string, string][] = [
253+
['right operand of &&', 'false && pnpm gate layering'],
254+
['right operand of ||', 'true || pnpm gate layering'],
255+
['argument to echo', 'echo pnpm gate layering'],
256+
['quoted in a message', 'echo "run pnpm gate layering to fix this"'],
257+
['inside a false branch', 'if false; then\n pnpm gate layering\nfi'],
258+
['inside a heredoc', "cat <<'EOF'\npnpm gate layering\nEOF"],
259+
['after a shell comment', 'echo hi\n# pnpm gate layering'],
260+
['not the first command', 'echo hi; pnpm gate layering'],
261+
];
262+
263+
for (const [name, run] of DEAD_TEXT) {
264+
test(`a gate ${name} earns no credit`, () => {
265+
assert.deepEqual(gateIdsIn(run), [], `"${run.replace(/\n/g, '\\n')}" must not credit`);
266+
});
267+
}
268+
269+
const LIVE: [string, string, string[]][] = [
270+
['a bare invocation', 'pnpm gate layering', ['layering']],
271+
['one per line', 'pnpm gate build\npnpm gate package', ['build', 'package']],
272+
['with arguments', 'pnpm gate fallow --base "$FALLOW_BASE"', ['fallow']],
273+
['after another command', 'pnpm clean:daemon\npnpm gate unit', ['unit']],
274+
['behind an env prefix', 'CI=1 pnpm gate lint', ['lint']],
275+
['with a pnpm flag', 'pnpm --silent gate lint', ['lint']],
276+
[
277+
'captured from a substitution',
278+
'modules=$(pnpm --silent gate mutation-affected --list-affected | tail -n1)',
279+
['mutation-affected'],
280+
],
281+
['carrying a GitHub expression', 'pnpm gate coverage --base "${{ github.sha }}"', ['coverage']],
282+
];
283+
284+
for (const [name, run, expected] of LIVE) {
285+
test(`a gate ${name} is credited`, () => {
286+
assert.deepEqual(gateIdsIn(run), expected);
287+
});
288+
}
289+
290+
test('a job-level `if:` guards every gate the job reaches', () => {
291+
// Six live jobs carry one, and it was not modelled at all: a job that cannot run still
292+
// credited every gate inside it.
293+
const model = plantWorkflow(`name: Planted
294+
on:
295+
pull_request:
296+
jobs:
297+
planted:
298+
if: github.actor == 'nobody'
299+
steps:
300+
- name: Planted
301+
run: pnpm gate layering`);
302+
assert.ok(
303+
found(model).some((m) => /github\.actor == 'nobody'/.test(m)),
304+
'the job-level condition must reach the sighting',
305+
);
306+
});
307+
308+
test("a caller's crediting `if:` cannot erase an inner `if: false`", () => {
309+
// `guard[0] ?? step.condition` REPLACED the nested condition, so an outer `always()` over
310+
// an inner `if: false` credited a gate that cannot execute. Every guard has to hold.
311+
const model = plantTree({
312+
'.github/workflows/planted.yml': `name: Planted
313+
on:
314+
pull_request:
315+
jobs:
316+
planted:
317+
steps:
318+
- uses: ./.github/actions/inner-false
319+
if: always()`,
320+
'.github/actions/inner-false/action.yml': `name: 'Inner'
321+
description: 'a gate its own step disables'
322+
runs:
323+
using: composite
324+
steps:
325+
- shell: bash
326+
if: false
327+
run: pnpm gate layering`,
328+
});
329+
const messages = found(model);
330+
assert.ok(
331+
messages.some((m) => /if: false/.test(m)),
332+
`the inner 'if: false' must survive the outer always(): ${messages.join(' | ')}`,
333+
);
334+
});

‎scripts/gate/declarations.ts‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -73,6 +73,21 @@ export const GATE_CONDITIONS: Readonly<Record<string, GateCondition>> = {
7373
'run it, and this declaration does not evaluate that value. Ownership therefore rests ' +
7474
'on the three lanes that pass `true`, and would need revisiting if they stopped.',
7575
},
76+
"needs.select.outputs.modules != '[]'": {
77+
credits: true,
78+
reason:
79+
'the affected-mutants job, which runs only when the selector found kernel modules the ' +
80+
'PR touches. An empty selection means there is nothing for this gate to mutate, so ' +
81+
'skipping is equivalent to running it over an empty set — the same claim as the ' +
82+
'content-keyed caches above, not a claim that the job always executes.',
83+
},
84+
"always() && needs.select.result == 'success'": {
85+
credits: true,
86+
reason:
87+
'the affected ratchet, which runs whenever selection succeeded regardless of whether ' +
88+
'the mutant shards passed — that is the point of a ratchet, and `always()` here only ' +
89+
'stops a red shard from cancelling the verdict.',
90+
},
7691
'failure()': {
7792
credits: false,
7893
reason:

‎scripts/gate/run.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,9 @@
22
//
33
// Every gating step in .github/workflows goes through here, which is what makes
44
// the workflow→check mapping a token scan instead of a shell interpreter, and
5-
// what makes an unregistered gate impossible to wire up: `check:gate-manifest`
6-
// fails on project code run in a qualifying lane outside this entry point.
5+
// the single spelling `check:gate-manifest` reads to decide what a lane runs. It does not
6+
// prevent a lane from running project code some other way — such a step simply earns no
7+
// ownership credit, so the check it would have satisfied reports unowned.
78
//
89
// Extra arguments are forwarded verbatim, so lane-specific flags (`--base`,
910
// `--udid`, `--report-junit`) stay in the workflow where they belong.

‎scripts/gate/workflows.ts‎

Lines changed: 83 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -12,7 +12,7 @@ import path from 'node:path';
1212
import { parse } from 'yaml';
1313
import type { CheckId } from '../check-affected/model.ts';
1414
import { GATE_ACTIONS, GATE_CONDITIONS } from './declarations.ts';
15-
import { commandSegments } from './shell.ts';
15+
import { ENV_PREFIX, commandSegments, stripExpressions } from './shell.ts';
1616

1717
/** A `pnpm gate <id>` a lane reaches, and every `if:` standing between the lane and it. */
1818
export type GateSighting = {
@@ -46,13 +46,72 @@ export type Lane = {
4646
readonly unsupported: readonly string[];
4747
};
4848

49-
/** One shell block a lane reaches, with the `if:` that decides whether it runs. */
49+
/**
50+
* One shell block a lane reaches, with EVERY `if:` standing between the lane and it.
51+
*
52+
* A list rather than one condition: an earlier revision stored the caller's guard with
53+
* `guard[0] ?? step.condition`, so a crediting outer condition (`always()`) REPLACED an
54+
* inner `if: false` and credited a gate that cannot execute. Every guard has to hold, so
55+
* every guard has to be kept.
56+
*/
5057
export type LaneStep = {
5158
readonly run: string;
52-
readonly condition: string | null;
59+
readonly conditions: readonly string[];
5360
};
5461

55-
const GATE_INVOCATION = /(?:^|[\s;&|(])pnpm\s+(?:--\S+\s+)*gate\s+([a-z0-9:-]+)/g;
62+
/** A whole command segment that is a gate invocation, with or without arguments. */
63+
const GATE_SEGMENT = /^pnpm(?:\s+--\S+)*\s+gate\s+([a-z0-9:-]+)(?:\s.*)?$/;
64+
65+
/**
66+
* `modules=$(pnpm --silent gate mutation-affected --list-affected …)` — a gate whose OUTPUT
67+
* the lane captures. It runs exactly as much as a bare invocation does, and the assignment
68+
* form is unambiguous, so it is read rather than left to fail as unowned.
69+
*/
70+
const GATE_CAPTURE = /^[A-Za-z_][A-Za-z0-9_]*=\$\((.*)$/;
71+
72+
/**
73+
* Shell that decides whether later commands run, or that quotes text as data. A body
74+
* containing any of it earns NO gate credit at all.
75+
*
76+
* #1429 requires that reachability is not inferred "from a command name merely appearing in
77+
* workflow text", and a substring scan does exactly that: `false && pnpm gate x`, a gate
78+
* inside `if false; then … fi`, and a gate named in a heredoc or an `echo` all credited a
79+
* gate that never runs. This tree has a live instance — conformance-regenerate.yml's
80+
* "Fail if regeneration changed anything" step names `pnpm gate maestro-regenerate` inside
81+
* an error message telling a human to run it.
82+
*
83+
* Deciding reachability inside a shell script is not decidable, so this does not try:
84+
* anything with structure is simply not credited, and the check reports unowned until a
85+
* human wires it as a plain step. Over-conservative in the safe direction.
86+
*/
87+
const SHELL_STRUCTURE =
88+
/(?:^|[\s;&|(])(?:if|then|elif|else|fi|case|esac|while|until|for|do|done|function|eval|exec|source|trap|exit)(?:\s|$)|<</m;
89+
90+
/**
91+
* The gate ids a `run:` body actually invokes.
92+
*
93+
* A gate counts only as the FIRST command segment of a line, so it is the command the line
94+
* runs rather than the right operand of a `&&`/`||` whose left side decides it, and never
95+
* an argument to something else (`echo pnpm gate lint`).
96+
*/
97+
export function gateIdsIn(run: string): CheckId[] {
98+
const body = stripExpressions(run);
99+
if (SHELL_STRUCTURE.test(body)) return [];
100+
return body
101+
.replace(/\\\n/g, ' ')
102+
.split('\n')
103+
.flatMap((line) => {
104+
const first = commandSegments(line)[0];
105+
if (first === undefined) return [];
106+
const bare = first.replace(ENV_PREFIX, '');
107+
// The closing `)` may be absent: segmentation splits `$(… | tail -n1)` at the pipe,
108+
// which is fine — the gate is the command being substituted either way.
109+
const captured = GATE_CAPTURE.exec(bare);
110+
const inner = captured ? (captured[1] as string).replace(/\)\s*$/, '').trim() : bare;
111+
const match = GATE_SEGMENT.exec(inner);
112+
return match ? [match[1] as CheckId] : [];
113+
});
114+
}
56115

57116
type RawStep = {
58117
name?: string;
@@ -168,6 +227,8 @@ type Job = {
168227
name?: string;
169228
steps?: RawStep[];
170229
uses?: string;
230+
/** Guards every step in the job. Six live jobs carry one; none were modelled before. */
231+
if?: unknown;
171232
};
172233

173234
type WorkflowDoc = {
@@ -199,11 +260,11 @@ function usedActionSurface(step: RawStep, root: string, chain: readonly string[]
199260
// A conditional `uses:` guards everything the action reaches.
200261
const guard = step.if === undefined ? [] : [String(step.if)];
201262
return {
263+
// The caller's `if:` applies to every step the action reaches, ON TOP OF whatever
264+
// guards that step already carries.
202265
steps: nested.steps.map((nestedStep) => ({
203266
...nestedStep,
204-
// The caller's `if:` applies to every step the action reaches. Only one condition is
205-
// carried per step because `creditsUnder` is applied to each in turn by `laneGates`.
206-
condition: guard[0] ?? nestedStep.condition,
267+
conditions: [...guard, ...nestedStep.conditions],
207268
})),
208269
gates: [...(gate ? [gate] : []), ...nested.gates].map((sighting) => ({
209270
...sighting,
@@ -217,7 +278,7 @@ function stepSurface(step: RawStep, root: string, chain: readonly string[]): Sur
217278
const used = usedActionSurface(step, root, chain);
218279
const own: LaneStep[] =
219280
typeof step.run === 'string'
220-
? [{ run: step.run, condition: step.if === undefined ? null : String(step.if) }]
281+
? [{ run: step.run, conditions: step.if === undefined ? [] : [String(step.if)] }]
221282
: [];
222283
return { steps: [...own, ...used.steps], gates: used.gates };
223284
}
@@ -236,13 +297,11 @@ function laneInvocations(
236297
): { gates: GateSighting[]; verbatim: string[] } {
237298
const gates: GateSighting[] = [];
238299
const verbatim = new Set<string>();
239-
for (const { run, condition } of steps) {
240-
const conditions = condition === null ? [] : [condition];
241-
for (const match of run.matchAll(GATE_INVOCATION))
242-
gates.push({ id: match[1] as CheckId, conditions });
300+
for (const { run, conditions } of steps) {
301+
for (const id of gateIdsIn(run)) gates.push({ id, conditions });
243302
// Verbatim credit is ownership too, and is filtered the same way: a lane that inlines
244303
// a script's body behind an undeclared `if:` has not shown that it runs it.
245-
if (creditsUnder(condition))
304+
if (conditions.every(creditsUnder))
246305
for (const name of verbatimScripts(run, scripts)) verbatim.add(name);
247306
}
248307
return { gates, verbatim: [...verbatim] };
@@ -271,9 +330,18 @@ function workflowLanes(
271330
const qualifying = 'pull_request' in on || 'schedule' in on;
272331
const { paths, pathsIgnore } = triggerPaths(on);
273332
return Object.entries(doc.jobs ?? {}).map(([jobId, job]) => {
333+
// A job-level `if:` decides whether ANY of its steps run, so it guards every sighting
334+
// the job produces — including gates reached through a composite action.
335+
const jobGuard = job.if === undefined ? [] : [String(job.if)];
274336
const surface = jobSurface(job, root);
275-
const scanned = laneInvocations(surface.steps, scripts);
276-
const sightings = [...scanned.gates, ...surface.gates];
337+
const scanned = laneInvocations(
338+
surface.steps.map((step) => ({ ...step, conditions: [...jobGuard, ...step.conditions] })),
339+
scripts,
340+
);
341+
const sightings = [...scanned.gates, ...surface.gates].map((sighting) => ({
342+
...sighting,
343+
conditions: [...jobGuard, ...sighting.conditions],
344+
}));
277345
return {
278346
workflow: file,
279347
label: laneLabel(doc.name ?? file, job.name ?? jobId),

0 commit comments

Comments
 (0)