Skip to content

Commit e3e3edc

Browse files
committed
fix(ci): audit the commands callers hand to composite actions (#1429)
`setup-apple-runner-build` runs `${{ inputs.build-command }}`, so its own step digest is constant and vouches for nothing: the command is written at the call site, in the caller's `with:` block, which the audit only recorded as an input. Replacing `pnpm gate swift-runner-ios` with arbitrary project code left every digest unchanged and raised no bypass. The fix is not another rule. A value passed to an input the action interpolates into a `run:` block IS shell the lane executes, so the model now emits it as a step of the calling lane and the existing construction rule decides it: a plain `pnpm gate <id>` is accepted by shape, anything else needs an inventory entry whose digest matches. `LaneStep.inputs` is gone — every lane step is a shell block again, whether it was spelled as `run:` or as a `with:` value. An input counts as executed if it is interpolated into any run block, directly or by forwarding to another action that does. No attempt is made to work out whether the value lands in command position; that is the shell-context reconstruction this design refuses to do, so data-ish neighbours ride along and cost one entry per call site (16 today). A new caller of the same action needs its own entry, which is the intended polarity: copying a call is a fact to review. Two more surfaces close alongside: - an inline env prefix (`NODE_OPTIONS=--import ./x.ts pnpm gate lint`) used to read as a plain gate step, because segmentation stripped the prefix before the grammar saw it. It is the shell spelling of a step-level `env:` block and is now rejected the same way. Env stripping moved to unit resolution, which is the only caller that wants it; - `defaults.run` and reusable-workflow jobs fail closed. Neither is used today; either would silently retarget or hide the shell the rule is asserted over. Both new cases are end-to-end: they load a planted workflow with the real loader against the real composite action, so parse, resolution and audit are all exercised. Removing `inputSurface` fails the first; removing the surface check fails the second. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ
1 parent 9c724df commit e3e3edc

7 files changed

Lines changed: 408 additions & 100 deletions

File tree

‎CONTEXT.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -464,8 +464,9 @@ when landing it, satisfy the gate where one exists.
464464
occur inside its declared owner. Encapsulation of the one shared mutable object, enforced
465465
per-field. This covers `SessionState`; other shared state has no equivalent gate today.
466466
- **A gate that stops running is worse than no gate.** Gate: the gate manifest
467-
(`scripts/gate/`, #1429). every `run:` step a qualifying lane reaches is either `pnpm gate <check-id>` or
468-
inventoried in `scripts/gate/declarations.ts`, so the workflow→check mapping is a shape check
467+
(`scripts/gate/`, #1429). every shell block a qualifying lane reaches — a `run:` step, or a value
468+
passed to an action input the action executes — is either `pnpm gate <check-id>` or inventoried in
469+
`scripts/gate/declarations.ts`, so the workflow→check mapping is a shape check
469470
rather than an interpretation of shell, and a gate that is not in the registry
470471
(`scripts/check-affected/checks.ts`) cannot run at all — the failure
471472
mode is a red build, not an invisible one. `check:gate-manifest` fails when a registered check

‎docs/agents/testing.md‎

Lines changed: 24 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -216,23 +216,33 @@ strings, and four suites (`check:tmpdir-leaks`, its model tests, `test:fixture-c
216216

217217
`pnpm check:gate-manifest` (`scripts/gate/`) answers it with one rule and one primitive.
218218

219-
The rule: **every `run:` step a qualifying lane reaches is either `pnpm gate <check-id>`
220-
or listed in `NON_GATE_STEPS`.** Nothing inspects what a command does. That is deliberate
219+
The rule: **every shell block a qualifying lane reaches is either `pnpm gate <check-id>`
220+
or listed in `NON_GATE_STEPS`.** A shell block is a `run:` step — in a workflow or in a local
221+
composite action — or a value passed to an action input that the action interpolates into a
222+
`run:`. That second kind matters because `setup-apple-runner-build` runs
223+
`${{ inputs.build-command }}`: its own step digest is constant, so the command is really written
224+
at the call site, and the caller's value is held to the rule instead. Nothing inspects what a
225+
command does. That is deliberate
221226
and was learned the hard way: a content-based classifier was defeated by `pnpm exec`, then
222227
by `pnpm exec --` and `npx --yes`, then by `node -e 'import("./scripts/…")'` — and after
223228
those, `eval`, heredocs and base64 remain. "Does this text run project code?" is not
224229
answerable from text, so the question is not asked. A step is allowed because of its
225230
shape; everything else is written down by a human, which is the only boundary an unknown
226231
spelling cannot walk through.
227232

228-
The price is a long inventory (74 steps today, keyed on the file that declares them, so a
229-
composite action shared by eight lanes is listed once). That is the census of everything
230-
CI does outside the runner, and it fails in both directions — an unlisted step fails as a
231-
bypass, a listed step that is renamed or deleted fails as inert. Every gate is a `CheckId`
232-
in the registry (`scripts/check-affected/checks.ts`) — the same universe the affected-selector
233-
uses — so the workflow→check mapping is a token scan over `run:` blocks and action inputs rather
234-
than an interpretation of shell. That is what makes a new gate impossible to hide: an unregistered
235-
one cannot be wired into a lane, because project code run outside the runner fails the manifest.
233+
The price is a long inventory (104 entries today, keyed on the file that declares them, so a
234+
composite action shared by eight lanes is listed once — but an executed input is listed per call
235+
site, because a new caller of the same action is a new command to review). That is the census of
236+
everything CI does outside the runner, and it fails in both directions — an unlisted step fails as
237+
a bypass, a listed step whose body is edited fails as both, since entries bind the step's digest
238+
(`run` plus `env`/`shell`/`working-directory`) rather than its name. Lane-level `env:` is
239+
fingerprinted the same way in `LANE_ENVIRONMENTS`, and the two execution surfaces the loader does
240+
not read — `defaults.run` and reusable-workflow jobs — fail closed rather than being ignored.
241+
Every gate is a `CheckId` in the registry (`scripts/check-affected/checks.ts`) — the same universe
242+
the affected-selector uses — so the workflow→check mapping is a token scan for `pnpm gate <id>`
243+
rather than an interpretation of shell. That is what makes a new gate impossible to hide: an
244+
unregistered one cannot be wired into a lane, because project code run outside the runner fails
245+
the manifest.
236246

237247
The primitive: `covered(check, path)` — some `pull_request`/`schedule` lane runs every *unit* of
238248
the check, and (when a path is given) a change to that path starts the lane. Units are Vitest
@@ -253,10 +263,10 @@ matching a live step fails as inert. That polarity is the point — the predeces
253263
list made a *deleted* entry quieter, which is the one direction a ledger must never have.
254264

255265
Worth knowing before you edit the selector: the category universe is not parsed out of
256-
`scripts/check-affected/model.ts` any more. The manifest *runs* the selector on a representative
257-
path per category (`PATH_SAMPLES`) and asserts against what it really returns, so there is no
258-
static reader to defeat. Samples must be tracked files that still classify — a sample naming a
259-
path no PR can touch fails rather than resolving by prefix and reading green.
266+
`scripts/check-affected/model.ts` any more, and it is not listed by hand either. The manifest runs
267+
the real selector over every tracked file and keeps the first path that produces each rule, so a
268+
category cannot name a path no PR can touch — the hand-written predecessor had eight such samples,
269+
which resolved by prefix and read green.
260270

261271
The gate is deterministic, offline, and needs no GitHub token — it runs from a clean checkout in
262272
the `Affected-check Selector` job. Branch-protection required-contexts drift is the one part

‎scripts/gate/audit.test.ts‎

Lines changed: 96 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -9,11 +9,21 @@
99

1010
import { execFileSync } from 'node:child_process';
1111
import assert from 'node:assert/strict';
12+
import fs from 'node:fs';
13+
import os from 'node:os';
1214
import path from 'node:path';
1315
import test from 'node:test';
1416
import { audit } from './audit.ts';
1517
import { LANE_ENVIRONMENTS, NON_GATE_STEPS } from './declarations.ts';
16-
import { categories, covered, loadModel, stepDigest, type Lane, type Model } from './model.ts';
18+
import {
19+
categories,
20+
covered,
21+
loadLanes,
22+
loadModel,
23+
stepDigest,
24+
type Lane,
25+
type Model,
26+
} from './model.ts';
1727
import { CHECK_CATALOG } from '../check-affected/checks.ts';
1828

1929
const repoRoot = path.resolve(import.meta.dirname, '../..');
@@ -44,7 +54,6 @@ function plantCommand(run: string, extras: Record<string, string> = {}): Model {
4454
name: 'Planted',
4555
source: 'ci.yml',
4656
run,
47-
inputs: [],
4857
extras,
4958
digest: stepDigest(run, extras),
5059
};
@@ -82,6 +91,88 @@ function editStep(
8291
const bypassesFor = (run: string, extras: Record<string, string> = {}) =>
8392
audit(plantCommand(run, extras)).filter((failure) => failure.assertion === 'bypass');
8493

94+
/**
95+
* Load a planted workflow with the REAL loader, resolving `./.github/actions/…` against the
96+
* real tree. The composite action under test is the one CI uses, so these cases exercise
97+
* parse, resolution and audit together rather than a hand-built lane.
98+
*/
99+
function plantedWorkflow(yaml: string): Model {
100+
const dir = fs.mkdtempSync(path.join(os.tmpdir(), 'gate-lane-'));
101+
try {
102+
fs.writeFileSync(path.join(dir, 'planted.yml'), yaml);
103+
const lanes = loadLanes(dir, repoRoot, base.scripts);
104+
return mutate((model) => ({ lanes: [...model.lanes, ...lanes] }));
105+
} finally {
106+
fs.rmSync(dir, { recursive: true, force: true });
107+
}
108+
}
109+
110+
const applyRunnerCall = (command: string) => `name: Planted
111+
on:
112+
pull_request:
113+
jobs:
114+
planted:
115+
name: Planted Apple Lane
116+
runs-on: macos-15
117+
steps:
118+
- name: Build the Apple runner
119+
uses: ./.github/actions/setup-apple-runner-build
120+
with:
121+
derived-path: /tmp/derived
122+
cache-key-prefix: planted
123+
build-command: ${command}
124+
`;
125+
126+
test('a caller cannot smuggle code through an input a composite action executes', () => {
127+
// Review round 5: `setup-apple-runner-build` runs `${{ inputs.build-command }}`, so its
128+
// own step digest is CONSTANT and vouches for nothing — the command lives at the call
129+
// site. Both models below reach a byte-identical action; only the `with:` value differs.
130+
assert.deepEqual(
131+
messages(plantedWorkflow(applyRunnerCall('pnpm gate swift-runner-ios'))),
132+
[],
133+
'a call site whose executable input is a gate needs no declaration',
134+
);
135+
136+
const smuggled = audit(
137+
plantedWorkflow(applyRunnerCall(`node -e 'import("./scripts/layering/check.ts")'`)),
138+
).filter((failure) => failure.assertion === 'bypass');
139+
assert.equal(smuggled.length, 1, 'and project code in the same value is a bypass');
140+
assert.match(smuggled[0]?.message ?? '', /build-command/);
141+
});
142+
143+
test('an execution surface the model does not read is rejected rather than ignored', () => {
144+
// Neither shape is used today. `defaults.run` retargets the shell of every step in a
145+
// lane, and a reusable-workflow job runs steps from a file the loader never opens, so
146+
// both would make the no-bypass rule quietly incomplete.
147+
const surfaces = (yaml: string) =>
148+
audit(plantedWorkflow(yaml)).filter((failure) => failure.assertion === 'surface');
149+
150+
const found = surfaces(`name: Planted
151+
on:
152+
pull_request:
153+
defaults:
154+
run:
155+
shell: bash -lc {0}
156+
jobs:
157+
planted:
158+
runs-on: ubuntu-latest
159+
steps:
160+
- run: pnpm gate layering
161+
`);
162+
assert.equal(found.length, 1);
163+
assert.match(found[0]?.message ?? '', /defaults:/);
164+
165+
const reusable = surfaces(`name: Planted
166+
on:
167+
pull_request:
168+
jobs:
169+
planted:
170+
uses: ./.github/workflows/ci.yml
171+
`);
172+
assert.equal(reusable.length, 1);
173+
assert.match(reusable[0]?.message ?? '', /reusable workflow/);
174+
});
175+
85176
test('the live tree is green — every planted failure below is a real difference', () => {
86177
assert.deepEqual(messages(base), []);
87178
});
@@ -175,6 +266,8 @@ test('a qualifying lane accepts nothing but a gate invocation or a listed step',
175266
'echo bm9kZSBzY3JpcHRzL2xheWVyaW5nL2NoZWNrLnRz | base64 -d | sh',
176267
"bash <<'EOS'\nnode scripts/layering/check.ts\nEOS",
177268
'printf "node scripts/layering/check.ts" > /tmp/x.sh && sh /tmp/x.sh',
269+
// An env prefix is a `env:` block spelled in the shell, and injects the same way.
270+
'NODE_OPTIONS=--import ./scripts/layering/check.ts pnpm gate layering',
178271
];
179272
for (const run of evasions) {
180273
assert.equal(bypassesFor(run).length, 1, `must be rejected: ${run}`);
@@ -307,9 +400,7 @@ test('an entry goes inert when the step it describes stops existing', () => {
307400
})),
308401
}));
309402
assert.ok(
310-
messages(deleted).some((message) =>
311-
/matches no `run:` step a qualifying lane reaches/.test(message),
312-
),
403+
messages(deleted).some((message) => /matches no shell a qualifying lane reaches/.test(message)),
313404
'a deleted step orphans its entry',
314405
);
315406

‎scripts/gate/audit.ts‎

Lines changed: 33 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -69,12 +69,14 @@ function gateSegment(segment: string): { id: string } | null {
6969
* declares no extras. Extras (`env`, `shell`, `working-directory`) change what runs
7070
* without appearing in a command — `NODE_OPTIONS=--import ./x.ts` is a bypass with an
7171
* empty command line — so a gate step carrying any of them is fingerprinted instead.
72+
* An inline `VAR=… pnpm gate x` is the same thing spelled in the shell, and is rejected
73+
* the same way: `commandSegments` keeps the prefix, and the grammar starts at `pnpm`.
7274
*/
7375
function plainGateStep(step: {
74-
run?: string;
76+
run: string;
7577
extras: Readonly<Record<string, string>>;
7678
}): string[] | null {
77-
if (step.run === undefined || Object.keys(step.extras).length > 0) return null;
79+
if (Object.keys(step.extras).length > 0) return null;
7880
const segments = commandSegments(stripExpressions(step.run));
7981
if (segments.length === 0) return null;
8082
const ids: string[] = [];
@@ -98,12 +100,16 @@ function plainGateStep(step: {
98100
// editing the body behind a listed name. A name is mutable metadata; the digest binds
99101
// `run` plus every execution-affecting key, so changing any of them makes the entry stop
100102
// matching — the step becomes unlisted (bypass) and the entry becomes inert, at once.
103+
//
104+
// Round 5 found the same hole one level out: a composite action whose step is
105+
// `run: ${{ inputs.build-command }}` has a CONSTANT digest, so a caller could put
106+
// anything in the `with:` value and nothing here would move. The answer is not another
107+
// rule — such a value is a step of the calling lane (model.ts `inputSurface`), so it
108+
// arrives here and this rule already decides it.
101109
function bypass(model: Model, declared: readonly Unrouted[]): Failure[] {
102110
const seen = new Map<string, Model['lanes'][number]['steps'][number]>();
103111
for (const lane of model.lanes.filter((entry) => entry.qualifying)) {
104-
for (const step of lane.steps) {
105-
if (step.run !== undefined) seen.set(`${step.source}\u0000${step.digest}`, step);
106-
}
112+
for (const step of lane.steps) seen.set(`${step.source}\u0000${step.digest}`, step);
107113
}
108114
return [...seen.values()].flatMap((step) => {
109115
const ids = plainGateStep(step);
@@ -129,7 +135,7 @@ function bypass(model: Model, declared: readonly Unrouted[]): Failure[] {
129135
return [
130136
fail(
131137
'bypass',
132-
`${step.source} / ${step.name}: a \`run:\` step a qualifying lane reaches must be ` +
138+
`${step.source} / ${step.name}: shell a qualifying lane reaches must be ` +
133139
`\`pnpm gate <id>\` with no env/shell/working-directory, or inventoried — ${detail}.`,
134140
),
135141
];
@@ -160,6 +166,25 @@ function laneEnvironments(model: Model, declared: readonly LaneEnvironment[]): F
160166
});
161167
}
162168

169+
// 3c. Execution surfaces the model does not read are rejected rather than ignored.
170+
// `defaults.run` retargets the shell of every step in a lane, and a reusable-workflow
171+
// job runs steps from a file this loader never opens — either would make the
172+
// construction rule quietly incomplete. Neither is used today, which is when a
173+
// fail-closed rule is cheap to add.
174+
function laneSurfaces(model: Model): Failure[] {
175+
return model.lanes
176+
.filter((lane) => lane.qualifying)
177+
.flatMap((lane) =>
178+
lane.unsupported.map((surface) =>
179+
fail(
180+
'surface',
181+
`${lane.workflow} / ${lane.label}: ${surface} is an execution surface the gate ` +
182+
`manifest does not model. Model it before using it, or the no-bypass rule is blind.`,
183+
),
184+
),
185+
);
186+
}
187+
163188
function inertEnvironment(entry: LaneEnvironment, model: Model): Failure[] {
164189
const live = model.lanes.some(
165190
(lane) =>
@@ -211,7 +236,7 @@ function inertStep(entry: Unrouted, qualifying: readonly Lane[]): Failure[] {
211236
fail(
212237
'inert',
213238
`NON_GATE_STEPS ${entry.workflow} / "${entry.step}" (digest ${entry.digest}) matches no ` +
214-
`\`run:\` step a qualifying lane reaches; the step was edited, renamed away or deleted.`,
239+
`shell a qualifying lane reaches; the step was edited, renamed away or deleted.`,
215240
),
216241
];
217242
}
@@ -294,6 +319,7 @@ export function audit(model: Model, declared: readonly Unrouted[] = NON_GATE_STE
294319
...unowned(model),
295320
...bypass(model, declared),
296321
...laneEnvironments(model, LANE_ENVIRONMENTS),
322+
...laneSurfaces(model),
297323
...LANE_ENVIRONMENTS.flatMap((entry) => inertEnvironment(entry, model)),
298324
...pathCoverage(model),
299325
...inert(model, declared),

‎scripts/gate/check.ts‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ const HEADINGS: Readonly<Record<string, string>> = {
1212
owned: 'Registered checks no lane runs',
1313
bypass: 'Project code run outside `pnpm gate`',
1414
'lane-env': 'Lane environments that are not inventoried',
15+
surface: 'Execution surfaces the manifest does not model',
1516
'path-coverage': 'Paths whose selected checks no triggered lane runs',
1617
inert: 'Declarations that no longer apply',
1718
registered: 'Suites and projects no registered check covers',

0 commit comments

Comments
 (0)