Skip to content

Commit 782cab3

Browse files
claudeclaude
committed
test(ci): prove every registered gate is owned and reachable (#1429)
A check that silently stops running looks exactly like a green build. Two suites had already stopped: `check:tmpdir-leaks` (with its model tests) and `test:fixture-cache` are real package scripts that no workflow ran, reachable only through the `check:unit` aggregate CI never invokes. `CHECK_CATALOG` becomes the registry of every check and `pnpm gate <id>` the only way CI runs one, so finding what a lane runs is a scan for `pnpm gate` rather than an attempt to interpret shell. `pnpm check:gate-manifest` then asserts against the real workflows that every registered check is run by some qualifying lane (per unit, not per script name), that every check the real selector activates for a path is run by a lane that path would start (#1420's class), and that every Vitest project and suite script belongs to a check. The wiring that keeps those honest is asserted too: a gate id must name a registered check, an `if:` must be ruled on in GATE_CONDITIONS so `if: false` unowns what it guards, an action declared to run a gate is proven to, and a job whose steps the loader cannot open fails closed. It deliberately does not try to prove CI runs project code only through `pnpm gate`. Whether a shell block executes project code is not decidable from its text, so shell this model does not recognise earns no ownership credit — the failure direction is a check reported unowned, never one waved through. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ
1 parent 057ab1c commit 782cab3

39 files changed

Lines changed: 2372 additions & 322 deletions

‎.github/actions/boot-ios-test-simulator/action.yml‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,12 +24,16 @@ runs:
2424
steps:
2525
- name: Resolve and boot iOS test simulator
2626
id: boot
27+
env:
28+
INPUT_RUNTIME_VERSION: ${{ inputs.runtime-version }}
29+
INPUT_PREFERRED_DEVICE_NAME: ${{ inputs.preferred-device-name }}
30+
INPUT_BOOT_TIMEOUT_SECONDS: ${{ inputs.boot-timeout-seconds }}
2731
run: |
2832
set -euo pipefail
29-
RUNTIME_VERSION="${{ inputs.runtime-version }}"
33+
RUNTIME_VERSION="$INPUT_RUNTIME_VERSION"
3034
RUNTIME_TOKEN="SimRuntime.iOS-${RUNTIME_VERSION//./-}"
3135
export RUNTIME_TOKEN
32-
export PREFERRED_DEVICE_NAME="${{ inputs.preferred-device-name }}"
36+
export PREFERRED_DEVICE_NAME="$INPUT_PREFERRED_DEVICE_NAME"
3337
UDID="$(
3438
xcrun simctl list devices -j | node -e '
3539
const fs = require("node:fs");
@@ -58,7 +62,7 @@ runs:
5862
echo "simulator-udid=$UDID" >> "$GITHUB_OUTPUT"
5963
xcrun simctl shutdown all || true
6064
xcrun simctl boot "$UDID" || true
61-
TIMEOUT_SECONDS="${{ inputs.boot-timeout-seconds }}"
65+
TIMEOUT_SECONDS="$INPUT_BOOT_TIMEOUT_SECONDS"
6266
set +e
6367
perl -e 'alarm shift; exec @ARGV' "$TIMEOUT_SECONDS" xcrun simctl bootstatus "$UDID" -b
6468
BOOTSTATUS_EXIT="$?"

‎.github/actions/setup-android-replay-host/action.yml‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -68,10 +68,7 @@ runs:
6868
- name: Package npm-bundled Android helpers
6969
if: inputs.package-helpers == 'true' && steps.android-helpers-cache.outputs.cache-hit != 'true'
7070
run: |
71-
VERSION="${{ steps.android-helper-source.outputs.version }}"
72-
rm -rf android/snapshot-helper/dist android/ime-helper/dist
73-
sh ./scripts/package-android-helper.sh snapshot "$VERSION" "v$VERSION" android/snapshot-helper/dist
74-
sh ./scripts/package-android-helper.sh ime "$VERSION" android/ime-helper/dist
71+
pnpm gate android-helpers
7572
shell: bash
7673

7774
- name: Verify packaged Android helpers

‎.github/actions/setup-apple-runner-build/action.yml‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@ inputs:
1212
description: 'Optional suffix for the cache key'
1313
required: false
1414
default: ''
15-
build-command:
16-
description: 'Command used to build Apple runner artifacts'
15+
gate:
16+
description: 'Registered check id built through `pnpm gate`'
1717
required: true
1818
xcuitest-platform:
1919
description: 'Optional AGENT_DEVICE_XCUITEST_PLATFORM value'
@@ -49,13 +49,17 @@ runs:
4949

5050
- name: Resolve Apple runner build variant
5151
id: build-variant
52+
env:
53+
INPUT_GATE: ${{ inputs.gate }}
54+
INPUT_XCUITEST_PLATFORM: ${{ inputs.xcuitest-platform }}
55+
INPUT_XCUITEST_DESTINATION: ${{ inputs.xcuitest-destination }}
5256
run: |
5357
set -euo pipefail
5458
VARIANT="$(
5559
printf '%s\n' \
56-
'${{ inputs.build-command }}' \
57-
'${{ inputs.xcuitest-platform }}' \
58-
'${{ inputs.xcuitest-destination }}' \
60+
"$INPUT_GATE" \
61+
"$INPUT_XCUITEST_PLATFORM" \
62+
"$INPUT_XCUITEST_DESTINATION" \
5963
"${AGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS:-0}" \
6064
| shasum -a 256 \
6165
| cut -c1-16
@@ -72,9 +76,10 @@ runs:
7276

7377
- name: Build Apple runner artifacts on cache miss
7478
if: steps.restore-runner-build.outputs.cache-hit != 'true'
75-
run: ${{ inputs.build-command }}
79+
run: pnpm gate "$INPUT_GATE"
7680
shell: bash
7781
env:
82+
INPUT_GATE: ${{ inputs.gate }}
7883
AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH: ${{ inputs.derived-path }}
7984
AGENT_DEVICE_XCUITEST_PLATFORM: ${{ inputs.xcuitest-platform }}
8085
AGENT_DEVICE_XCUITEST_DESTINATION: ${{ inputs.xcuitest-destination }}

‎.github/actions/setup-fixture-app/action.yml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -94,12 +94,15 @@ runs:
9494
shell: bash
9595
env:
9696
GH_TOKEN: ${{ github.token }}
97+
INPUT_PLATFORM: ${{ inputs.platform }}
98+
INPUT_WAIT_FOR_ARTIFACT_SECONDS: ${{ inputs.wait-for-artifact-seconds }}
99+
INPUT_REQUIRE_ARTIFACT: ${{ inputs.require-artifact }}
97100
run: |
98101
DEST="${{ github.workspace }}/.tmp/fixture-app"
99102
bash "$GITHUB_ACTION_PATH/fetch-artifact.sh" \
100-
"${{ inputs.platform }}" "$DEST" "${{ inputs.wait-for-artifact-seconds }}" \
103+
"$INPUT_PLATFORM" "$DEST" "$INPUT_WAIT_FOR_ARTIFACT_SECONDS" \
101104
"${{ github.repository }}" "${{ github.event.pull_request.head.sha || github.sha }}" \
102-
"$GITHUB_ACTION_PATH" "${{ inputs.require-artifact }}"
105+
"$GITHUB_ACTION_PATH" "$INPUT_REQUIRE_ARTIFACT"
103106
104107
# Fallback only: the producer builds the same way. `--device generic` avoids
105108
# needing a booted simulator; Release embeds the bundle so no Metro is needed.

‎.github/workflows/android.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ jobs:
6666
set -eu
6767
ANDROID_SERIAL="$(adb devices | awk 'NR > 1 && $2 == "device" { print $1; exit }')"
6868
test -n "$ANDROID_SERIAL"
69-
pnpm build
69+
pnpm gate build
7070
pnpm clean:daemon
7171
AGENT_DEVICE_ANDROID_E2E=1 AGENT_DEVICE_ANDROID_E2E_TIER=smoke AGENT_DEVICE_ANDROID_SERIAL="$ANDROID_SERIAL" AGENT_DEVICE_FIXTURE_APP_PATH="${{ steps.fixture-app.outputs.apk-path }}" AGENT_DEVICE_FIXTURE_APP_ID="${{ steps.fixture-app.outputs.app-id }}" node --experimental-strip-types scripts/node-test-tmpdir.ts --test test/integration/smoke-android-emulator.test.ts
7272
node --experimental-strip-types src/bin.ts test test/integration/replays/android/01-settings.ad --retries 2 --report-junit test/artifacts/replays-android-smoke.junit.xml

‎.github/workflows/ci.yml‎

Lines changed: 77 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,10 @@ jobs:
5353
name: Swift Runner Unit Compile
5454
runs-on: macos-26
5555
timeout-minutes: 20
56+
# Was an inline `VAR=1 pnpm gate …` prefix on the action's build-command input. The
57+
# input is a gate id now, so the variable lives where ios.yml already puts it: the job.
58+
env:
59+
AGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS: '1'
5660
steps:
5761
- name: Checkout
5862
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
@@ -65,7 +69,7 @@ jobs:
6569
with:
6670
derived-path: ${{ github.workspace }}/.tmp/swift-runner-unit-derived
6771
cache-key-prefix: swift-runner-unit
68-
build-command: AGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS=1 pnpm build:xcuitest:macos
72+
gate: swift-runner-macos
6973
xcuitest-platform: macos
7074
xcuitest-destination: platform=macOS,arch=arm64
7175

@@ -81,10 +85,10 @@ jobs:
8185
uses: ./.github/actions/setup-node-pnpm
8286

8387
- name: Run oxlint
84-
run: pnpm lint
88+
run: pnpm gate lint
8589

8690
- name: Check formatting
87-
run: pnpm format:check
91+
run: pnpm gate format
8892

8993
layering-guard:
9094
name: Layering Guard
@@ -107,14 +111,28 @@ jobs:
107111
# Generalizes the former inline commands/-import grep into a structured
108112
# import-direction lint over the resolved graph. See scripts/layering/check.ts
109113
# and CONTEXT.md (Architecture: folder DAG + layering lint).
110-
run: pnpm check:layering
114+
run: pnpm gate layering
111115

112116
- name: Check the depgraph report agrees with the gate
113117
# scripts/depgraph reads the same model as the gate, so its inversion count must
114118
# reproduce TYPE_INVERSION_BASELINE. Free two-sources check: if the tree changes
115119
# and only one side is updated, this fails and names the difference. Runs here
116120
# rather than in its own job so the two can never be green independently.
117-
run: pnpm depgraph:test
121+
run: pnpm gate depgraph
122+
123+
# The TMPDIR redirection both test lanes rely on (#1593/#1595). Its own tests ran
124+
# in no CI job at all until the #1429 gate manifest found them unowned.
125+
#
126+
# Deliberately NOT in the Coverage job next to `check:tmpdir-leaks`, where the
127+
# subject matter would put it: vitest-tmpdir-global-setup.test.ts proves the
128+
# lifecycle by spawning a real nested `vitest run`, and the Coverage lane already
129+
# loses runs to `[vitest-pool]: Worker forks emitted error` when a fork is slow to
130+
# terminate (observed on main at 18291ba8, unrelated to this gate). Starting a
131+
# nested Vitest seconds before the full instrumented suite is a contention risk
132+
# with nothing to gain — the manifest proves ownership per unit of work, so any
133+
# lane satisfies it.
134+
- name: Check the tmpdir redirection model
135+
run: pnpm gate tmpdir-leaks-model
118136

119137
affected-selector:
120138
name: Affected-check Selector
@@ -135,7 +153,18 @@ jobs:
135153
# The selector is fail-open and advisory (GitHub CI stays authoritative),
136154
# so the gate only guards the derivation model.
137155
- name: Check affected-selector model
138-
run: pnpm check:affected:test
156+
run: pnpm gate affected-selector
157+
158+
# The gate-of-gates (#1429). It shares this job because it validates the
159+
# same artifact the selector is built on — CHECK_CATALOG's `ciJobs` — and
160+
# because a gate that proves the other gates are wired must not be the one
161+
# gate sitting in its own job, green on its own. Deterministic and
162+
# network-free: every input is a file in the checkout.
163+
- name: Check the gate manifest model
164+
run: pnpm gate gate-manifest-model
165+
166+
- name: Check every gate is owned, wired, and reachable
167+
run: pnpm gate gate-manifest
139168

140169
maestro-conformance:
141170
name: Maestro Conformance Oracle
@@ -157,7 +186,7 @@ jobs:
157186
# device-backed layer 3 runs on the scheduled conformance-differential
158187
# workflow. See scripts/maestro-conformance/README.md.
159188
- name: Verify Maestro conformance fixtures
160-
run: pnpm maestro:conformance
189+
run: pnpm gate maestro-conformance
161190

162191
packaged-cli-node-22-12:
163192
name: Packaged CLI Node 22.12
@@ -172,8 +201,8 @@ jobs:
172201

173202
- name: Build CLI
174203
run: |
175-
pnpm build
176-
pnpm check:bundle-owner-files
204+
pnpm gate build
205+
pnpm gate bundle-owner-files
177206
178207
# The build runs on the default toolchain Node and the package is verified on the minimum
179208
# supported Node, so this job covers what a user on `engines.node` floor actually installs.
@@ -207,10 +236,10 @@ jobs:
207236
- name: Run Fallow audit
208237
env:
209238
FALLOW_BASE: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.event.before }}
210-
run: pnpm check:fallow --base "$FALLOW_BASE"
239+
run: pnpm gate fallow --base "$FALLOW_BASE"
211240

212241
- name: Check for production-unused exports
213-
run: pnpm check:production-exports
242+
run: pnpm gate production-exports
214243

215244
replay-compat-provenance:
216245
# The frozen replay-compat corpus (#1417) claims each entry was published by
@@ -230,7 +259,7 @@ jobs:
230259
uses: ./.github/actions/setup-node-pnpm
231260

232261
- name: Verify corpus entries against their released blobs
233-
run: pnpm check:replay-compat
262+
run: pnpm gate replay-compat
234263

235264
released-surface-compat:
236265
# The daemon RPC wire ledger (#1432) is compared against the ledger as it
@@ -251,10 +280,10 @@ jobs:
251280
uses: ./.github/actions/setup-node-pnpm
252281

253282
- name: Verify the wire-compat rules
254-
run: pnpm check:daemon-wire-compat:test
283+
run: pnpm gate wire-compat-model
255284

256285
- name: Compare the daemon RPC wire surface against the last released tag
257-
run: pnpm check:daemon-wire-compat
286+
run: pnpm gate daemon-wire-compat
258287

259288
coverage:
260289
# Runs the full unit + provider-integration suites under coverage with
@@ -272,20 +301,29 @@ jobs:
272301
uses: ./.github/actions/setup-node-pnpm
273302

274303
- name: Test changed-line coverage gate
275-
run: pnpm check:coverage-changed:test
304+
run: pnpm gate coverage-model
276305

277306
# The retry list is an enumerated set of owned waivers, so an expired entry
278307
# must fail before the suite runs rather than quietly keeping its retry.
279308
- name: Check contention retry policy
280-
run: pnpm check:contention-retry
309+
run: pnpm gate contention-retry
281310

282311
# Wrapped in the single-retry policy (#1419): a timeout-shaped failure in
283312
# an enumerated contention-flaky file reruns that file once and reports it
284313
# in the job summary. Assertion failures fail here on the first run.
285314
- name: Run coverage
286315
env:
287316
OUTPUT_ECONOMY_BASE: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.event.before }}
288-
run: pnpm test:coverage:ci
317+
run: pnpm gate unit-ci
318+
319+
# A run whose globalTeardown never fired (crash, OOM, timeout kill) leaves
320+
# its tmpdir behind. `check:unit` runs this locally after the suite; until
321+
# #1429 nothing ran it in CI, which is the environment where those kills
322+
# actually happen. Runs even on failure — a killed run is exactly the case
323+
# worth reporting.
324+
- name: Check for leaked test tmpdirs
325+
if: always()
326+
run: pnpm gate tmpdir-leaks
289327

290328
- name: Upload contention-retry envelope
291329
if: always()
@@ -303,7 +341,7 @@ jobs:
303341
if: always() && github.event_name == 'pull_request'
304342
env:
305343
AGENT_DEVICE_COVERAGE_WAIVER: ${{ contains(github.event.pull_request.labels.*.name, 'coverage-waiver') }}
306-
run: pnpm check:coverage-changed --base "${{ github.event.pull_request.base.sha }}"
344+
run: pnpm gate coverage --base "${{ github.event.pull_request.base.sha }}"
307345

308346
typecheck:
309347
name: Typecheck
@@ -317,7 +355,14 @@ jobs:
317355
uses: ./.github/actions/setup-node-pnpm
318356

319357
- name: Run typecheck
320-
run: pnpm typecheck
358+
run: pnpm gate typecheck
359+
360+
# CHECK_CATALOG has always claimed this job mirrors the `mcp-metadata`
361+
# check; until #1429 checked the claim, no PR job ran it at all, so
362+
# server.json/smithery.yaml drift only surfaced at publish time (where
363+
# publish-mcp-registry.yml duplicates the same command). Parse-only.
364+
- name: Check MCP registry metadata is in sync
365+
run: pnpm gate mcp-metadata
321366

322367
freerange:
323368
name: FreeRange
@@ -334,7 +379,7 @@ jobs:
334379
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
335380

336381
- name: Check numeric ranges
337-
run: pnpm check:freerange
382+
run: pnpm gate freerange
338383

339384
integration:
340385
name: Integration Tests
@@ -350,27 +395,33 @@ jobs:
350395
- name: Run integration tests
351396
run: |
352397
pnpm clean:daemon
353-
pnpm test:integration:node
398+
pnpm gate integration-node
354399
355400
- name: Run seeded concurrency torture lane (fast PR sweep)
356401
# #1416's nightly torture lane lives under test/integration/nightly/, out
357402
# of the test:integration:node glob, so this is a *deliberate* fast PR
358403
# sweep (TORTURE_RUNS default 128 seeds, ~sub-second) — not an accidental
359404
# glob inclusion. The Concurrency Torture Nightly workflow sweeps a much
360405
# larger seed range on schedule.
361-
run: pnpm test:concurrency-torture
406+
run: pnpm gate concurrency-torture
362407

363408
- name: Run provider-backed integration tests
364-
run: pnpm test:integration:provider
409+
run: pnpm gate provider-integration
365410

366411
- name: Check Provider-backed integration architecture progress
367-
run: pnpm test:integration:progress:check
412+
run: pnpm gate integration-progress
368413

369414
# A build-cache lookup outage must degrade setup-fixture-app to an inline
370415
# build, not fail the caller. This drives that step's real shell against a
371416
# failing `gh`.
372417
- name: Setup-fixture-app cache-failure fallback
373-
run: sh ./test/scripts/setup-fixture-app-fallback-smoke.sh
418+
run: pnpm gate fixture-fallback
419+
420+
# The trust rules that decide whether a cached fixture APK/app may be
421+
# reused at all — same subject as the fallback smoke above, and parse-only.
422+
# Ran in no CI job until the #1429 gate manifest found it unowned.
423+
- name: Check trusted fixture-artifact selection
424+
run: pnpm gate fixture-cache
374425

375426
web-smoke:
376427
name: Web Platform Smoke
@@ -390,7 +441,7 @@ jobs:
390441
- name: Run live web smoke
391442
run: |
392443
pnpm clean:daemon
393-
pnpm test:smoke:web
444+
pnpm gate web-smoke
394445
395446
- name: Upload web smoke artifacts
396447
if: always()

‎.github/workflows/concurrency-torture-nightly.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ name: Concurrency Torture Nightly
1010
# Scheduled + manual only: the PR gate already runs a fast default sweep via the
1111
# Node integration lane; this nightly sweeps a much larger seed range to keep
1212
# mining for ordering bugs. A failure prints the seed and the exact
13-
# `TORTURE_SEED=<n> pnpm test:concurrency-torture` replay command.
13+
# `TORTURE_SEED=<n> pnpm gate concurrency-torture` replay command.
1414

1515
on:
1616
schedule:
@@ -54,7 +54,7 @@ jobs:
5454
uses: ./.github/actions/setup-node-pnpm
5555

5656
- name: Run concurrency torture sweep
57-
run: pnpm test:concurrency-torture
57+
run: pnpm gate concurrency-torture
5858

5959
- name: Upload torture lane envelope
6060
if: always()

0 commit comments

Comments
 (0)