Skip to content

Commit 81ded2e

Browse files
claudethymikee
claude
authored and
thymikee
committed
test(ci): prove every registered gate is owned and reachable (#1429)
A check that silently stops running looks exactly like a green build. Two suites had already stopped: `check:tmpdir-leaks` (with its model tests) and `test:fixture-cache` are real package scripts that no workflow ran, reachable only through the `check:unit` aggregate CI never invokes. `CHECK_CATALOG` becomes the registry of every check and `pnpm gate <id>` the only way CI runs one, so finding what a lane runs is a scan for `pnpm gate` rather than an attempt to interpret shell. `pnpm check:gate-manifest` then asserts against the real workflows that every registered check is run by some qualifying lane (per unit, not per script name), that every check the real selector activates for a path is run by a lane that path would start (#1420's class), and that every Vitest project and suite script belongs to a check. The wiring that keeps those honest is asserted too: a gate id must name a registered check, an `if:` must be ruled on in GATE_CONDITIONS so `if: false` unowns what it guards, an action declared to run a gate is proven to, and a job whose steps the loader cannot open fails closed. It deliberately does not try to prove CI runs project code only through `pnpm gate`. Whether a shell block executes project code is not decidable from its text, so shell this model does not recognise earns no ownership credit — the failure direction is a check reported unowned, never one waved through. Co-Authored-By: Claude Opus 5 <[email protected]> Claude-Session: https://claude.ai/code/session_01SkS4S8XXrfkJ8TD1VBKkvJ
1 parent c7565cb commit 81ded2e

39 files changed

Lines changed: 2372 additions & 322 deletions

‎.github/actions/boot-ios-test-simulator/action.yml‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -24,12 +24,16 @@ runs:
2424
steps:
2525
- name: Resolve and boot iOS test simulator
2626
id: boot
27+
env:
28+
INPUT_RUNTIME_VERSION: ${{ inputs.runtime-version }}
29+
INPUT_PREFERRED_DEVICE_NAME: ${{ inputs.preferred-device-name }}
30+
INPUT_BOOT_TIMEOUT_SECONDS: ${{ inputs.boot-timeout-seconds }}
2731
run: |
2832
set -euo pipefail
29-
RUNTIME_VERSION="${{ inputs.runtime-version }}"
33+
RUNTIME_VERSION="$INPUT_RUNTIME_VERSION"
3034
RUNTIME_TOKEN="SimRuntime.iOS-${RUNTIME_VERSION//./-}"
3135
export RUNTIME_TOKEN
32-
export PREFERRED_DEVICE_NAME="${{ inputs.preferred-device-name }}"
36+
export PREFERRED_DEVICE_NAME="$INPUT_PREFERRED_DEVICE_NAME"
3337
UDID="$(
3438
xcrun simctl list devices -j | node -e '
3539
const fs = require("node:fs");
@@ -58,7 +62,7 @@ runs:
5862
echo "simulator-udid=$UDID" >> "$GITHUB_OUTPUT"
5963
xcrun simctl shutdown all || true
6064
xcrun simctl boot "$UDID" || true
61-
TIMEOUT_SECONDS="${{ inputs.boot-timeout-seconds }}"
65+
TIMEOUT_SECONDS="$INPUT_BOOT_TIMEOUT_SECONDS"
6266
set +e
6367
perl -e 'alarm shift; exec @ARGV' "$TIMEOUT_SECONDS" xcrun simctl bootstatus "$UDID" -b
6468
BOOTSTATUS_EXIT="$?"

‎.github/actions/setup-android-replay-host/action.yml‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -68,10 +68,7 @@ runs:
6868
- name: Package npm-bundled Android helpers
6969
if: inputs.package-helpers == 'true' && steps.android-helpers-cache.outputs.cache-hit != 'true'
7070
run: |
71-
VERSION="${{ steps.android-helper-source.outputs.version }}"
72-
rm -rf android/snapshot-helper/dist android/ime-helper/dist
73-
sh ./scripts/package-android-helper.sh snapshot "$VERSION" "v$VERSION" android/snapshot-helper/dist
74-
sh ./scripts/package-android-helper.sh ime "$VERSION" android/ime-helper/dist
71+
pnpm gate android-helpers
7572
shell: bash
7673

7774
- name: Verify packaged Android helpers

‎.github/actions/setup-apple-runner-build/action.yml‎

Lines changed: 11 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@ inputs:
1212
description: 'Optional suffix for the cache key'
1313
required: false
1414
default: ''
15-
build-command:
16-
description: 'Command used to build Apple runner artifacts'
15+
gate:
16+
description: 'Registered check id built through `pnpm gate`'
1717
required: true
1818
xcuitest-platform:
1919
description: 'Optional AGENT_DEVICE_XCUITEST_PLATFORM value'
@@ -49,13 +49,17 @@ runs:
4949

5050
- name: Resolve Apple runner build variant
5151
id: build-variant
52+
env:
53+
INPUT_GATE: ${{ inputs.gate }}
54+
INPUT_XCUITEST_PLATFORM: ${{ inputs.xcuitest-platform }}
55+
INPUT_XCUITEST_DESTINATION: ${{ inputs.xcuitest-destination }}
5256
run: |
5357
set -euo pipefail
5458
VARIANT="$(
5559
printf '%s\n' \
56-
'${{ inputs.build-command }}' \
57-
'${{ inputs.xcuitest-platform }}' \
58-
'${{ inputs.xcuitest-destination }}' \
60+
"$INPUT_GATE" \
61+
"$INPUT_XCUITEST_PLATFORM" \
62+
"$INPUT_XCUITEST_DESTINATION" \
5963
"${AGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS:-0}" \
6064
| shasum -a 256 \
6165
| cut -c1-16
@@ -72,9 +76,10 @@ runs:
7276

7377
- name: Build Apple runner artifacts on cache miss
7478
if: steps.restore-runner-build.outputs.cache-hit != 'true'
75-
run: ${{ inputs.build-command }}
79+
run: pnpm gate "$INPUT_GATE"
7680
shell: bash
7781
env:
82+
INPUT_GATE: ${{ inputs.gate }}
7883
AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH: ${{ inputs.derived-path }}
7984
AGENT_DEVICE_XCUITEST_PLATFORM: ${{ inputs.xcuitest-platform }}
8085
AGENT_DEVICE_XCUITEST_DESTINATION: ${{ inputs.xcuitest-destination }}

‎.github/actions/setup-fixture-app/action.yml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -94,12 +94,15 @@ runs:
9494
shell: bash
9595
env:
9696
GH_TOKEN: ${{ github.token }}
97+
INPUT_PLATFORM: ${{ inputs.platform }}
98+
INPUT_WAIT_FOR_ARTIFACT_SECONDS: ${{ inputs.wait-for-artifact-seconds }}
99+
INPUT_REQUIRE_ARTIFACT: ${{ inputs.require-artifact }}
97100
run: |
98101
DEST="${{ github.workspace }}/.tmp/fixture-app"
99102
bash "$GITHUB_ACTION_PATH/fetch-artifact.sh" \
100-
"${{ inputs.platform }}" "$DEST" "${{ inputs.wait-for-artifact-seconds }}" \
103+
"$INPUT_PLATFORM" "$DEST" "$INPUT_WAIT_FOR_ARTIFACT_SECONDS" \
101104
"${{ github.repository }}" "${{ github.event.pull_request.head.sha || github.sha }}" \
102-
"$GITHUB_ACTION_PATH" "${{ inputs.require-artifact }}"
105+
"$GITHUB_ACTION_PATH" "$INPUT_REQUIRE_ARTIFACT"
103106
104107
# Fallback only: the producer builds the same way. `--device generic` avoids
105108
# needing a booted simulator; Release embeds the bundle so no Metro is needed.

‎.github/workflows/android.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -66,7 +66,7 @@ jobs:
6666
set -eu
6767
ANDROID_SERIAL="$(adb devices | awk 'NR > 1 && $2 == "device" { print $1; exit }')"
6868
test -n "$ANDROID_SERIAL"
69-
pnpm build
69+
pnpm gate build
7070
pnpm clean:daemon
7171
AGENT_DEVICE_ANDROID_E2E=1 AGENT_DEVICE_ANDROID_E2E_TIER=smoke AGENT_DEVICE_ANDROID_SERIAL="$ANDROID_SERIAL" AGENT_DEVICE_FIXTURE_APP_PATH="${{ steps.fixture-app.outputs.apk-path }}" AGENT_DEVICE_FIXTURE_APP_ID="${{ steps.fixture-app.outputs.app-id }}" node --experimental-strip-types scripts/node-test-tmpdir.ts --test test/integration/smoke-android-emulator.test.ts
7272
node --experimental-strip-types src/bin.ts test test/integration/replays/android/01-settings.ad --retries 2 --report-junit test/artifacts/replays-android-smoke.junit.xml

‎.github/workflows/ci.yml‎

Lines changed: 77 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -53,6 +53,10 @@ jobs:
5353
name: Swift Runner Unit Compile
5454
runs-on: macos-26
5555
timeout-minutes: 20
56+
# Was an inline `VAR=1 pnpm gate …` prefix on the action's build-command input. The
57+
# input is a gate id now, so the variable lives where ios.yml already puts it: the job.
58+
env:
59+
AGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS: '1'
5660
steps:
5761
- name: Checkout
5862
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
@@ -65,7 +69,7 @@ jobs:
6569
with:
6670
derived-path: ${{ github.workspace }}/.tmp/swift-runner-unit-derived
6771
cache-key-prefix: swift-runner-unit
68-
build-command: AGENT_DEVICE_XCUITEST_INCLUDE_UNIT_TESTS=1 pnpm build:xcuitest:macos
72+
gate: swift-runner-macos
6973
xcuitest-platform: macos
7074
xcuitest-destination: platform=macOS,arch=arm64
7175

@@ -81,10 +85,10 @@ jobs:
8185
uses: ./.github/actions/setup-node-pnpm
8286

8387
- name: Run oxlint
84-
run: pnpm lint
88+
run: pnpm gate lint
8589

8690
- name: Check formatting
87-
run: pnpm format:check
91+
run: pnpm gate format
8892

8993
layering-guard:
9094
name: Layering Guard
@@ -107,14 +111,28 @@ jobs:
107111
# Generalizes the former inline commands/-import grep into a structured
108112
# import-direction lint over the resolved graph. See scripts/layering/check.ts
109113
# and CONTEXT.md (Architecture: folder DAG + layering lint).
110-
run: pnpm check:layering
114+
run: pnpm gate layering
111115

112116
- name: Check the depgraph report agrees with the gate
113117
# scripts/depgraph reads the same model as the gate, so its inversion count must
114118
# reproduce TYPE_INVERSION_BASELINE. Free two-sources check: if the tree changes
115119
# and only one side is updated, this fails and names the difference. Runs here
116120
# rather than in its own job so the two can never be green independently.
117-
run: pnpm depgraph:test
121+
run: pnpm gate depgraph
122+
123+
# The TMPDIR redirection both test lanes rely on (#1593/#1595). Its own tests ran
124+
# in no CI job at all until the #1429 gate manifest found them unowned.
125+
#
126+
# Deliberately NOT in the Coverage job next to `check:tmpdir-leaks`, where the
127+
# subject matter would put it: vitest-tmpdir-global-setup.test.ts proves the
128+
# lifecycle by spawning a real nested `vitest run`, and the Coverage lane already
129+
# loses runs to `[vitest-pool]: Worker forks emitted error` when a fork is slow to
130+
# terminate (observed on main at 18291ba8, unrelated to this gate). Starting a
131+
# nested Vitest seconds before the full instrumented suite is a contention risk
132+
# with nothing to gain — the manifest proves ownership per unit of work, so any
133+
# lane satisfies it.
134+
- name: Check the tmpdir redirection model
135+
run: pnpm gate tmpdir-leaks-model
118136

119137
# Tests for the TMPDIR redirection itself, hidden the same way as the check above.
120138
#
@@ -146,7 +164,18 @@ jobs:
146164
# The selector is fail-open and advisory (GitHub CI stays authoritative),
147165
# so the gate only guards the derivation model.
148166
- name: Check affected-selector model
149-
run: pnpm check:affected:test
167+
run: pnpm gate affected-selector
168+
169+
# The gate-of-gates (#1429). It shares this job because it validates the
170+
# same artifact the selector is built on — CHECK_CATALOG's `ciJobs` — and
171+
# because a gate that proves the other gates are wired must not be the one
172+
# gate sitting in its own job, green on its own. Deterministic and
173+
# network-free: every input is a file in the checkout.
174+
- name: Check the gate manifest model
175+
run: pnpm gate gate-manifest-model
176+
177+
- name: Check every gate is owned, wired, and reachable
178+
run: pnpm gate gate-manifest
150179

151180
maestro-conformance:
152181
name: Maestro Conformance Oracle
@@ -168,7 +197,7 @@ jobs:
168197
# device-backed layer 3 runs on the scheduled conformance-differential
169198
# workflow. See scripts/maestro-conformance/README.md.
170199
- name: Verify Maestro conformance fixtures
171-
run: pnpm maestro:conformance
200+
run: pnpm gate maestro-conformance
172201

173202
packaged-cli-node-22-12:
174203
name: Packaged CLI Node 22.12
@@ -183,8 +212,8 @@ jobs:
183212

184213
- name: Build CLI
185214
run: |
186-
pnpm build
187-
pnpm check:bundle-owner-files
215+
pnpm gate build
216+
pnpm gate bundle-owner-files
188217
189218
# The build runs on the default toolchain Node and the package is verified on the minimum
190219
# supported Node, so this job covers what a user on `engines.node` floor actually installs.
@@ -218,10 +247,10 @@ jobs:
218247
- name: Run Fallow audit
219248
env:
220249
FALLOW_BASE: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.event.before }}
221-
run: pnpm check:fallow --base "$FALLOW_BASE"
250+
run: pnpm gate fallow --base "$FALLOW_BASE"
222251

223252
- name: Check for production-unused exports
224-
run: pnpm check:production-exports
253+
run: pnpm gate production-exports
225254

226255
replay-compat-provenance:
227256
# The frozen replay-compat corpus (#1417) claims each entry was published by
@@ -241,7 +270,7 @@ jobs:
241270
uses: ./.github/actions/setup-node-pnpm
242271

243272
- name: Verify corpus entries against their released blobs
244-
run: pnpm check:replay-compat
273+
run: pnpm gate replay-compat
245274

246275
released-surface-compat:
247276
# The daemon RPC wire ledger (#1432) is compared against the ledger as it
@@ -262,10 +291,10 @@ jobs:
262291
uses: ./.github/actions/setup-node-pnpm
263292

264293
- name: Verify the wire-compat rules
265-
run: pnpm check:daemon-wire-compat:test
294+
run: pnpm gate wire-compat-model
266295

267296
- name: Compare the daemon RPC wire surface against the last released tag
268-
run: pnpm check:daemon-wire-compat
297+
run: pnpm gate daemon-wire-compat
269298

270299
coverage:
271300
# Runs the full unit + provider-integration suites under coverage with
@@ -283,20 +312,29 @@ jobs:
283312
uses: ./.github/actions/setup-node-pnpm
284313

285314
- name: Test changed-line coverage gate
286-
run: pnpm check:coverage-changed:test
315+
run: pnpm gate coverage-model
287316

288317
# The retry list is an enumerated set of owned waivers, so an expired entry
289318
# must fail before the suite runs rather than quietly keeping its retry.
290319
- name: Check contention retry policy
291-
run: pnpm check:contention-retry
320+
run: pnpm gate contention-retry
292321

293322
# Wrapped in the single-retry policy (#1419): a timeout-shaped failure in
294323
# an enumerated contention-flaky file reruns that file once and reports it
295324
# in the job summary. Assertion failures fail here on the first run.
296325
- name: Run coverage
297326
env:
298327
OUTPUT_ECONOMY_BASE: ${{ github.event_name == 'pull_request' && github.event.pull_request.base.sha || github.event.before }}
299-
run: pnpm test:coverage:ci
328+
run: pnpm gate unit-ci
329+
330+
# A run whose globalTeardown never fired (crash, OOM, timeout kill) leaves
331+
# its tmpdir behind. `check:unit` runs this locally after the suite; until
332+
# #1429 nothing ran it in CI, which is the environment where those kills
333+
# actually happen. Runs even on failure — a killed run is exactly the case
334+
# worth reporting.
335+
- name: Check for leaked test tmpdirs
336+
if: always()
337+
run: pnpm gate tmpdir-leaks
300338

301339
# The TMPDIR redirection both test lanes depend on (#1593/#1595). The check is a real
302340
# package script that no workflow ran: it is reachable only through `check:unit`, an
@@ -321,7 +359,7 @@ jobs:
321359
if: always() && github.event_name == 'pull_request'
322360
env:
323361
AGENT_DEVICE_COVERAGE_WAIVER: ${{ contains(github.event.pull_request.labels.*.name, 'coverage-waiver') }}
324-
run: pnpm check:coverage-changed --base "${{ github.event.pull_request.base.sha }}"
362+
run: pnpm gate coverage --base "${{ github.event.pull_request.base.sha }}"
325363

326364
typecheck:
327365
name: Typecheck
@@ -335,7 +373,14 @@ jobs:
335373
uses: ./.github/actions/setup-node-pnpm
336374

337375
- name: Run typecheck
338-
run: pnpm typecheck
376+
run: pnpm gate typecheck
377+
378+
# CHECK_CATALOG has always claimed this job mirrors the `mcp-metadata`
379+
# check; until #1429 checked the claim, no PR job ran it at all, so
380+
# server.json/smithery.yaml drift only surfaced at publish time (where
381+
# publish-mcp-registry.yml duplicates the same command). Parse-only.
382+
- name: Check MCP registry metadata is in sync
383+
run: pnpm gate mcp-metadata
339384

340385
freerange:
341386
name: FreeRange
@@ -352,7 +397,7 @@ jobs:
352397
uses: oven-sh/setup-bun@0c5077e51419868618aeaa5fe8019c62421857d6 # v2.2.0
353398

354399
- name: Check numeric ranges
355-
run: pnpm check:freerange
400+
run: pnpm gate freerange
356401

357402
integration:
358403
name: Integration Tests
@@ -368,27 +413,33 @@ jobs:
368413
- name: Run integration tests
369414
run: |
370415
pnpm clean:daemon
371-
pnpm test:integration:node
416+
pnpm gate integration-node
372417
373418
- name: Run seeded concurrency torture lane (fast PR sweep)
374419
# #1416's nightly torture lane lives under test/integration/nightly/, out
375420
# of the test:integration:node glob, so this is a *deliberate* fast PR
376421
# sweep (TORTURE_RUNS default 128 seeds, ~sub-second) — not an accidental
377422
# glob inclusion. The Concurrency Torture Nightly workflow sweeps a much
378423
# larger seed range on schedule.
379-
run: pnpm test:concurrency-torture
424+
run: pnpm gate concurrency-torture
380425

381426
- name: Run provider-backed integration tests
382-
run: pnpm test:integration:provider
427+
run: pnpm gate provider-integration
383428

384429
- name: Check Provider-backed integration architecture progress
385-
run: pnpm test:integration:progress:check
430+
run: pnpm gate integration-progress
386431

387432
# A build-cache lookup outage must degrade setup-fixture-app to an inline
388433
# build, not fail the caller. This drives that step's real shell against a
389434
# failing `gh`.
390435
- name: Setup-fixture-app cache-failure fallback
391-
run: sh ./test/scripts/setup-fixture-app-fallback-smoke.sh
436+
run: pnpm gate fixture-fallback
437+
438+
# The trust rules that decide whether a cached fixture APK/app may be
439+
# reused at all — same subject as the fallback smoke above, and parse-only.
440+
# Ran in no CI job until the #1429 gate manifest found it unowned.
441+
- name: Check trusted fixture-artifact selection
442+
run: pnpm gate fixture-cache
392443

393444
# The trusted-artifact contract the device lanes rely on to decide a cached fixture
394445
# app is the one this commit expects. A real test file that no workflow ran.
@@ -413,7 +464,7 @@ jobs:
413464
- name: Run live web smoke
414465
run: |
415466
pnpm clean:daemon
416-
pnpm test:smoke:web
467+
pnpm gate web-smoke
417468
418469
- name: Upload web smoke artifacts
419470
if: always()

‎.github/workflows/concurrency-torture-nightly.yml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ name: Concurrency Torture Nightly
1010
# Scheduled + manual only: the PR gate already runs a fast default sweep via the
1111
# Node integration lane; this nightly sweeps a much larger seed range to keep
1212
# mining for ordering bugs. A failure prints the seed and the exact
13-
# `TORTURE_SEED=<n> pnpm test:concurrency-torture` replay command.
13+
# `TORTURE_SEED=<n> pnpm gate concurrency-torture` replay command.
1414

1515
on:
1616
schedule:
@@ -54,7 +54,7 @@ jobs:
5454
uses: ./.github/actions/setup-node-pnpm
5555

5656
- name: Run concurrency torture sweep
57-
run: pnpm test:concurrency-torture
57+
run: pnpm gate concurrency-torture
5858

5959
- name: Upload torture lane envelope
6060
if: always()

0 commit comments

Comments
 (0)