Skip to content

Commit ccf64f6

Browse files
thymikeethymikee
authored
ci: move parked device replay suites to a dispatch-only workflow (#1781 A1) (#1794)
* ci: move parked device replay suites to a dispatch-only workflow (#1781 A1) Both full-tier device jobs have failed every scheduled run since 2026-07-24: the Android suite inside full-tier scenarios that had never executed end to end, the iOS suite on varying steps. They move to .github/workflows/replays-manual.yml, which has no `schedule:`, so the schedule stops emitting a guaranteed failure while the suites stay runnable on demand. A job-level `if: github.event_name == 'workflow_dispatch'` would have looked the same and lied: `workflowLanes()` decides `qualifying` per workflow FILE and never reads job-level `if:`, so the manifest kept reporting replay-android, replay-ios, and replay-ios-device as scheduled-lane owners — the silent-owner-loss failure the manifest exists to catch. A separate file is what the file-level model already reads correctly. Those three checks now have no pull_request/schedule owner, so they are declared as MANUAL_ONLY_OWNERS rather than folded into UNPROVABLE_OWNERS, whose claim ("it runs, this loader cannot see it") is no longer true for replay-android. check:gate-manifest drops from 48 to 46 wired checks and names the three on every run. Two tests pin it: a dispatch-only lane is non-qualifying however many gates it declares, and every manual-only declaration must name a registered check that no qualifying lane owns, so a re-scheduled lane cannot keep a stale exemption. * ci: attest manual-only checks against their dispatch lane (#1781 A1) Review P1: MANUAL_ONLY_OWNERS was a negative allowlist — it proved each entry named a registered check no qualifying lane owned, but nothing tied the entry to a lane that can still run it. Deleting a parked job, or its run-gate step, would have left the manifest green and still printing the check as manual-only: parked coverage silently becoming deleted coverage. Each entry now names its dispatch lane, and a new 'manual-only' audit assertion resolves that name against the derived model: the lane must exist, must still be dispatch-only, and must still declare the gate. replay-android carries an explicit `opaque` flag because its gate sits inside the third-party emulator action's `script:` (#1429), so the job's existence is the whole attestation the model can make — and the flag says so rather than letting an unreadable lane look like a declaring one. Four regressions pin both directions: deleting a declaration reports the check as unowned; deleting the parked job fails with 'no workflow defines'; re-scheduling the lane fails until the entry is dropped; and a parked lane that loses its run-gate step fails unless the entry is opaque. * ci: make manual-only mean dispatch-only, not merely non-qualifying (#1781 A1) Review follow-up: the attestation checked `qualifying === false`, which is true of any lane that is not pull_request/schedule. Swapping `workflow_dispatch` for `push` in replays-manual.yml would have kept the audit green and the checks printed as manual-only, while the runs nobody starts by hand quietly started themselves on every push. The lane model now keeps the trigger names instead of collapsing them into that one bit, and the manual-only assertion requires `workflow_dispatch` and nothing else. Three planted regressions cover the gap the review named: a parked lane re-triggered by `push` fails, a parked lane with no trigger at all fails, and the loader test pins that trigger kinds survive into the model (a push lane reads `[push]`, the nightly reads `[schedule, workflow_dispatch]`).
1 parent e3f3a24 commit ccf64f6

9 files changed

Lines changed: 532 additions & 201 deletions

File tree

‎.github/workflows/replays-manual.yml‎

Lines changed: 197 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,197 @@
1+
name: Replay Manual
2+
3+
# Parked here by #1781 A1. Both suites failed every scheduled run from 2026-07-24 on — the
4+
# Android suite inside full-tier scenarios that had never executed end to end, the iOS suite
5+
# on varying steps — so on a schedule they only produced noise. Manual dispatch keeps them
6+
# runnable while they are being repaired; `pnpm check:gate-manifest` reports the checks they
7+
# declare as manual-only (scripts/gate/declarations.ts) so the missing owner stays visible.
8+
# Move these jobs back into replays-nightly.yml once a dispatch run is green.
9+
10+
on:
11+
workflow_dispatch:
12+
13+
permissions:
14+
contents: read
15+
actions: read
16+
17+
concurrency:
18+
group: ci-${{ github.workflow }}-${{ github.ref }}
19+
cancel-in-progress: true
20+
21+
jobs:
22+
manual-android:
23+
name: Android Full Emulator Suite
24+
runs-on: ubuntu-latest
25+
timeout-minutes: 15
26+
env:
27+
AGENT_DEVICE_STATE_DIR: ${{ github.workspace }}/.tmp/android-e2e-state
28+
steps:
29+
- name: Checkout
30+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
31+
32+
- name: Setup toolchain
33+
uses: ./.github/actions/setup-node-pnpm
34+
35+
- name: Setup Android replay host
36+
id: android-replay-host
37+
uses: ./.github/actions/setup-android-replay-host
38+
with:
39+
package-helpers: 'true'
40+
41+
- name: Restore Android fixture APK
42+
id: fixture-app
43+
uses: ./.github/actions/setup-fixture-app
44+
with:
45+
platform: android
46+
install: 'false'
47+
require-artifact: 'true'
48+
49+
- name: Mark Android emulator setup complete
50+
id: android-setup
51+
run: echo "seconds=$(date +%s)" >> "$GITHUB_OUTPUT"
52+
53+
- name: Run Android full emulator suite
54+
uses: reactivecircus/android-emulator-runner@b530d96654c385303d652368551fb075bc2f0b6b # v2.35.0
55+
with:
56+
api-level: 36
57+
arch: x86_64
58+
profile: pixel_7
59+
target: google_apis_playstore
60+
emulator-options: -no-window -gpu swiftshader_indirect -no-snapshot -noaudio -no-boot-anim -no-metrics
61+
script: |
62+
# android-emulator-runner executes this with /usr/bin/sh, which is dash on
63+
# ubuntu runners and rejects `-o pipefail`. The serial assignment below is
64+
# validated by the `test -n` guard instead.
65+
set -eu
66+
ANDROID_SERIAL="$(adb devices | awk 'NR > 1 && $2 == "device" { print $1; exit }')"
67+
test -n "$ANDROID_SERIAL"
68+
BOOT_FINISHED_AT="$(date +%s)"
69+
pnpm gate build
70+
pnpm clean:daemon
71+
adb -s "$ANDROID_SERIAL" install -r "${{ steps.fixture-app.outputs.apk-path }}"
72+
node --experimental-strip-types src/bin.ts replay examples/test-app/replays/drag-android.ad --platform android --serial "$ANDROID_SERIAL" --session nightly-android-target-drag --json
73+
pnpm clean:daemon
74+
FULL_STARTED_AT="$(date +%s)"
75+
AGENT_DEVICE_ANDROID_E2E=1 AGENT_DEVICE_ANDROID_E2E_TIER=full AGENT_DEVICE_ANDROID_SERIAL="$ANDROID_SERIAL" AGENT_DEVICE_FIXTURE_APP_PATH="${{ steps.fixture-app.outputs.apk-path }}" AGENT_DEVICE_FIXTURE_APP_ID="${{ steps.fixture-app.outputs.app-id }}" node --experimental-strip-types scripts/node-test-tmpdir.ts --test test/integration/smoke-android-emulator.test.ts
76+
FULL_FINISHED_AT="$(date +%s)"
77+
echo "Android setup before emulator: $((BOOT_FINISHED_AT - ${{ steps.android-setup.outputs.seconds }}))s"
78+
echo "Android full scenario wall time: $((FULL_FINISHED_AT - FULL_STARTED_AT))s"
79+
pnpm gate replay-android --retries 2 --artifacts-dir test/artifacts/replays-android --report-junit test/artifacts/replays-android.junit.xml
80+
81+
- name: Upload Android artifacts
82+
if: always()
83+
uses: ./.github/actions/upload-agent-device-artifacts
84+
with:
85+
artifact-name: replay-nightly-android-artifacts
86+
agent-state-dir: ${{ env.AGENT_DEVICE_STATE_DIR }}
87+
88+
manual-ios:
89+
name: iOS Replay Suite
90+
runs-on: macos-26
91+
timeout-minutes: 80
92+
env:
93+
IOS_RUNTIME_VERSION: '26.2'
94+
AGENT_DEVICE_STATE_DIR: ${{ github.workspace }}/.tmp/agent-device-state
95+
AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH: ${{ github.workspace }}/.tmp/ios-runner-derived
96+
AGENT_DEVICE_IOS_PREPARE_TIMEOUT_MS: '420000'
97+
AGENT_DEVICE_IOS_APP_EVENT_URL_TEMPLATE: agent-device-test-app:///automation?event={event}&payload={payload}
98+
steps:
99+
- name: Checkout
100+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
101+
102+
- name: Setup toolchain
103+
uses: ./.github/actions/setup-node-pnpm
104+
with:
105+
cache-dependency-path: |
106+
pnpm-lock.yaml
107+
examples/test-app/pnpm-lock.yaml
108+
109+
- name: Restore and build iOS XCTest runner
110+
uses: ./.github/actions/setup-apple-runner-build
111+
with:
112+
derived-path: ${{ env.AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH }}
113+
cache-key-prefix: ios-runner-prebuilt
114+
cache-key-suffix: -ios-${{ env.IOS_RUNTIME_VERSION }}
115+
gate: swift-runner-ios
116+
xcuitest-platform: ios
117+
xcuitest-destination: generic/platform=iOS Simulator
118+
119+
- name: Boot iOS test simulator
120+
id: ios-simulator
121+
uses: ./.github/actions/boot-ios-test-simulator
122+
with:
123+
runtime-version: ${{ env.IOS_RUNTIME_VERSION }}
124+
preferred-device-name: iPhone 17 Pro
125+
126+
- name: Preflight iOS runner through public CLI
127+
run: |
128+
pnpm clean:daemon
129+
node --experimental-strip-types src/bin.ts prepare ios-runner --platform ios --udid "${{ steps.ios-simulator.outputs.simulator-udid }}" --timeout "$AGENT_DEVICE_IOS_PREPARE_TIMEOUT_MS" --json
130+
pnpm clean:daemon
131+
132+
- name: Run iOS simulator replay suite
133+
uses: ./.github/actions/run-gate
134+
with:
135+
gate: replay-ios
136+
args: |
137+
--udid
138+
${{ steps.ios-simulator.outputs.simulator-udid }}
139+
--retries
140+
2
141+
--artifacts-dir
142+
test/artifacts/replays-ios-simulator
143+
--report-junit
144+
test/artifacts/replays-ios-simulator.junit.xml
145+
146+
- name: Fetch current fixture app
147+
id: fixture-app
148+
uses: ./.github/actions/setup-fixture-app
149+
with:
150+
install: 'false'
151+
152+
- name: Report fixture cache source
153+
run: |
154+
echo "Fixture app source: ${{ steps.fixture-app.outputs.source }}" >> "$GITHUB_STEP_SUMMARY"
155+
156+
- name: Prove selector drag reaches its destination on iOS
157+
run: |
158+
xcrun simctl install "${{ steps.ios-simulator.outputs.simulator-udid }}" "${{ steps.fixture-app.outputs.app-path }}"
159+
pnpm clean:daemon
160+
node --experimental-strip-types src/bin.ts replay examples/test-app/replays/drag.ad --platform ios --udid "${{ steps.ios-simulator.outputs.simulator-udid }}" --session nightly-ios-target-drag --json
161+
pnpm clean:daemon
162+
163+
- name: Run full fixture-backed iOS simulator E2E
164+
env:
165+
AGENT_DEVICE_FIXTURE_APP_ID: ${{ steps.fixture-app.outputs.app-id }}
166+
AGENT_DEVICE_FIXTURE_APP_PATH: ${{ steps.fixture-app.outputs.app-path }}
167+
AGENT_DEVICE_IOS_E2E: '1'
168+
AGENT_DEVICE_IOS_E2E_TIER: full
169+
AGENT_DEVICE_IOS_UDID: ${{ steps.ios-simulator.outputs.simulator-udid }}
170+
run: |
171+
pnpm gate build
172+
node --experimental-strip-types scripts/node-test-tmpdir.ts --test test/integration/smoke-ios-simulator-coverage.test.ts test/integration/smoke-ios-simulator.test.ts
173+
174+
- name: Run iOS physical device replay suite
175+
if: env.IOS_UDID != ''
176+
env:
177+
IOS_UDID: ${{ vars.IOS_UDID }}
178+
uses: ./.github/actions/run-gate
179+
with:
180+
gate: replay-ios-device
181+
args: |
182+
--udid
183+
${{ env.IOS_UDID }}
184+
--retries
185+
2
186+
--artifacts-dir
187+
test/artifacts/replays-ios-device
188+
--report-junit
189+
test/artifacts/replays-ios-device.junit.xml
190+
191+
- name: Upload iOS artifacts
192+
if: always()
193+
uses: ./.github/actions/upload-agent-device-artifacts
194+
with:
195+
artifact-name: replay-nightly-ios-artifacts
196+
agent-state-dir: ${{ env.AGENT_DEVICE_STATE_DIR }}
197+
runner-derived-path: ${{ env.AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH }}

‎.github/workflows/replays-nightly.yml‎

Lines changed: 4 additions & 177 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
name: Replay Nightly
22

3+
# The device replay suites this workflow used to carry moved to replays-manual.yml (#1781 A1)
4+
# after failing every scheduled run from 2026-07-24 on. What stays here is what a schedule can
5+
# still keep honest: the device-free parser fuzz lane.
6+
37
on:
48
schedule:
59
- cron: '0 3 * * *'
@@ -99,180 +103,3 @@ jobs:
99103
done
100104
[ "$found" = 1 ] || echo 'No run envelope was produced — the lane failed before it could run.'
101105
} >> "$GITHUB_STEP_SUMMARY"
102-
103-
nightly-android:
104-
name: Android Full Emulator Suite
105-
runs-on: ubuntu-latest
106-
timeout-minutes: 15
107-
env:
108-
AGENT_DEVICE_STATE_DIR: ${{ github.workspace }}/.tmp/android-e2e-state
109-
steps:
110-
- name: Checkout
111-
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
112-
113-
- name: Setup toolchain
114-
uses: ./.github/actions/setup-node-pnpm
115-
116-
- name: Setup Android replay host
117-
id: android-replay-host
118-
uses: ./.github/actions/setup-android-replay-host
119-
with:
120-
package-helpers: 'true'
121-
122-
- name: Restore Android fixture APK
123-
id: fixture-app
124-
uses: ./.github/actions/setup-fixture-app
125-
with:
126-
platform: android
127-
install: 'false'
128-
require-artifact: 'true'
129-
130-
- name: Mark Android emulator setup complete
131-
id: android-setup
132-
run: echo "seconds=$(date +%s)" >> "$GITHUB_OUTPUT"
133-
134-
- name: Run Android full emulator suite
135-
uses: reactivecircus/android-emulator-runner@b530d96654c385303d652368551fb075bc2f0b6b # v2.35.0
136-
with:
137-
api-level: 36
138-
arch: x86_64
139-
profile: pixel_7
140-
target: google_apis_playstore
141-
emulator-options: -no-window -gpu swiftshader_indirect -no-snapshot -noaudio -no-boot-anim -no-metrics
142-
script: |
143-
# android-emulator-runner executes this with /usr/bin/sh, which is dash on
144-
# ubuntu runners and rejects `-o pipefail`. The serial assignment below is
145-
# validated by the `test -n` guard instead.
146-
set -eu
147-
ANDROID_SERIAL="$(adb devices | awk 'NR > 1 && $2 == "device" { print $1; exit }')"
148-
test -n "$ANDROID_SERIAL"
149-
BOOT_FINISHED_AT="$(date +%s)"
150-
pnpm gate build
151-
pnpm clean:daemon
152-
adb -s "$ANDROID_SERIAL" install -r "${{ steps.fixture-app.outputs.apk-path }}"
153-
node --experimental-strip-types src/bin.ts replay examples/test-app/replays/drag-android.ad --platform android --serial "$ANDROID_SERIAL" --session nightly-android-target-drag --json
154-
pnpm clean:daemon
155-
FULL_STARTED_AT="$(date +%s)"
156-
AGENT_DEVICE_ANDROID_E2E=1 AGENT_DEVICE_ANDROID_E2E_TIER=full AGENT_DEVICE_ANDROID_SERIAL="$ANDROID_SERIAL" AGENT_DEVICE_FIXTURE_APP_PATH="${{ steps.fixture-app.outputs.apk-path }}" AGENT_DEVICE_FIXTURE_APP_ID="${{ steps.fixture-app.outputs.app-id }}" node --experimental-strip-types scripts/node-test-tmpdir.ts --test test/integration/smoke-android-emulator.test.ts
157-
FULL_FINISHED_AT="$(date +%s)"
158-
echo "Android setup before emulator: $((BOOT_FINISHED_AT - ${{ steps.android-setup.outputs.seconds }}))s"
159-
echo "Android full scenario wall time: $((FULL_FINISHED_AT - FULL_STARTED_AT))s"
160-
pnpm gate replay-android --retries 2 --artifacts-dir test/artifacts/replays-android --report-junit test/artifacts/replays-android.junit.xml
161-
162-
- name: Upload Android artifacts
163-
if: always()
164-
uses: ./.github/actions/upload-agent-device-artifacts
165-
with:
166-
artifact-name: replay-nightly-android-artifacts
167-
agent-state-dir: ${{ env.AGENT_DEVICE_STATE_DIR }}
168-
169-
nightly-ios:
170-
name: iOS Replay Suite
171-
runs-on: macos-26
172-
timeout-minutes: 80
173-
env:
174-
IOS_RUNTIME_VERSION: '26.2'
175-
AGENT_DEVICE_STATE_DIR: ${{ github.workspace }}/.tmp/agent-device-state
176-
AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH: ${{ github.workspace }}/.tmp/ios-runner-derived
177-
AGENT_DEVICE_IOS_PREPARE_TIMEOUT_MS: '420000'
178-
AGENT_DEVICE_IOS_APP_EVENT_URL_TEMPLATE: agent-device-test-app:///automation?event={event}&payload={payload}
179-
steps:
180-
- name: Checkout
181-
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
182-
183-
- name: Setup toolchain
184-
uses: ./.github/actions/setup-node-pnpm
185-
with:
186-
cache-dependency-path: |
187-
pnpm-lock.yaml
188-
examples/test-app/pnpm-lock.yaml
189-
190-
- name: Restore and build iOS XCTest runner
191-
uses: ./.github/actions/setup-apple-runner-build
192-
with:
193-
derived-path: ${{ env.AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH }}
194-
cache-key-prefix: ios-runner-prebuilt
195-
cache-key-suffix: -ios-${{ env.IOS_RUNTIME_VERSION }}
196-
gate: swift-runner-ios
197-
xcuitest-platform: ios
198-
xcuitest-destination: generic/platform=iOS Simulator
199-
200-
- name: Boot iOS test simulator
201-
id: ios-simulator
202-
uses: ./.github/actions/boot-ios-test-simulator
203-
with:
204-
runtime-version: ${{ env.IOS_RUNTIME_VERSION }}
205-
preferred-device-name: iPhone 17 Pro
206-
207-
- name: Preflight iOS runner through public CLI
208-
run: |
209-
pnpm clean:daemon
210-
node --experimental-strip-types src/bin.ts prepare ios-runner --platform ios --udid "${{ steps.ios-simulator.outputs.simulator-udid }}" --timeout "$AGENT_DEVICE_IOS_PREPARE_TIMEOUT_MS" --json
211-
pnpm clean:daemon
212-
213-
- name: Run iOS simulator replay suite
214-
uses: ./.github/actions/run-gate
215-
with:
216-
gate: replay-ios
217-
args: |
218-
--udid
219-
${{ steps.ios-simulator.outputs.simulator-udid }}
220-
--retries
221-
2
222-
--artifacts-dir
223-
test/artifacts/replays-ios-simulator
224-
--report-junit
225-
test/artifacts/replays-ios-simulator.junit.xml
226-
227-
- name: Fetch current fixture app
228-
id: fixture-app
229-
uses: ./.github/actions/setup-fixture-app
230-
with:
231-
install: 'false'
232-
233-
- name: Report fixture cache source
234-
run: |
235-
echo "Fixture app source: ${{ steps.fixture-app.outputs.source }}" >> "$GITHUB_STEP_SUMMARY"
236-
237-
- name: Prove selector drag reaches its destination on iOS
238-
run: |
239-
xcrun simctl install "${{ steps.ios-simulator.outputs.simulator-udid }}" "${{ steps.fixture-app.outputs.app-path }}"
240-
pnpm clean:daemon
241-
node --experimental-strip-types src/bin.ts replay examples/test-app/replays/drag.ad --platform ios --udid "${{ steps.ios-simulator.outputs.simulator-udid }}" --session nightly-ios-target-drag --json
242-
pnpm clean:daemon
243-
244-
- name: Run full fixture-backed iOS simulator E2E
245-
env:
246-
AGENT_DEVICE_FIXTURE_APP_ID: ${{ steps.fixture-app.outputs.app-id }}
247-
AGENT_DEVICE_FIXTURE_APP_PATH: ${{ steps.fixture-app.outputs.app-path }}
248-
AGENT_DEVICE_IOS_E2E: '1'
249-
AGENT_DEVICE_IOS_E2E_TIER: full
250-
AGENT_DEVICE_IOS_UDID: ${{ steps.ios-simulator.outputs.simulator-udid }}
251-
run: |
252-
pnpm gate build
253-
node --experimental-strip-types scripts/node-test-tmpdir.ts --test test/integration/smoke-ios-simulator-coverage.test.ts test/integration/smoke-ios-simulator.test.ts
254-
255-
- name: Run iOS physical device replay suite
256-
if: env.IOS_UDID != ''
257-
env:
258-
IOS_UDID: ${{ vars.IOS_UDID }}
259-
uses: ./.github/actions/run-gate
260-
with:
261-
gate: replay-ios-device
262-
args: |
263-
--udid
264-
${{ env.IOS_UDID }}
265-
--retries
266-
2
267-
--artifacts-dir
268-
test/artifacts/replays-ios-device
269-
--report-junit
270-
test/artifacts/replays-ios-device.junit.xml
271-
272-
- name: Upload iOS artifacts
273-
if: always()
274-
uses: ./.github/actions/upload-agent-device-artifacts
275-
with:
276-
artifact-name: replay-nightly-ios-artifacts
277-
agent-state-dir: ${{ env.AGENT_DEVICE_STATE_DIR }}
278-
runner-derived-path: ${{ env.AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH }}

0 commit comments

Comments
 (0)