Skip to content

Commit 2198ff5

Browse files
thymikeethymikee
committed
ci: move parked device replay suites to a dispatch-only workflow (#1781 A1)
Both full-tier device jobs have failed every scheduled run since 2026-07-24: the Android suite inside full-tier scenarios that had never executed end to end, the iOS suite on varying steps. They move to .github/workflows/replays-manual.yml, which has no `schedule:`, so the schedule stops emitting a guaranteed failure while the suites stay runnable on demand. A job-level `if: github.event_name == 'workflow_dispatch'` would have looked the same and lied: `workflowLanes()` decides `qualifying` per workflow FILE and never reads job-level `if:`, so the manifest kept reporting replay-android, replay-ios, and replay-ios-device as scheduled-lane owners — the silent-owner-loss failure the manifest exists to catch. A separate file is what the file-level model already reads correctly. Those three checks now have no pull_request/schedule owner, so they are declared as MANUAL_ONLY_OWNERS rather than folded into UNPROVABLE_OWNERS, whose claim ("it runs, this loader cannot see it") is no longer true for replay-android. check:gate-manifest drops from 48 to 46 wired checks and names the three on every run. Two tests pin it: a dispatch-only lane is non-qualifying however many gates it declares, and every manual-only declaration must name a registered check that no qualifying lane owns, so a re-scheduled lane cannot keep a stale exemption.
1 parent 3020195 commit 2198ff5

8 files changed

Lines changed: 282 additions & 193 deletions

File tree

‎.github/workflows/replays-manual.yml‎

Lines changed: 197 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,197 @@
1+
name: Replay Manual
2+
3+
# Parked here by #1781 A1. Both suites failed every scheduled run from 2026-07-24 on — the
4+
# Android suite inside full-tier scenarios that had never executed end to end, the iOS suite
5+
# on varying steps — so on a schedule they only produced noise. Manual dispatch keeps them
6+
# runnable while they are being repaired; `pnpm check:gate-manifest` reports the checks they
7+
# declare as manual-only (scripts/gate/declarations.ts) so the missing owner stays visible.
8+
# Move these jobs back into replays-nightly.yml once a dispatch run is green.
9+
10+
on:
11+
workflow_dispatch:
12+
13+
permissions:
14+
contents: read
15+
actions: read
16+
17+
concurrency:
18+
group: ci-${{ github.workflow }}-${{ github.ref }}
19+
cancel-in-progress: true
20+
21+
jobs:
22+
manual-android:
23+
name: Android Full Emulator Suite
24+
runs-on: ubuntu-latest
25+
timeout-minutes: 15
26+
env:
27+
AGENT_DEVICE_STATE_DIR: ${{ github.workspace }}/.tmp/android-e2e-state
28+
steps:
29+
- name: Checkout
30+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
31+
32+
- name: Setup toolchain
33+
uses: ./.github/actions/setup-node-pnpm
34+
35+
- name: Setup Android replay host
36+
id: android-replay-host
37+
uses: ./.github/actions/setup-android-replay-host
38+
with:
39+
package-helpers: 'true'
40+
41+
- name: Restore Android fixture APK
42+
id: fixture-app
43+
uses: ./.github/actions/setup-fixture-app
44+
with:
45+
platform: android
46+
install: 'false'
47+
require-artifact: 'true'
48+
49+
- name: Mark Android emulator setup complete
50+
id: android-setup
51+
run: echo "seconds=$(date +%s)" >> "$GITHUB_OUTPUT"
52+
53+
- name: Run Android full emulator suite
54+
uses: reactivecircus/android-emulator-runner@b530d96654c385303d652368551fb075bc2f0b6b # v2.35.0
55+
with:
56+
api-level: 36
57+
arch: x86_64
58+
profile: pixel_7
59+
target: google_apis_playstore
60+
emulator-options: -no-window -gpu swiftshader_indirect -no-snapshot -noaudio -no-boot-anim -no-metrics
61+
script: |
62+
# android-emulator-runner executes this with /usr/bin/sh, which is dash on
63+
# ubuntu runners and rejects `-o pipefail`. The serial assignment below is
64+
# validated by the `test -n` guard instead.
65+
set -eu
66+
ANDROID_SERIAL="$(adb devices | awk 'NR > 1 && $2 == "device" { print $1; exit }')"
67+
test -n "$ANDROID_SERIAL"
68+
BOOT_FINISHED_AT="$(date +%s)"
69+
pnpm gate build
70+
pnpm clean:daemon
71+
adb -s "$ANDROID_SERIAL" install -r "${{ steps.fixture-app.outputs.apk-path }}"
72+
node --experimental-strip-types src/bin.ts replay examples/test-app/replays/drag-android.ad --platform android --serial "$ANDROID_SERIAL" --session nightly-android-target-drag --json
73+
pnpm clean:daemon
74+
FULL_STARTED_AT="$(date +%s)"
75+
AGENT_DEVICE_ANDROID_E2E=1 AGENT_DEVICE_ANDROID_E2E_TIER=full AGENT_DEVICE_ANDROID_SERIAL="$ANDROID_SERIAL" AGENT_DEVICE_FIXTURE_APP_PATH="${{ steps.fixture-app.outputs.apk-path }}" AGENT_DEVICE_FIXTURE_APP_ID="${{ steps.fixture-app.outputs.app-id }}" node --experimental-strip-types scripts/node-test-tmpdir.ts --test test/integration/smoke-android-emulator.test.ts
76+
FULL_FINISHED_AT="$(date +%s)"
77+
echo "Android setup before emulator: $((BOOT_FINISHED_AT - ${{ steps.android-setup.outputs.seconds }}))s"
78+
echo "Android full scenario wall time: $((FULL_FINISHED_AT - FULL_STARTED_AT))s"
79+
pnpm gate replay-android --retries 2 --artifacts-dir test/artifacts/replays-android --report-junit test/artifacts/replays-android.junit.xml
80+
81+
- name: Upload Android artifacts
82+
if: always()
83+
uses: ./.github/actions/upload-agent-device-artifacts
84+
with:
85+
artifact-name: replay-nightly-android-artifacts
86+
agent-state-dir: ${{ env.AGENT_DEVICE_STATE_DIR }}
87+
88+
manual-ios:
89+
name: iOS Replay Suite
90+
runs-on: macos-26
91+
timeout-minutes: 80
92+
env:
93+
IOS_RUNTIME_VERSION: '26.2'
94+
AGENT_DEVICE_STATE_DIR: ${{ github.workspace }}/.tmp/agent-device-state
95+
AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH: ${{ github.workspace }}/.tmp/ios-runner-derived
96+
AGENT_DEVICE_IOS_PREPARE_TIMEOUT_MS: '420000'
97+
AGENT_DEVICE_IOS_APP_EVENT_URL_TEMPLATE: agent-device-test-app:///automation?event={event}&payload={payload}
98+
steps:
99+
- name: Checkout
100+
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
101+
102+
- name: Setup toolchain
103+
uses: ./.github/actions/setup-node-pnpm
104+
with:
105+
cache-dependency-path: |
106+
pnpm-lock.yaml
107+
examples/test-app/pnpm-lock.yaml
108+
109+
- name: Restore and build iOS XCTest runner
110+
uses: ./.github/actions/setup-apple-runner-build
111+
with:
112+
derived-path: ${{ env.AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH }}
113+
cache-key-prefix: ios-runner-prebuilt
114+
cache-key-suffix: -ios-${{ env.IOS_RUNTIME_VERSION }}
115+
gate: swift-runner-ios
116+
xcuitest-platform: ios
117+
xcuitest-destination: generic/platform=iOS Simulator
118+
119+
- name: Boot iOS test simulator
120+
id: ios-simulator
121+
uses: ./.github/actions/boot-ios-test-simulator
122+
with:
123+
runtime-version: ${{ env.IOS_RUNTIME_VERSION }}
124+
preferred-device-name: iPhone 17 Pro
125+
126+
- name: Preflight iOS runner through public CLI
127+
run: |
128+
pnpm clean:daemon
129+
node --experimental-strip-types src/bin.ts prepare ios-runner --platform ios --udid "${{ steps.ios-simulator.outputs.simulator-udid }}" --timeout "$AGENT_DEVICE_IOS_PREPARE_TIMEOUT_MS" --json
130+
pnpm clean:daemon
131+
132+
- name: Run iOS simulator replay suite
133+
uses: ./.github/actions/run-gate
134+
with:
135+
gate: replay-ios
136+
args: |
137+
--udid
138+
${{ steps.ios-simulator.outputs.simulator-udid }}
139+
--retries
140+
2
141+
--artifacts-dir
142+
test/artifacts/replays-ios-simulator
143+
--report-junit
144+
test/artifacts/replays-ios-simulator.junit.xml
145+
146+
- name: Fetch current fixture app
147+
id: fixture-app
148+
uses: ./.github/actions/setup-fixture-app
149+
with:
150+
install: 'false'
151+
152+
- name: Report fixture cache source
153+
run: |
154+
echo "Fixture app source: ${{ steps.fixture-app.outputs.source }}" >> "$GITHUB_STEP_SUMMARY"
155+
156+
- name: Prove selector drag reaches its destination on iOS
157+
run: |
158+
xcrun simctl install "${{ steps.ios-simulator.outputs.simulator-udid }}" "${{ steps.fixture-app.outputs.app-path }}"
159+
pnpm clean:daemon
160+
node --experimental-strip-types src/bin.ts replay examples/test-app/replays/drag.ad --platform ios --udid "${{ steps.ios-simulator.outputs.simulator-udid }}" --session nightly-ios-target-drag --json
161+
pnpm clean:daemon
162+
163+
- name: Run full fixture-backed iOS simulator E2E
164+
env:
165+
AGENT_DEVICE_FIXTURE_APP_ID: ${{ steps.fixture-app.outputs.app-id }}
166+
AGENT_DEVICE_FIXTURE_APP_PATH: ${{ steps.fixture-app.outputs.app-path }}
167+
AGENT_DEVICE_IOS_E2E: '1'
168+
AGENT_DEVICE_IOS_E2E_TIER: full
169+
AGENT_DEVICE_IOS_UDID: ${{ steps.ios-simulator.outputs.simulator-udid }}
170+
run: |
171+
pnpm gate build
172+
node --experimental-strip-types scripts/node-test-tmpdir.ts --test test/integration/smoke-ios-simulator-coverage.test.ts test/integration/smoke-ios-simulator.test.ts
173+
174+
- name: Run iOS physical device replay suite
175+
if: env.IOS_UDID != ''
176+
env:
177+
IOS_UDID: ${{ vars.IOS_UDID }}
178+
uses: ./.github/actions/run-gate
179+
with:
180+
gate: replay-ios-device
181+
args: |
182+
--udid
183+
${{ env.IOS_UDID }}
184+
--retries
185+
2
186+
--artifacts-dir
187+
test/artifacts/replays-ios-device
188+
--report-junit
189+
test/artifacts/replays-ios-device.junit.xml
190+
191+
- name: Upload iOS artifacts
192+
if: always()
193+
uses: ./.github/actions/upload-agent-device-artifacts
194+
with:
195+
artifact-name: replay-nightly-ios-artifacts
196+
agent-state-dir: ${{ env.AGENT_DEVICE_STATE_DIR }}
197+
runner-derived-path: ${{ env.AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH }}

‎.github/workflows/replays-nightly.yml‎

Lines changed: 4 additions & 177 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,9 @@
11
name: Replay Nightly
22

3+
# The device replay suites this workflow used to carry moved to replays-manual.yml (#1781 A1)
4+
# after failing every scheduled run from 2026-07-24 on. What stays here is what a schedule can
5+
# still keep honest: the device-free parser fuzz lane.
6+
37
on:
48
schedule:
59
- cron: '0 3 * * *'
@@ -99,180 +103,3 @@ jobs:
99103
done
100104
[ "$found" = 1 ] || echo 'No run envelope was produced — the lane failed before it could run.'
101105
} >> "$GITHUB_STEP_SUMMARY"
102-
103-
nightly-android:
104-
name: Android Full Emulator Suite
105-
runs-on: ubuntu-latest
106-
timeout-minutes: 15
107-
env:
108-
AGENT_DEVICE_STATE_DIR: ${{ github.workspace }}/.tmp/android-e2e-state
109-
steps:
110-
- name: Checkout
111-
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
112-
113-
- name: Setup toolchain
114-
uses: ./.github/actions/setup-node-pnpm
115-
116-
- name: Setup Android replay host
117-
id: android-replay-host
118-
uses: ./.github/actions/setup-android-replay-host
119-
with:
120-
package-helpers: 'true'
121-
122-
- name: Restore Android fixture APK
123-
id: fixture-app
124-
uses: ./.github/actions/setup-fixture-app
125-
with:
126-
platform: android
127-
install: 'false'
128-
require-artifact: 'true'
129-
130-
- name: Mark Android emulator setup complete
131-
id: android-setup
132-
run: echo "seconds=$(date +%s)" >> "$GITHUB_OUTPUT"
133-
134-
- name: Run Android full emulator suite
135-
uses: reactivecircus/android-emulator-runner@b530d96654c385303d652368551fb075bc2f0b6b # v2.35.0
136-
with:
137-
api-level: 36
138-
arch: x86_64
139-
profile: pixel_7
140-
target: google_apis_playstore
141-
emulator-options: -no-window -gpu swiftshader_indirect -no-snapshot -noaudio -no-boot-anim -no-metrics
142-
script: |
143-
# android-emulator-runner executes this with /usr/bin/sh, which is dash on
144-
# ubuntu runners and rejects `-o pipefail`. The serial assignment below is
145-
# validated by the `test -n` guard instead.
146-
set -eu
147-
ANDROID_SERIAL="$(adb devices | awk 'NR > 1 && $2 == "device" { print $1; exit }')"
148-
test -n "$ANDROID_SERIAL"
149-
BOOT_FINISHED_AT="$(date +%s)"
150-
pnpm gate build
151-
pnpm clean:daemon
152-
adb -s "$ANDROID_SERIAL" install -r "${{ steps.fixture-app.outputs.apk-path }}"
153-
node --experimental-strip-types src/bin.ts replay examples/test-app/replays/drag-android.ad --platform android --serial "$ANDROID_SERIAL" --session nightly-android-target-drag --json
154-
pnpm clean:daemon
155-
FULL_STARTED_AT="$(date +%s)"
156-
AGENT_DEVICE_ANDROID_E2E=1 AGENT_DEVICE_ANDROID_E2E_TIER=full AGENT_DEVICE_ANDROID_SERIAL="$ANDROID_SERIAL" AGENT_DEVICE_FIXTURE_APP_PATH="${{ steps.fixture-app.outputs.apk-path }}" AGENT_DEVICE_FIXTURE_APP_ID="${{ steps.fixture-app.outputs.app-id }}" node --experimental-strip-types scripts/node-test-tmpdir.ts --test test/integration/smoke-android-emulator.test.ts
157-
FULL_FINISHED_AT="$(date +%s)"
158-
echo "Android setup before emulator: $((BOOT_FINISHED_AT - ${{ steps.android-setup.outputs.seconds }}))s"
159-
echo "Android full scenario wall time: $((FULL_FINISHED_AT - FULL_STARTED_AT))s"
160-
pnpm gate replay-android --retries 2 --artifacts-dir test/artifacts/replays-android --report-junit test/artifacts/replays-android.junit.xml
161-
162-
- name: Upload Android artifacts
163-
if: always()
164-
uses: ./.github/actions/upload-agent-device-artifacts
165-
with:
166-
artifact-name: replay-nightly-android-artifacts
167-
agent-state-dir: ${{ env.AGENT_DEVICE_STATE_DIR }}
168-
169-
nightly-ios:
170-
name: iOS Replay Suite
171-
runs-on: macos-26
172-
timeout-minutes: 80
173-
env:
174-
IOS_RUNTIME_VERSION: '26.2'
175-
AGENT_DEVICE_STATE_DIR: ${{ github.workspace }}/.tmp/agent-device-state
176-
AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH: ${{ github.workspace }}/.tmp/ios-runner-derived
177-
AGENT_DEVICE_IOS_PREPARE_TIMEOUT_MS: '420000'
178-
AGENT_DEVICE_IOS_APP_EVENT_URL_TEMPLATE: agent-device-test-app:///automation?event={event}&payload={payload}
179-
steps:
180-
- name: Checkout
181-
uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd # v6.0.2
182-
183-
- name: Setup toolchain
184-
uses: ./.github/actions/setup-node-pnpm
185-
with:
186-
cache-dependency-path: |
187-
pnpm-lock.yaml
188-
examples/test-app/pnpm-lock.yaml
189-
190-
- name: Restore and build iOS XCTest runner
191-
uses: ./.github/actions/setup-apple-runner-build
192-
with:
193-
derived-path: ${{ env.AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH }}
194-
cache-key-prefix: ios-runner-prebuilt
195-
cache-key-suffix: -ios-${{ env.IOS_RUNTIME_VERSION }}
196-
gate: swift-runner-ios
197-
xcuitest-platform: ios
198-
xcuitest-destination: generic/platform=iOS Simulator
199-
200-
- name: Boot iOS test simulator
201-
id: ios-simulator
202-
uses: ./.github/actions/boot-ios-test-simulator
203-
with:
204-
runtime-version: ${{ env.IOS_RUNTIME_VERSION }}
205-
preferred-device-name: iPhone 17 Pro
206-
207-
- name: Preflight iOS runner through public CLI
208-
run: |
209-
pnpm clean:daemon
210-
node --experimental-strip-types src/bin.ts prepare ios-runner --platform ios --udid "${{ steps.ios-simulator.outputs.simulator-udid }}" --timeout "$AGENT_DEVICE_IOS_PREPARE_TIMEOUT_MS" --json
211-
pnpm clean:daemon
212-
213-
- name: Run iOS simulator replay suite
214-
uses: ./.github/actions/run-gate
215-
with:
216-
gate: replay-ios
217-
args: |
218-
--udid
219-
${{ steps.ios-simulator.outputs.simulator-udid }}
220-
--retries
221-
2
222-
--artifacts-dir
223-
test/artifacts/replays-ios-simulator
224-
--report-junit
225-
test/artifacts/replays-ios-simulator.junit.xml
226-
227-
- name: Fetch current fixture app
228-
id: fixture-app
229-
uses: ./.github/actions/setup-fixture-app
230-
with:
231-
install: 'false'
232-
233-
- name: Report fixture cache source
234-
run: |
235-
echo "Fixture app source: ${{ steps.fixture-app.outputs.source }}" >> "$GITHUB_STEP_SUMMARY"
236-
237-
- name: Prove selector drag reaches its destination on iOS
238-
run: |
239-
xcrun simctl install "${{ steps.ios-simulator.outputs.simulator-udid }}" "${{ steps.fixture-app.outputs.app-path }}"
240-
pnpm clean:daemon
241-
node --experimental-strip-types src/bin.ts replay examples/test-app/replays/drag.ad --platform ios --udid "${{ steps.ios-simulator.outputs.simulator-udid }}" --session nightly-ios-target-drag --json
242-
pnpm clean:daemon
243-
244-
- name: Run full fixture-backed iOS simulator E2E
245-
env:
246-
AGENT_DEVICE_FIXTURE_APP_ID: ${{ steps.fixture-app.outputs.app-id }}
247-
AGENT_DEVICE_FIXTURE_APP_PATH: ${{ steps.fixture-app.outputs.app-path }}
248-
AGENT_DEVICE_IOS_E2E: '1'
249-
AGENT_DEVICE_IOS_E2E_TIER: full
250-
AGENT_DEVICE_IOS_UDID: ${{ steps.ios-simulator.outputs.simulator-udid }}
251-
run: |
252-
pnpm gate build
253-
node --experimental-strip-types scripts/node-test-tmpdir.ts --test test/integration/smoke-ios-simulator-coverage.test.ts test/integration/smoke-ios-simulator.test.ts
254-
255-
- name: Run iOS physical device replay suite
256-
if: env.IOS_UDID != ''
257-
env:
258-
IOS_UDID: ${{ vars.IOS_UDID }}
259-
uses: ./.github/actions/run-gate
260-
with:
261-
gate: replay-ios-device
262-
args: |
263-
--udid
264-
${{ env.IOS_UDID }}
265-
--retries
266-
2
267-
--artifacts-dir
268-
test/artifacts/replays-ios-device
269-
--report-junit
270-
test/artifacts/replays-ios-device.junit.xml
271-
272-
- name: Upload iOS artifacts
273-
if: always()
274-
uses: ./.github/actions/upload-agent-device-artifacts
275-
with:
276-
artifact-name: replay-nightly-ios-artifacts
277-
agent-state-dir: ${{ env.AGENT_DEVICE_STATE_DIR }}
278-
runner-derived-path: ${{ env.AGENT_DEVICE_IOS_RUNNER_DERIVED_PATH }}

0 commit comments

Comments
 (0)