Skip to content

Commit 0ddbca9

Browse files
HyunwookYooclaude
andcommitted
ci: try the iOS integration tests up to three times (flutter#181771)
CI run 37637131765 on 627500e: both attempts of the ios job (jobs 112846191730 and 112864072742) hung in both test steps. Each printed "Xcode build done." and then nothing until its timeout (15 and 12 minutes). The simulator log shows Runner logging "The Dart VM service is listening on http://127.0.0.1:...", and there is no crash report. core_test.dart never runs main(), HomeScreen or the chat_app/ios_probe channel, so this is the log-reader race in flutter test (flutter/flutter#181771, fix PR #193142 still open), not the app. One retry is not enough: - the second attempt gets id tests2, an 8 minute timeout and continue-on-error; - a third attempt runs when the first two both failed or hung. It has an 8 minute timeout and no continue-on-error, so a third failure fails the job; - the ios job timeout goes from 45 to 50 minutes; - the test automation design says "up to two more attempts". Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_01G3n8VQCkYmLnMTjmfu5PUV
1 parent 627500e commit 0ddbca9

2 files changed

Lines changed: 15 additions & 4 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -211,7 +211,7 @@ jobs:
211211

212212
ios:
213213
runs-on: macos-26
214-
timeout-minutes: 45
214+
timeout-minutes: 50
215215
env:
216216
# One Rust slice (arm64) for the simulator instead of arm64 + x86_64.
217217
FLUTTER_XCODE_ARCHS: arm64
@@ -254,21 +254,32 @@ jobs:
254254
xcrun simctl bootstatus "$udid" -b
255255
# flutter test sometimes misses the VM service line in the simulator log and then waits
256256
# forever after "Xcode build done" (flutter/flutter#181771, fix proposed in #193142). The
257-
# timeout ends that hang and the next step tries once more. Drop both once Flutter has the fix.
257+
# timeouts end that hang and the next steps try up to two more times. Drop them once Flutter
258+
# has the fix.
258259
- name: Integration tests on the simulator
259260
id: tests
260261
working-directory: app
261262
timeout-minutes: 15
262263
continue-on-error: true
263264
run: flutter test integration_test/core_test.dart -d "${{ steps.sim.outputs.udid }}"
264265
- name: Integration tests on the simulator, second attempt
266+
id: tests2
265267
if: steps.tests.outcome != 'success'
266268
working-directory: app
267-
timeout-minutes: 12
269+
timeout-minutes: 8
270+
continue-on-error: true
268271
run: |
269272
echo "::warning::The first test attempt failed or hung; trying once more (flutter/flutter#181771)."
270273
xcrun simctl terminate "${{ steps.sim.outputs.udid }}" dev.chatproject.chatapp || true
271274
flutter test integration_test/core_test.dart -d "${{ steps.sim.outputs.udid }}"
275+
- name: Integration tests on the simulator, third attempt
276+
if: steps.tests.outcome != 'success' && steps.tests2.outcome != 'success'
277+
working-directory: app
278+
timeout-minutes: 8
279+
run: |
280+
echo "::warning::The second test attempt also failed or hung; trying a last time (flutter/flutter#181771)."
281+
xcrun simctl terminate "${{ steps.sim.outputs.udid }}" dev.chatproject.chatapp || true
282+
flutter test integration_test/core_test.dart -d "${{ steps.sim.outputs.udid }}"
272283
- name: The extension is embedded in the test build
273284
run: |
274285
appex="$(find app/build/ios/iphonesimulator -maxdepth 3 -name NotificationService.appex -print -quit)"

‎docs/superpowers/specs/2026-10-05-test-automation-design.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -153,7 +153,7 @@ Dart 패키지 하나다. M1a에서는 기기 관리 명령만 만들고(`bin/de
153153

154154
## 6. iOS 시뮬레이터 (M1b)
155155

156-
- macOS 러너에서 `tools/ci/pick_simulator.py`가 가장 새 iOS 런타임의 iPhone을 고른다. `xcrun simctl bootstatus <udid> -b`로 그 시뮬레이터를 켠다. `flutter test integration_test/core_test.dart -d <udid>`로 시험한다. 시뮬레이터는 앱 서명이 필요 없다. flutter/flutter#181771 때문에 시험이 멈출 수 있어서, 첫 시도가 멈추거나 실패하면 한 번 더 시도한다.
156+
- macOS 러너에서 `tools/ci/pick_simulator.py`가 가장 새 iOS 런타임의 iPhone을 고른다. `xcrun simctl bootstatus <udid> -b`로 그 시뮬레이터를 켠다. `flutter test integration_test/core_test.dart -d <udid>`로 시험한다. 시뮬레이터는 앱 서명이 필요 없다. flutter/flutter#181771 때문에 시험이 멈출 수 있어서, 앞 시도가 멈추거나 실패하면 두 번까지 더 시도한다.
157157
- `xcrun simctl push`는 알림 확장(NSE)을 실행하지 않는다. Xcode 11.4 릴리스 노트의 Known Issues(55822721)와 Xcode 14 릴리스 노트에 적혀 있고, Xcode 26.x까지 바뀌지 않았다. GitHub 러너의 시뮬레이터는 실제 APNs 토큰도 받지 못한다. 그래서 CI는 NSE가 빌드되어 앱 안(`Runner.app/PlugIns/NotificationService.appex`)에 들어갔는지만 확인한다. 복호와 메모리는 TestFlight 빌드로 잰다.
158158
- 알림 확장 메모리와 APNs 보관 개수는 시뮬레이터로 대신할 수 없다. TestFlight로 실제 iPhone에서 잰다.
159159

0 commit comments

Comments
 (0)