Skip to content

Commit f0d82ae

Browse files
committed
ci(ios): run simulator integration tests through xctest
On Flutter 3.47.8 `flutter test -d <simulator>` hangs: the app starts and its VM service is listening within seconds, but the tool's log reader misses it (flutter/flutter#181771; the fix, #193142, is not in any stable release). Run the test through integration_test's iOS runner instead: a RunnerTests target, a warm-up build-for-testing, then one `xcodebuild test` per file, gated on the xcresulttool summary (passed > 0, no failures or skips). Swift Package Manager is turned off in this job so integration_test comes in through CocoaPods and RunnerTests sees it via inherit! :search_paths. The job runs only m11_ios_test.dart: the M6 files need a Brasov pack that CI does not have and would only soft-pass (README known gap #36).
1 parent ec711d5 commit f0d82ae

7 files changed

Lines changed: 1134 additions & 182 deletions

File tree

‎.github/workflows/ios.yml‎

Lines changed: 92 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -42,8 +42,10 @@ jobs:
4242
simulator:
4343
name: integration_test on iPhone 17 simulator
4444
runs-on: macos-26
45-
# About 7 min of setup, the test step's 45 and the log upload.
46-
timeout-minutes: 55
45+
# About 7 min of setup, the warm-up's 15, the test step's 10 and 5 for the
46+
# crash reports and the log upload: 37, so 90 leaves 53 of slack (kept at
47+
# 90 by Leader's R54 L1).
48+
timeout-minutes: 90
4749
env:
4850
DEVELOPER_DIR: /Applications/Xcode_26.6.app/Contents/Developer
4951
steps:
@@ -81,17 +83,49 @@ jobs:
8183
with:
8284
model: iPhone 17
8385
wait_for_boot: true
86+
# SwiftPM off for this job, before pub get writes the plugin list: then
87+
# integration_test comes in through CocoaPods like rustline_flutter, and
88+
# RunnerTests' `inherit! :search_paths` (Podfile) resolves
89+
# `@import integration_test` now that RunnerTests no longer links
90+
# FlutterGeneratedPluginSwiftPackage. Flutter leaves the pbxproj's package
91+
# reference alone and generates that package empty.
8492
- name: Pub get
8593
working-directory: rustline_flutter/example
86-
run: flutter pub get --enforce-lockfile
87-
# One file at a time, each limited to 7 min, so a hang names its file.
88-
# Flutter 3.47.8 can launch the app before its simulator log reader is
89-
# up, miss the VM Service URL and wait forever (flutter/flutter#181771,
90-
# fixed on master by flutter/flutter#193142). A file that times out
91-
# stops the run; a file that fails does not.
94+
run: |
95+
flutter config --no-enable-swift-package-manager
96+
flutter pub get --enforce-lockfile
97+
# Pod install and the full app build (Rust included) once, outside the
98+
# per-file cap; `xcodebuild test` in the loop reuses the products in the
99+
# default DerivedData. Same file as the loop.
100+
- name: Warm-up build
101+
working-directory: rustline_flutter/example
102+
timeout-minutes: 15
103+
shell: bash --noprofile --norc -eo pipefail {0}
104+
env:
105+
UDID: ${{ steps.sim.outputs.udid }}
106+
run: |
107+
set -- integration_test/m11_ios_test.dart
108+
{
109+
flutter build ios --config-only --simulator "$1" \
110+
--dart-define=RUSTLINE_M11_PACK_ZST="$PACK_ZST" \
111+
--dart-define=RUSTLINE_M11_REQUIRE_PACK=true
112+
xcodebuild build-for-testing -workspace ios/Runner.xcworkspace -scheme Runner \
113+
-configuration Debug -destination "platform=iOS Simulator,id=$UDID" \
114+
CODE_SIGNING_ALLOWED=NO
115+
} 2>&1 | tee "$LOGS/warmup.log"
116+
# Only m11_ios_test.dart: the M6 files (rustline_smoke, lifecycle,
117+
# pack_safety, panic, perf) need a Brașov pack with streets that CI does
118+
# not have, and without it they only soft-pass (known gap, R54 H2).
119+
# One file at a time, each limited to 7 min, so a hang names its file:
120+
# 1 file x 7 min plus 3.
121+
# Runs through XCTest (integration_test's iOS runner, RunnerTests.m), not
122+
# `flutter test -d`: Flutter 3.47.8 can launch the app before its
123+
# simulator log reader is up, miss the VM Service URL and wait forever
124+
# (flutter/flutter#181771, fixed on master by flutter/flutter#193142).
125+
# A file that times out stops the run; a file that fails does not.
92126
- name: Integration tests
93127
working-directory: rustline_flutter/example
94-
timeout-minutes: 45
128+
timeout-minutes: 10
95129
shell: bash --noprofile --norc -eo pipefail {0}
96130
env:
97131
UDID: ${{ steps.sim.outputs.udid }}
@@ -128,18 +162,62 @@ jobs:
128162
exit(($? & 127) ? 128 + ($? & 127) : $? >> 8);
129163
' "$@"
130164
}
165+
# One file: point the Xcode project at it (FLUTTER_TARGET and the
166+
# defines go into Generated.xcconfig; pod install), then build and
167+
# run it through XCTest. One XCTest case per Dart test; a failing
168+
# Dart test fails its case and xcodebuild.
169+
test_file() {
170+
flutter build ios --config-only --simulator integration_test/"$1".dart \
171+
--dart-define=RUSTLINE_M11_PACK_ZST="$PACK_ZST" \
172+
--dart-define=RUSTLINE_M11_REQUIRE_PACK=true &&
173+
xcodebuild test -workspace ios/Runner.xcworkspace -scheme Runner \
174+
-configuration Debug -destination "platform=iOS Simulator,id=$UDID" \
175+
-parallel-testing-enabled NO -resultBundlePath "$LOGS/$1.xcresult" \
176+
CODE_SIGNING_ALLOWED=NO
177+
}
178+
export -f test_file
179+
# XCTest only reports the Dart tests whose body ran, and a run with
180+
# none passes, so a file passes only if its result bundle exists and
181+
# counts passed tests and no failed or skipped ones. The summary is
182+
# kept as $LOGS/<name>.summary.json.
183+
check_summary() {
184+
local xr="$LOGS/$1.xcresult" js="$LOGS/$1.summary.json"
185+
if [ ! -d "$xr" ]; then
186+
echo "::error::$1: no result bundle ($xr)"
187+
return 1
188+
fi
189+
if ! xcrun xcresulttool get test-results summary --path "$xr" > "$js"; then
190+
echo "::error::$1: xcresulttool could not read $xr"
191+
return 1
192+
fi
193+
cat "$js"
194+
if ! python3 - "$js" <<'PY'
195+
import json, sys
196+
s = json.load(open(sys.argv[1]))
197+
p, f, k = (s.get(n) for n in ("passedTests", "failedTests", "skippedTests"))
198+
print(f"passed={p} failed={f} skipped={k}")
199+
sys.exit(0 if isinstance(p, int) and p > 0 and f == 0 and k == 0 else 1)
200+
PY
201+
then
202+
echo "::error::$1: needs passedTests > 0, failedTests 0, skippedTests 0"
203+
return 1
204+
fi
205+
}
131206
# The files still to run.
132-
set -- integration_test/*_test.dart
207+
set -- integration_test/m11_ios_test.dart
133208
failed=""
134-
for f in integration_test/*_test.dart; do
209+
# shellcheck disable=SC2043 # one file for now (see above).
210+
for f in integration_test/m11_ios_test.dart; do
135211
name=$(basename "$f" .dart)
136212
echo "::group::$f"
137213
set +e
138-
run_with_timeout 420 flutter test integration_test/"$name".dart -d "$UDID" --verbose --reporter expanded \
139-
--dart-define=RUSTLINE_M11_PACK_ZST="$PACK_ZST" \
140-
--dart-define=RUSTLINE_M11_REQUIRE_PACK=true 2>&1 | tee "$LOGS/$name.log"
214+
# shellcheck disable=SC2016 # $1 is the child bash's argument.
215+
run_with_timeout 420 bash -c 'test_file "$1"' _ "$name" 2>&1 | tee "$LOGS/$name.log"
141216
rc=${PIPESTATUS[0]}
142217
set -e
218+
if [ "$rc" -ne 124 ] && ! check_summary "$name" && [ "$rc" -eq 0 ]; then
219+
rc=1
220+
fi
143221
echo "::endgroup::"
144222
shift
145223
if [ "$rc" -eq 124 ]; then

‎README.md‎

Lines changed: 6 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -133,7 +133,7 @@ flutter run -d "iPhone 17" # or any booted arm64 simulator
133133

134134
Install packs into the directory that `defaultPackDirectory()` returns: `packs/` under Application Support, created if missing. On iOS that directory is excluded from iCloud backup, because a pack can be downloaded again. If the exclusion fails, `defaultPackDirectory()` throws the `PlatformException` instead of returning a directory that would be backed up. Call it on every launch instead of saving the path, because the iOS container path changes between installs.
135135

136-
To run the iOS integration tests on a simulator with a real pack, as `ios.yml` does:
136+
To run the iOS integration tests on a simulator with the same pack and defines as `ios.yml`:
137137

138138
```bash
139139
cargo run --locked --release -p rustline-cli -- build-pack --gtfs data/fixtures/tiny --out /tmp/pack --feed-id tiny
@@ -143,6 +143,8 @@ flutter test integration_test -d "iPhone 17" \
143143
--dart-define=RUSTLINE_M11_REQUIRE_PACK=true
144144
```
145145

146+
On Flutter 3.47.8, `flutter test -d <simulator>` can hang before the first test ([flutter/flutter#181771](https://github.com/flutter/flutter/issues/181771)). `ios.yml` therefore runs the file through XCTest with `xcodebuild test`.
147+
146148
## Usage
147149

148150
### CLI
@@ -341,7 +343,7 @@ cargo test --locked --release -p rustline-osm --test m4_osm -- --ignored --nocap
341343

342344
The Flutter plugin's integration tests run on an x86_64 android-34 emulator (AVD `rustline_x64`) locally, or in CI with `android-emulator.yml`. See [Continuous integration](#continuous-integration).
343345

344-
The iOS integration tests run on an iPhone 17 simulator in `ios.yml` (see [Building on a Mac (iOS)](#building-on-a-mac-ios)). Without a Mac, `scripts/m11-ios-check.sh` runs the checks that work on Linux: actionlint on `ios.yml`, the static M11 config tests, `cargo check` and clippy of `rustline-ffi` for both iOS targets (Rust code only; C code goes through a stub compiler and nothing links), and the plugin's storage unit tests and `flutter analyze`.
346+
`ios.yml` runs `integration_test/m11_ios_test.dart` on an iPhone 17 simulator through XCTest (see [Building on a Mac (iOS)](#building-on-a-mac-ios)). Without a Mac, `scripts/m11-ios-check.sh` runs the checks that work on Linux: actionlint on `ios.yml`, the static M11 config tests, `cargo check` and clippy of `rustline-ffi` for both iOS targets (Rust code only; C code goes through a stub compiler and nothing links), and the plugin's storage unit tests and `flutter analyze`.
345347

346348
## Continuous integration
347349

@@ -352,7 +354,7 @@ GitHub Actions workflows in `.github/workflows/`:
352354
| `ci.yml` (CI) | push to `main`, every pull request, manual | `rust`: `cargo fmt --check`, `clippy -D warnings`, tests debug + release (stable); `msrv`: tests on Rust 1.88; `flutter`: `flutter analyze` (plugin + example), plugin `flutter test`, and a check that `flutter_rust_bridge_codegen generate` (2.13.0) leaves no diff |
353355
| `android.yml` (Android) | push to `main`, every pull request, manual | `ffi`: `cargo ndk` release build of `rustline-ffi` for arm64-v8a and x86_64 on Rust 1.88, `.so` size in the job summary; `apk`: example `flutter build apk --release`, APK and `.so` sizes in the summary |
354356
| `android-emulator.yml` | manual only | example `integration_test` on an x86_64 API 34 emulator; pack-based tests print `skip:` (no pack) |
355-
| `ios.yml` (iOS) | pull requests and pushes to `main` that touch `rustline_flutter/`, `crates/`, Cargo files, `rust-toolchain.toml` or the tiny fixture; manual | macOS 26, Xcode 26.6. `simulator`: example `integration_test` on an iPhone 17 simulator with the tiny pack; `device`: `flutter build ios --release --no-codesign` of the example and of a blank app, sizes in the job summary and the `ios-size` artifact; `ffi`: `cargo test -p rustline-ffi` on macOS |
357+
| `ios.yml` (iOS) | pull requests and pushes to `main` that touch `rustline_flutter/`, `crates/`, Cargo files, `rust-toolchain.toml` or the tiny fixture; manual | macOS 26, Xcode 26.6. `simulator`: only `integration_test/m11_ios_test.dart`, with the tiny pack, on an iPhone 17 simulator, run through XCTest (`RunnerTests` with integration_test's iOS runner, one `xcodebuild test` per file) instead of `flutter test -d`, which hangs on Flutter 3.47.8 ([flutter/flutter#181771](https://github.com/flutter/flutter/issues/181771); the fix, [flutter/flutter#193142](https://github.com/flutter/flutter/pull/193142), is not in a stable release yet). A file passes only if its xcresulttool summary has passed > 0 and failed = skipped = 0. Swift Package Manager is off in this job, so plugins come in through CocoaPods; `device`: `flutter build ios --release --no-codesign` of the example and of a blank app, sizes in the job summary and the `ios-size` artifact; `ffi`: `cargo test -p rustline-ffi` on macOS |
356358
| `real-feeds.yml` | manual, and Mondays 04:00 UTC | fetches and verifies real feeds, then `cargo test --workspace --release` with them present |
357359
| `bench.yml` | manual only | `cargo bench -p rustline-core --bench plan` (spot checks; shared runners are noisy) |
358360

@@ -439,6 +441,7 @@ GitHub Actions workflows in `.github/workflows/`:
439441
33. **No real iPhone** – tests run on a simulator; device builds are unsigned (`--no-codesign`) and have not been installed on a phone.
440442
34. **Slices and linkage** – arm64 only (no x86_64 simulator), and only dynamic frameworks (`use_frameworks!`); static linkage fails at `RustLib.init()`.
441443
35. **Stable Rust only on iOS** – `ios.yml` uses the stable toolchain; the iOS targets are not tested on the MSRV (1.88).
444+
36. **M6 integration tests need a Brașov pack** – the pack-based tests in `rustline_smoke`, `rustline_lifecycle`, `rustline_pack_safety`, `rustline_panic` and `rustline_perf` print `skip:` and return when `RUSTLINE_M6_PACK` (or `RUSTLINE_M6_PACK_ZST`) is unset, and still count as passed. CI has no feeds, so on `android-emulator.yml` they pass without really running, and the iOS simulator job does not run them at all.
442445

443446
## Similar projects
444447

0 commit comments

Comments
 (0)