Skip to content

Commit 293e2af

Browse files
authored
Merge pull request #16 from noeGnh/ci/ios-xctest
ci: run the iOS integration tests through XCTest (flutter/flutter#181771)
2 parents 2778026 + b9f6cef commit 293e2af

6 files changed

Lines changed: 36 additions & 54 deletions

File tree

‎.github/scripts/ios_integration_tests.sh‎

Lines changed: 0 additions & 41 deletions
This file was deleted.

‎.github/workflows/ci.yml‎

Lines changed: 16 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -95,7 +95,7 @@ jobs:
9595
with:
9696
flutter-version: ${{ env.FLUTTER_VERSION }}
9797
cache: true
98-
- run: flutter build ios --simulator --debug
98+
- run: flutter pub get
9999
working-directory: example
100100
- name: Boot a simulator
101101
id: simulator
@@ -116,16 +116,21 @@ jobs:
116116
xcrun simctl bootstatus "$udid"
117117
flutter devices
118118
echo "udid=$udid" >> "$GITHUB_OUTPUT"
119-
# Before xcodebuild test, which shuts the simulator down when it ends. All the tests run in
120-
# one app launch (all_test.dart), with a watchdog that retries when flutter test misses the
121-
# app's start (flutter/flutter#181771).
122-
- name: Integration tests
119+
# The native RunnerTests and the integration tests (example/ios/RunnerTests/IntegrationTests.m)
120+
# in one XCTest run: flutter test often misses the app's VM service on iOS simulators and
121+
# waits forever (flutter/flutter#181771).
122+
- name: Build the tests
123+
working-directory: example
124+
run: |
125+
flutter build ios --config-only --simulator --debug integration_test/all_test.dart
126+
xcodebuild build-for-testing -workspace ios/Runner.xcworkspace -scheme Runner \
127+
-destination "id=${{ steps.simulator.outputs.udid }}" -derivedDataPath build/xctest -quiet
128+
- name: RunnerTests and integration tests
123129
timeout-minutes: 45
124-
run: ../.github/scripts/ios_integration_tests.sh ${{ steps.simulator.outputs.udid }}
125130
working-directory: example
126-
- name: RunnerTests (native unit tests)
127-
working-directory: example/ios
128-
# flutter test leaves the Xcode config pointing at its deleted test entry point.
129131
run: |
130-
flutter build ios --config-only --simulator --debug
131-
xcodebuild test -workspace Runner.xcworkspace -scheme Runner -destination "id=${{ steps.simulator.outputs.udid }}" -only-testing:RunnerTests -parallel-testing-enabled NO -quiet
132+
# The Dart tests' progress ("flutter: ...") and XCTest's results.
133+
xcodebuild test-without-building -xctestrun build/xctest/Build/Products/*.xctestrun \
134+
-destination "id=${{ steps.simulator.outputs.udid }}" -parallel-testing-enabled NO 2>&1 |
135+
grep -E 'flutter: |Test Case|Executed|error|TEST|Testing failed|timed out'
136+
exit "${PIPESTATUS[0]}"

‎README.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -120,7 +120,10 @@ Requirements: Android API 24+, iOS 15+.
120120
## Development
121121

122122
- Dart tests: `flutter test` (the editor screen runs against a fake `FilmkitPlatform` and video player, see `test/fakes.dart`).
123-
- CI (`.github/workflows/ci.yml`): format, analysis and Dart tests; Android build and Kotlin unit tests; the integration tests on an Android emulator and an iOS simulator, with the `RunnerTests`.
124-
- Native tests: `./gradlew :filmkit:testDebugUnitTest` in `example/android`; `RunnerTests` in `example/ios` (`xcodebuild test -workspace Runner.xcworkspace -scheme Runner -destination 'platform=iOS Simulator,name=<device>' -only-testing:RunnerTests`).
123+
- CI (`.github/workflows/ci.yml`): format, analysis and Dart tests; Android build and Kotlin unit tests; the integration tests on an Android emulator and, through XCTest with the `RunnerTests`, on an iOS simulator.
124+
- Native tests: `./gradlew :filmkit:testDebugUnitTest` in `example/android`; `RunnerTests` in `example/ios` (`xcodebuild test -workspace Runner.xcworkspace -scheme Runner -destination 'platform=iOS Simulator,name=<device>' -only-testing:RunnerTests/RunnerTests`: the `RunnerTests` class only).
125+
- On iOS simulators, `flutter test integration_test` often never sees the app start (flutter/flutter#181771). The RunnerTests target also runs the integration tests through XCTest (`RunnerTests/IntegrationTests.m`), which is what the CI does:
126+
1. In `example`, `flutter build ios --config-only --simulator --debug integration_test/all_test.dart`.
127+
2. `xcodebuild test` as above, with `-only-testing:RunnerTests`. The integration tests then show up as `IntegrationTests` cases.
125128
- Export tests on a device: `flutter test integration_test -d <device>` in `example`, with the sample videos of `example/assets/videos` (colored quadrants and a gray band that encodes time, a color gradient) and photos of `example/assets/photos` (a gradient, the quadrant pattern in the 8 EXIF orientations with metadata), see `example/lib/sample_videos.dart`). They compare the export and the preview with the CPU reference pixel by pixel.
126129
- On the Android emulator, add `--dart-define=EMULATOR=true`: its graphics layer converts BT.709 video frames with the BT.601 matrix, which shifts the colors of every Media3 export there (test `export keeps the source colors`).

‎example/integration_test/all_test.dart‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,5 @@
11
import 'package:flutter_test/flutter_test.dart';
2+
import 'package:integration_test/integration_test.dart';
23

34
import 'editor_test.dart' as editor;
45
import 'export_test.dart' as video_export;
@@ -8,6 +9,9 @@ import 'lut_test.dart' as lut;
89
/// Every integration test in one app launch, for CI: each test file is otherwise built and
910
/// launched separately, and an app launch on a CI simulator sometimes never connects.
1011
void main() {
12+
// Here, outside the groups: the first call registers the end-of-run report, which XCTest
13+
// (ios/RunnerTests/IntegrationTests.m) waits for; inside a group it would only cover that group.
14+
IntegrationTestWidgetsFlutterBinding.ensureInitialized();
1115
group('export', video_export.main);
1216
group('lut', lut.main);
1317
group('image', image.main);

‎example/ios/Runner.xcodeproj/project.pbxproj‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@
99
/* Begin PBXBuildFile section */
1010
1498D2341E8E89220040F4C2 /* GeneratedPluginRegistrant.m in Sources */ = {isa = PBXBuildFile; fileRef = 1498D2331E8E89220040F4C2 /* GeneratedPluginRegistrant.m */; };
1111
331C808B294A63AB00263BE5 /* RunnerTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 331C807B294A618700263BE5 /* RunnerTests.swift */; };
12+
5A1E0C3B2F4D6A8B9C0D1E2F /* IntegrationTests.m in Sources */ = {isa = PBXBuildFile; fileRef = 5A1E0C3C2F4D6A8B9C0D1E2F /* IntegrationTests.m */; };
1213
3B3967161E833CAA004F5970 /* AppFrameworkInfo.plist in Resources */ = {isa = PBXBuildFile; fileRef = 3B3967151E833CAA004F5970 /* AppFrameworkInfo.plist */; };
1314
74858FAF1ED2DC5600515810 /* AppDelegate.swift in Sources */ = {isa = PBXBuildFile; fileRef = 74858FAE1ED2DC5600515810 /* AppDelegate.swift */; };
1415
7884E8682EC3CC0700C636F2 /* SceneDelegate.swift in Sources */ = {isa = PBXBuildFile; fileRef = 7884E8672EC3CC0400C636F2 /* SceneDelegate.swift */; };
@@ -45,6 +46,7 @@
4546
1498D2321E8E86230040F4C2 /* GeneratedPluginRegistrant.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = GeneratedPluginRegistrant.h; sourceTree = "<group>"; };
4647
1498D2331E8E89220040F4C2 /* GeneratedPluginRegistrant.m */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.c.objc; path = GeneratedPluginRegistrant.m; sourceTree = "<group>"; };
4748
331C807B294A618700263BE5 /* RunnerTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = RunnerTests.swift; sourceTree = "<group>"; };
49+
5A1E0C3C2F4D6A8B9C0D1E2F /* IntegrationTests.m */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.objc; path = IntegrationTests.m; sourceTree = "<group>"; };
4850
331C8081294A63A400263BE5 /* RunnerTests.xctest */ = {isa = PBXFileReference; explicitFileType = wrapper.cfbundle; includeInIndex = 0; path = RunnerTests.xctest; sourceTree = BUILT_PRODUCTS_DIR; };
4951
3B3967151E833CAA004F5970 /* AppFrameworkInfo.plist */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = text.plist.xml; name = AppFrameworkInfo.plist; path = Flutter/AppFrameworkInfo.plist; sourceTree = "<group>"; };
5052
74858FAD1ED2DC5600515810 /* Runner-Bridging-Header.h */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.c.h; path = "Runner-Bridging-Header.h"; sourceTree = "<group>"; };
@@ -79,6 +81,7 @@
7981
isa = PBXGroup;
8082
children = (
8183
331C807B294A618700263BE5 /* RunnerTests.swift */,
84+
5A1E0C3C2F4D6A8B9C0D1E2F /* IntegrationTests.m */,
8285
);
8386
path = RunnerTests;
8487
sourceTree = "<group>";
@@ -278,6 +281,7 @@
278281
buildActionMask = 2147483647;
279282
files = (
280283
331C808B294A63AB00263BE5 /* RunnerTests.swift in Sources */,
284+
5A1E0C3B2F4D6A8B9C0D1E2F /* IntegrationTests.m in Sources */,
281285
);
282286
runOnlyForDeploymentPostprocessing = 0;
283287
};
Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
@import XCTest;
2+
@import integration_test;
3+
4+
// Runs integration_test/all_test.dart (the target of `flutter build ios --config-only`) as XCTest
5+
// cases, next to the native RunnerTests: unlike `flutter test`, it doesn't depend on the tool
6+
// finding the app's VM service, which often fails on iOS simulators (flutter/flutter#181771).
7+
INTEGRATION_TEST_IOS_RUNNER(IntegrationTests)

0 commit comments

Comments
 (0)