Refactor iOS integration_test API to support Swift, dynamically add native tests by jmagman · Pull Request #88013 · flutter/flutter

jmagman · 2021-08-11T02:20:24Z

Change the iOS native integration_tests to run the dart tests, and dynamically add the test results so that each dart test appears as a native iOS test.

Add a FLTIntegrationTestCase to become the XCTestCase base class, removing the need for the Swift-incompatible INTEGRATION_TEST_IOS_RUNNER macro. This class swizzles a new class instance method per completed dart test.
Create a screenshot placeholder test for any captureScreenshot requests over the channel. Unfortunately there's no mapping to know which test it came from (yet) so they are all grouped together.
Replace FLTIntegrationTestScreenshotDelegate with the property IntegrationTestPlugin.capturedScreenshotsByName. I was seeing issues where only the first FLTIntegrationTestCase was able to see the screenshots because new FLTIntegrationTestRunner delegates were being created every time. Instead, store it on the static +[IntegrationTestPlugin instance] so it's accessible from different FLTIntegrationTestRunners.
Add a RunnerSwiftTests.swift test file to prove it works in a Swift test.
Update the README.

Success (note there are three instances of the tests because I created FLTIntegrationTestCase to test different scenarios):

Failure:

Next I need to actually run the integration test during CI: #87770

Fixes #71873

flutter-dashboard · 2021-08-11T02:20:26Z

It looks like this pull request may not have tests. Please make sure to add tests before merging. If you need an exemption to this rule, contact Hixie on the #hackers channel in Chat.

If you are not sure if you need tests, consider this rule of thumb: the purpose of a test is to make sure someone doesn't accidentally revert the fix. Ask yourself, is there anything in your PR that you feel it is important we not accidentally revert back to how it was before your fix?

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

jmagman · 2021-08-11T02:29:45Z

These build settings were automatically changed when I added a Swift file.

jmagman · 2021-08-11T02:30:29Z

Include the RunnerTests target during analysis.

jmagman · 2021-08-11T02:30:57Z

Updated and moved from IntegrationTestIosTest.h

jmagman · 2021-08-11T02:31:31Z

FLTIntegrationTestCase only works from a XCTest target (not from the app itself).

jmagman · 2021-08-11T04:23:00Z

@stuartmorgan This refactor was larger than I was expecting... #84611 (comment)

jmagman · 2021-08-11T17:37:56Z

\cc @cbracken this shouldn't be too bad to also implement for macOS. The only rough edge I can think of is UIImage instead of NSImage.

stuartmorgan-g

LGTM!

stuartmorgan-g · 2021-08-13T16:02:15Z

This is slick! Out of curiosity, do we know if it's guaranteed to work? It seems a bit modifying-an-iterator-while-iterating, which makes me wonder if it has defined behavior.

That's actually the purpose of + (NSArray<NSInvocation *> *)testInvocations--a hook for dynamically changing the test methods to use for a test case. https://stackoverflow.com/a/55204082 was a helpful reference.

stuartmorgan-g · 2021-08-13T16:05:29Z

Any chance we could fix the casing of "IOS" while we're here, or is that breaking to a public API?

That's a good idea, and there's nothing iOS-y about it other than the UIImage. I'll deprecate the entire IntegrationTestIosTest instead of just testIntegrationTest: and replace it with a FLTIntegrationTestRunner class. It would be the same result if anyone was using this API (which the docs don't say to do, but it was technically possible).

jmagman · 2021-08-16T22:42:08Z

+// Test deprecated macro. Do not use.
+INTEGRATION_TEST_IOS_RUNNER(RunnerObjCMacroTests)
+
+@interface DeprecatedIntegrationTestIosTests : XCTestCase


Just added this, validates the deprecated IntegrationTestIosTest still works:

Test Suite 'DeprecatedIntegrationTestIosTests' started at 2021-08-16 15:40:22.353 Test Case '-[DeprecatedIntegrationTestIosTests testIntegrationTest]' started. 2021-08-16 15:40:22.355741-0700 Runner[87799:71056731] ==================== Test Results ===================== 2021-08-16 15:40:22.355888-0700 Runner[87799:71056731] verify text passed. 2021-08-16 15:40:22.355992-0700 Runner[87799:71056731] verify screenshot passed. 2021-08-16 15:40:22.356089-0700 Runner[87799:71056731] ================== Test Results End ==================== Test Case '-[DeprecatedIntegrationTestIosTests testIntegrationTest]' passed (0.004 seconds).

stuartmorgan-g

LGTM

…ly add native tests (#88013)" This reverts commit 9e3de9a.

…ly add native tests (#88013)" (#88377) This reverts commit 9e3de9a.

…ative tests (flutter#88013)

…ly add native tests (flutter#88013)" (flutter#88377) This reverts commit 9e3de9a.

jmagman added platform-ios iOS applications specifically f: integration_test The flutter/packages/integration_test plugin labels Aug 11, 2021

jmagman self-assigned this Aug 11, 2021

google-cla Bot added the cla: yes label Aug 11, 2021

jmagman force-pushed the flat-integration-test-api branch from fa301a9 to 090b9b0 Compare August 11, 2021 02:26

jmagman changed the title ~~Refactor iOS integration_test API to support Swift~~ Refactor iOS integration_test API to support Swift, dynamically add native tests Aug 11, 2021

jmagman commented Aug 11, 2021

View reviewed changes

jmagman force-pushed the flat-integration-test-api branch from 090b9b0 to 0d2b4c9 Compare August 11, 2021 02:33

jmagman requested review from dnfield and stuartmorgan-g August 11, 2021 04:22

stuartmorgan-g approved these changes Aug 13, 2021

View reviewed changes

jmagman added 2 commits August 13, 2021 12:56

Refactor iOS integration_test API to support Swift

8d00de4

Deprecate IntegrationTestIosTest

2488f62

jmagman force-pushed the flat-integration-test-api branch from 0d2b4c9 to 2488f62 Compare August 16, 2021 22:33

jmagman added 2 commits August 16, 2021 15:35

Remove unnecessary @testable

3ee9023

Add DeprecatedIntegrationTestIosTests test

490fd1a

jmagman commented Aug 16, 2021

View reviewed changes

stuartmorgan-g approved these changes Aug 17, 2021

View reviewed changes

jmagman added the waiting for tree to go green label Aug 17, 2021

fluttergithubbot merged commit 9e3de9a into flutter:master Aug 17, 2021

jmagman deleted the flat-integration-test-api branch August 17, 2021 18:42

zanderso added a commit that referenced this pull request Aug 17, 2021

Revert "Refactor iOS integration_test API to support Swift, dynamical…

45df84c

…ly add native tests (#88013)" This reverts commit 9e3de9a.

zanderso mentioned this pull request Aug 17, 2021

Revert "Refactor iOS integration_test API to support Swift, dynamically add native tests" #88377

Merged

jmagman pushed a commit that referenced this pull request Aug 17, 2021

Revert "Refactor iOS integration_test API to support Swift, dynamical…

463f973

…ly add native tests (#88013)" (#88377) This reverts commit 9e3de9a.

jmagman mentioned this pull request Aug 18, 2021

Run plugin_lint_mac task when either flutter_tools or integration_test packages change #88396

Merged

blasten pushed a commit to blasten/flutter that referenced this pull request Aug 19, 2021

Refactor iOS integration_test API to support Swift, dynamically add n…

2a7472f

…ative tests (flutter#88013)

blasten pushed a commit to blasten/flutter that referenced this pull request Aug 19, 2021

Revert "Refactor iOS integration_test API to support Swift, dynamical…

dfe783e

…ly add native tests (flutter#88013)" (flutter#88377) This reverts commit 9e3de9a.

HelloCore mentioned this pull request Aug 27, 2021

[ios] [integration_test] avoid using preprocessor macros to make it easier to use from Swift #71873

Open

jmagman mentioned this pull request Dec 22, 2021

Native iOS tests should generate a test per dart test with separate failure statuses and error messages instead of one "testIntegrationTest" #95648

Closed

jmagman restored the flat-integration-test-api branch December 22, 2021 19:49

jmagman mentioned this pull request Dec 23, 2021

Dynamically add integration_tests and screenshots to native iOS test results #95704

Merged

8 tasks

jmagman deleted the flat-integration-test-api branch December 23, 2021 04:52

jmagman mentioned this pull request Oct 22, 2024

Strip plugins created from dev_dependencies from release-mode apps #56591

Closed

smallTrogdor mentioned this pull request Mar 19, 2026

Unable to complete any iOS UI tests that lasts longer than ~6 minutes #145143

Open

Uh oh!

Conversation

jmagman commented Aug 11, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flutter-dashboard Bot commented Aug 11, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jmagman commented Aug 11, 2021

Uh oh!

jmagman commented Aug 11, 2021

Uh oh!

stuartmorgan-g left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

stuartmorgan-g left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jmagman commented Aug 11, 2021 •

edited

Loading