Skip to content

[in_app_purchase_storekit] Fixes Storekit 2 flaky tests - #13213

Merged
auto-submit[bot] merged 9 commits into
flutter:mainfrom
LouiseHsu:fix-iap-storekit2-flaky-tests
Oct 13, 2026
Merged

[in_app_purchase_storekit] Fixes Storekit 2 flaky tests#13213
auto-submit[bot] merged 9 commits into
flutter:mainfrom
LouiseHsu:fix-iap-storekit2-flaky-tests

Conversation

@LouiseHsu

@LouiseHsu LouiseHsu commented Oct 12, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Fixes flutter/flutter#174478
Fixes flutter/flutter#193451

  1. Increase purchase expectation timeout from 5s to 10s:
    On every StoreKit 2 purchase, iOS's storekitd daemon makes an XPC call to amsengagementd before returning the transaction. Normally amsengagementd responds in a few milliseconds (~0.2s total per test). However, on CI simulators, amsengagementd occasionally stalls while cold-starting or syncing background config bags. When that happens, storekitd waits for a hardcoded 6-second internal timeout before ignoring amsengagementd and completing the purchase (~6.5s total).
    Because our test timeout was 5s (< 6s), any test that coincided with an amsengagementd stall timed out just before StoreKit finished the purchase, and tearDown() cleared the session mid-purchase—wedging storekitd's serial transaction queue for the rest of the suite. Bumping the timeout to 10s gives StoreKit enough headroom to recover from the 6s timeout without affecting normal run times (~0.2s). Very notably, the tests that I've previously increased the time out to 10s have never flaked, so there is good evidence this is the root cause.

  2. Skip plugin registration in the host app during XCTests (NSClassFromString(@"XCTestCase") != nil):
    Because RunnerTests runs inside Runner.app as its host application, Xcode launches the example Flutter app in the background when running unit tests. Previously, the background example app registered InAppPurchasePlugin and listened to StoreKit's process-wide Transaction.updates stream, which would automatically finish test transactions behind the unit tests' backs. Skipping GeneratedPluginRegistrant when XCTestCase is loaded (matching camera_avfoundation) prevents the host app from interfering with the tests on both iOS (AppDelegate.m) and macOS (MainFlutterWindow.swift).

  3. Clean up transaction listeners in tearDown() + fail fast on errors:
    Stops the plugin's Transaction.updates listener task in tearDown() so tasks don't leak across tests, and fulfills expectations in .failure branches so unexpected errors fail immediately instead of waiting for a timeout.

Pre-Review Checklist

  • I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
  • I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools.
  • I read the [Tree Hygiene] page, which explains my responsibilities.
  • I read and followed the [relevant style guides] and ran [the auto-formatter].
  • I signed the [CLA].
  • The title of the PR starts with the name of the package surrounded by square brackets, e.g. [shared_preferences]
  • I [linked to at least one issue that this PR fixes] in the description above.
  • I followed [the version and CHANGELOG instructions], using [semantic versioning] and the [repository CHANGELOG style], or I have commented below to indicate which documented exception this PR falls under[^1].
  • I updated/added any relevant documentation (doc comments with ///).
  • I added new tests to check the change I am making, or I have commented below to indicate which [test exemption] this PR falls under[^1].
  • All existing and new tests are passing.

@LouiseHsu LouiseHsu added the CICD Run CI/CD label Oct 12, 2026
@LouiseHsu
LouiseHsu marked this pull request as ready for review October 12, 2026 23:18
@LouiseHsu LouiseHsu changed the title Fixes Storekit 2 flaky tests [in_app_purchase_storekit] Fixes Storekit 2 flaky tests Oct 12, 2026
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Oct 12, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request re-enables previously skipped flaky StoreKit 2 purchase tests by introducing a longer timeout to handle cold-starting simulators, skipping plugin registration during unit tests on iOS and macOS, and updating test cases to use explicit expectations for transaction updates. The review feedback highlights several test cases where transaction update expectations are not fulfilled in failure blocks, which would cause the tests to hang for the full timeout duration instead of failing immediately.

Comment thread .../in_app_purchase_storekit/example/shared/RunnerTests/InAppPurchaseStoreKit2PluginTests.swift Resolved
Comment thread .../in_app_purchase_storekit/example/shared/RunnerTests/InAppPurchaseStoreKit2PluginTests.swift Resolved
Comment thread .../in_app_purchase_storekit/example/shared/RunnerTests/InAppPurchaseStoreKit2PluginTests.swift Resolved
Comment thread .../in_app_purchase_storekit/example/shared/RunnerTests/InAppPurchaseStoreKit2PluginTests.swift Resolved
Comment thread .../in_app_purchase_storekit/example/shared/RunnerTests/InAppPurchaseStoreKit2PluginTests.swift Resolved
Comment thread .../in_app_purchase_storekit/example/shared/RunnerTests/InAppPurchaseStoreKit2PluginTests.swift Resolved
LongCatIsLooong
LongCatIsLooong reviewed Oct 13, 2026
View reviewed changes

@LongCatIsLooong LongCatIsLooong left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM overall just a few comments regarding having NSClassFromString(@"XCTestCase") != nil checks in the app and also the timeout.

Comment thread .../in_app_purchase_storekit/example/shared/RunnerTests/InAppPurchaseStoreKit2PluginTests.swift Outdated
final class InAppPurchase2PluginTests: XCTestCase {
// StoreKit 2's storekitd awaits AMSEngagement over XPC on every purchase, which has an
// internal ~6-7.5s timeout when amsengagementd is cold-starting on a fresh simulator.
private let purchaseTimeout: TimeInterval = 10

@LongCatIsLooong LongCatIsLooong Oct 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

According to the flutter style guide:

Never check if a port is available before using it, never add timeouts, and other race conditions.

https://github.com/flutter/flutter/blob/master/docs/contributing/Style-guide-for-Flutter-repo.md#never-check-if-a-port-is-available-before-using-it-never-add-timeouts-and-other-race-conditions

@LongCatIsLooong LongCatIsLooong Oct 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also in swift testing I think the min timeout you can set is 1 minute: https://developer.apple.com/documentation/testing/limitingexecutiontime

@LongCatIsLooong LongCatIsLooong Oct 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel we should change the timeout to at least 1 min since if we switch to swift testing there won't be a smaller timeout value.

@LouiseHsu LouiseHsu Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay yes, that makes sense to me!

Comment thread packages/in_app_purchase/in_app_purchase_storekit/example/ios/Runner/AppDelegate.m
// Plugin registration eventually starts listening to StoreKit transactions in the background,
// which would run concurrently with the test cases during unit tests and auto-finish
// transactions under test. Skip it when running unit tests.
if (NSClassFromString(@"XCTestCase") != nil) {

@LongCatIsLooong LongCatIsLooong Oct 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel having test specific logic in production code defeats the purpose of writing tests.

@LongCatIsLooong LongCatIsLooong Oct 13, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would it be possible to create a different (blank) app and tell the test target to use that as the host app instead?

@LouiseHsu LouiseHsu Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://github.com/flutter/packages/blob/main/packages/camera/camera_avfoundation/example/ios/Runner/AppDelegate.swift#L18-L25
we do this in camera already!

Also this isnt production code! this is already the standalone example app.

@LongCatIsLooong LongCatIsLooong Oct 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok it seems we can get rid of the test host app for camera_avfoundation unit tests, but for IAP it seems getting rid of the XCTest framework check takes a bit extra work so I'm ok with keeping this.

@LouiseHsu LouiseHsu added the autosubmit Merge PR when tree becomes green via auto submit App label Oct 13, 2026
@auto-submit
auto-submit Bot merged commit 772218b into flutter:main Oct 13, 2026
13 checks passed
This was referenced Oct 14, 2026
Closed
Merged
opensource-fork-sync Bot pushed a commit to cla7aye15I4nd/opensource-flutter__flutter that referenced this pull request Oct 15, 2026
@engine-flutter-autoroll
Roll Packages from a1ed23c to da40eaf (33 revisions) (flutter#194459)
b04f487 
Roll Packages from a1ed23c to da40eaf (33 revisions)

flutter/packages@a1ed23c...da40eaf

2026-10-15 [email protected] [pigeon]add support for sealed classes
in native interop (flutter/packages#13175)
2026-10-15 [email protected] [pigeon] Replace full JNI bindings with
stubs for API classes that aren't needed (flutter/packages#13173)
2026-10-14 [email protected] [material_ui] Fix
self-comparison and assert bugs in semantics_tester
(flutter/packages#12722)
2026-10-14 [email protected] [camera_avfoundation] Replace the
deprecated 'videoOrientation' with 'videoRotationAngle' on iOS 17+
(flutter/packages#13247)
2026-10-14 [email protected] [material_ui] Fix GridTile crash when the
incoming constraints are unbounded (flutter/packages#12586)
2026-10-14 [email protected] [in_app_purchase_storekit] Remove unused
and deprecated "SKDownload" hosted content (flutter/packages#13222)
2026-10-14 [email protected]
[material_ui] Add supportingTextPadding to InputDecorator
(flutter/packages#12769)
2026-10-14 [email protected]
[vector_graphics_codec] Add support for encoding and decoding paint blur
commands (flutter/packages#13148)
2026-10-14 [email protected] [go_router] Fix redirect zone async error
handling (flutter/packages#12836)
2026-10-14 [email protected] Sync release-go_router-18.1.0 to
main (flutter/packages#13260)
2026-10-14 [email protected] Sync release-cupertino_ui-1.1.3 to
main (flutter/packages#13258)
2026-10-14 [email protected] [material_ui] add enableInlinePrediction
parameter to TextFormField (flutter/packages#12793)
2026-10-14 [email protected] Sync release-material_ui-1.7.0 to
main (flutter/packages#13259)
2026-10-14 [email protected] [material_ui] Add the
Material shapes. (flutter/packages#12771)
2026-10-14 [email protected] [material_ui]
Add tertiary tap callbacks to InkResponse and InkWell
(flutter/packages#12676)
2026-10-14 [email protected] Roll Flutter from
4249ccb to 426be96 (44 revisions) (flutter/packages#13256)
2026-10-14 [email protected] [material_ui]
fixes expanded state in the a11y_assessment home page dropdown
(flutter/packages#13219)
2026-10-14 [email protected] [VPAT] Fix semantics in
a11y_assessments drawer, radio, and rail demos (flutter/packages#13159)
2026-10-14 [email protected]
[google_maps_flutter] Add onPointOfInterestTap (flutter/packages#11872)
2026-10-14 [email protected] [go_router] Fix shell route key collisions
and dynamic branch updates (flutter/packages#12549)
2026-10-14 [email protected] Manual roll Flutter from
d03768e to 4249ccb (158 revisions) (flutter/packages#13220)
2026-10-14 [email protected] [image_picker_ios] Require Flutter
3.47 / iOS 15 and drop deprecated kUTType code paths
(flutter/packages#13224)
2026-10-13 [email protected] [in_app_purchase_storekit] Fixes
Storekit 2 flaky tests (flutter/packages#13213)
2026-10-13 [email protected] [local_auth] Convert Android to new
Pigeon suspend support (flutter/packages#12894)
2026-10-13 [email protected] [material_ui] Add
enabled parameter to DropdownButtonFormField (flutter/packages#12610)
2026-10-13 [email protected] [go_router] Link ShellRoute docs to
StatefulShellRoute (flutter/packages#13120)
2026-10-13 [email protected] FadeForwards page transition
performance (flutter/packages#13161)
2026-10-13 [email protected]
[in_app_purchase_android] Guard async billing callbacks against
duplicate listener responses (flutter/packages#12640)
2026-10-13 [email protected] [google_sign_in] Remove the
Objective-C exception catcher (flutter/packages#13156)
2026-10-13 [email protected] [camera_avfoundation] Remove availability
checks below the package minimum iOS 13 (flutter/packages#13214)
2026-10-13 [email protected] [google_maps_flutter_ios] Remove iOS 14
availability check from example UI tests (flutter/packages#13218)
2026-10-13 [email protected] [google_sign_in] Convert
implementation to Kotlin (flutter/packages#13005)
2026-10-13 [email protected] [image_picker_ios] PR 2/5 Add a
PHImageManager image-data seam and migrate plugin native tests to Swift
Testing (flutter/packages#13079)

If this roll has caused a breakage, revert this CL and set the roller
to dry run mode using the controls here:
https://autoroll.skia.org/r/flutter-packages-flutter-autoroll
Please CC [email protected] on the revert to ensure that a
human
is aware of the problem.

To file a bug in Flutter:
https://github.com/flutter/flutter/issues/new/choose

To report a problem with the AutoRoller itself, please file a bug:
https://issues.skia.org/issues/new?component=1389291&template=1850622

Documentation for the AutoRoller is here:
...
@LouiseHsu LouiseHsu mentioned this pull request Oct 15, 2026
Merged
10 of 11 tasks
auto-submit Bot pushed a commit that referenced this pull request Oct 27, 2026
@LouiseHsu
[in_app_purchase_storekit} Fix more test flakes (#13279)
559aeed 
Fixes flutter/flutter#194285 (hopefully)

The main cause of flutter/flutter#194285 (where `Product.products(for:)` returned `[]` across StoreKit 2 tests on the iOS 27 simulator) was fixed in #13213 by preventing the example app from calling StoreKit APIs before `SKTestSession` initialized. However, `testGetProductsWithStoreKitError` and `testFailedNetworkErrorPurchase` remains fragile in iOS 26.2+

This is still because product caching in `StoreKitTest` is still 1. Opaque 2. unable to be manually cleared - the product cache sticks around even after calling `resetToDefaultState()`. Thus, the state of the cache depends on the order that the tests will run. :/ So, if a test that purchases a product succeeds, if we later hit `setSimulatedError(..., forAPI: .loadProducts)`, the cached product will be returned, causing the test to fail. 

This is an issue that RevenueCat also ran into as shown here: RevenueCat/purchases-ios#6088, where they skip tests using `.loadProduct` for anything above ios 26. Instead of keeping dead tests around, I think its fair to just remove them and avoid just keeping dead code around till the end of time. Additionally, the code paths tested by the removed tests are already being hit by `testFailedProductUnavilablePurchase`, so we arent really losing coverage. 

Additionally in this PR I fix up some leaky states between tests.

## Pre-Review Checklist
suryaKommana2662 pushed a commit to suryaKommana2662/flutter that referenced this pull request Oct 29, 2026
@engine-flutter-autoroll @suryaKommana2662
Roll Packages from a1ed23c to da40eaf (33 revisions) (flutter#194459)
ef6c4ac 
Roll Packages from a1ed23c to da40eaf (33 revisions)

flutter/packages@a1ed23c...da40eaf

2026-10-15 [email protected] [pigeon]add support for sealed classes
in native interop (flutter/packages#13175)
2026-10-15 [email protected] [pigeon] Replace full JNI bindings with
stubs for API classes that aren't needed (flutter/packages#13173)
2026-10-14 [email protected] [material_ui] Fix
self-comparison and assert bugs in semantics_tester
(flutter/packages#12722)
2026-10-14 [email protected] [camera_avfoundation] Replace the
deprecated 'videoOrientation' with 'videoRotationAngle' on iOS 17+
(flutter/packages#13247)
2026-10-14 [email protected] [material_ui] Fix GridTile crash when the
incoming constraints are unbounded (flutter/packages#12586)
2026-10-14 [email protected] [in_app_purchase_storekit] Remove unused
and deprecated "SKDownload" hosted content (flutter/packages#13222)
2026-10-14 [email protected]
[material_ui] Add supportingTextPadding to InputDecorator
(flutter/packages#12769)
2026-10-14 [email protected]
[vector_graphics_codec] Add support for encoding and decoding paint blur
commands (flutter/packages#13148)
2026-10-14 [email protected] [go_router] Fix redirect zone async error
handling (flutter/packages#12836)
2026-10-14 [email protected] Sync release-go_router-18.1.0 to
main (flutter/packages#13260)
2026-10-14 [email protected] Sync release-cupertino_ui-1.1.3 to
main (flutter/packages#13258)
2026-10-14 [email protected] [material_ui] add enableInlinePrediction
parameter to TextFormField (flutter/packages#12793)
2026-10-14 [email protected] Sync release-material_ui-1.7.0 to
main (flutter/packages#13259)
2026-10-14 [email protected] [material_ui] Add the
Material shapes. (flutter/packages#12771)
2026-10-14 [email protected] [material_ui]
Add tertiary tap callbacks to InkResponse and InkWell
(flutter/packages#12676)
2026-10-14 [email protected] Roll Flutter from
4249ccb to 426be96 (44 revisions) (flutter/packages#13256)
2026-10-14 [email protected] [material_ui]
fixes expanded state in the a11y_assessment home page dropdown
(flutter/packages#13219)
2026-10-14 [email protected] [VPAT] Fix semantics in
a11y_assessments drawer, radio, and rail demos (flutter/packages#13159)
2026-10-14 [email protected]
[google_maps_flutter] Add onPointOfInterestTap (flutter/packages#11872)
2026-10-14 [email protected] [go_router] Fix shell route key collisions
and dynamic branch updates (flutter/packages#12549)
2026-10-14 [email protected] Manual roll Flutter from
d03768e to 4249ccb (158 revisions) (flutter/packages#13220)
2026-10-14 [email protected] [image_picker_ios] Require Flutter
3.47 / iOS 15 and drop deprecated kUTType code paths
(flutter/packages#13224)
2026-10-13 [email protected] [in_app_purchase_storekit] Fixes
Storekit 2 flaky tests (flutter/packages#13213)
2026-10-13 [email protected] [local_auth] Convert Android to new
Pigeon suspend support (flutter/packages#12894)
2026-10-13 [email protected] [material_ui] Add
enabled parameter to DropdownButtonFormField (flutter/packages#12610)
2026-10-13 [email protected] [go_router] Link ShellRoute docs to
StatefulShellRoute (flutter/packages#13120)
2026-10-13 [email protected] FadeForwards page transition
performance (flutter/packages#13161)
2026-10-13 [email protected]
[in_app_purchase_android] Guard async billing callbacks against
duplicate listener responses (flutter/packages#12640)
2026-10-13 [email protected] [google_sign_in] Remove the
Objective-C exception catcher (flutter/packages#13156)
2026-10-13 [email protected] [camera_avfoundation] Remove availability
checks below the package minimum iOS 13 (flutter/packages#13214)
2026-10-13 [email protected] [google_maps_flutter_ios] Remove iOS 14
availability check from example UI tests (flutter/packages#13218)
2026-10-13 [email protected] [google_sign_in] Convert
implementation to Kotlin (flutter/packages#13005)
2026-10-13 [email protected] [image_picker_ios] PR 2/5 Add a
PHImageManager image-data seam and migrate plugin native tests to Swift
Testing (flutter/packages#13079)

If this roll has caused a breakage, revert this CL and set the roller
to dry run mode using the controls here:
https://autoroll.skia.org/r/flutter-packages-flutter-autoroll
Please CC [email protected] on the revert to ensure that a
human
is aware of the problem.

To file a bug in Flutter:
https://github.com/flutter/flutter/issues/new/choose

To report a problem with the AutoRoller itself, please file a bug:
https://issues.skia.org/issues/new?component=1389291&template=1850622

Documentation for the AutoRoller is here:
...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@LongCatIsLooong LongCatIsLooong LongCatIsLooong approved these changes
+1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

autosubmit Merge PR when tree becomes green via auto submit App CICD Run CI/CD p: in_app_purchase platform-ios platform-macos

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[in_app_purchase_storekit] InAppPurchase2PluginTests purchase tests flakily time out on iOS in_app_purchase iOS native plugin tests are flaky

2 participants

@LouiseHsu @LongCatIsLooong