Skip to content

[in_app_purchase_storekit} Fix more test flakes - #13279

Merged
auto-submit[bot] merged 3 commits into
flutter:mainfrom
LouiseHsu:fix-iap-storekit2-ios27-tests
Oct 27, 2026
Merged

[in_app_purchase_storekit} Fix more test flakes #13279
auto-submit[bot] merged 3 commits into
flutter:mainfrom
LouiseHsu:fix-iap-storekit2-ios27-tests

Conversation

@LouiseHsu

@LouiseHsu LouiseHsu commented Oct 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Fixes flutter/flutter#194285 (hopefully)

The main cause of flutter/flutter#194285 (where Product.products(for:) returned [] across StoreKit 2 tests on the iOS 27 simulator) was fixed in #13213 by preventing the example app from calling StoreKit APIs before SKTestSession initialized. However, testGetProductsWithStoreKitError and testFailedNetworkErrorPurchase remains fragile in iOS 26.2+

This is still because product caching in StoreKitTest is still 1. Opaque 2. unable to be manually cleared - the product cache sticks around even after calling resetToDefaultState(). Thus, the state of the cache depends on the order that the tests will run. :/ So, if a test that purchases a product succeeds, if we later hit setSimulatedError(..., forAPI: .loadProducts), the cached product will be returned, causing the test to fail.

This is an issue that RevenueCat also ran into as shown here: RevenueCat/purchases-ios#6088, where they skip tests using .loadProduct for anything above ios 26. Instead of keeping dead tests around, I think its fair to just remove them and avoid just keeping dead code around till the end of time. Additionally, the code paths tested by the removed tests are already being hit by testFailedProductUnavilablePurchase, so we arent really losing coverage.

Additionally in this PR I fix up some leaky states between tests.

Pre-Review Checklist

  • I read the [Contributor Guide] and followed the process outlined there for submitting PRs.
  • I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools.
  • I read the [Tree Hygiene] page, which explains my responsibilities.
  • I read and followed the [relevant style guides] and ran [the auto-formatter].
  • I signed the [CLA].
  • The title of the PR starts with the name of the package surrounded by square brackets, e.g. [shared_preferences]
  • I [linked to at least one issue that this PR fixes] in the description above.
  • I followed [the version and CHANGELOG instructions], using [semantic versioning] and the [repository CHANGELOG style], or I have commented below to indicate which documented exception this PR falls under[^1].
  • I updated/added any relevant documentation (doc comments with ///).
  • I added new tests to check the change I am making, or I have commented below to indicate which [test exemption] this PR falls under[^1].
  • All existing and new tests are passing.

@flutter-dashboard flutter-dashboard Bot added the CICD Run CI/CD label Oct 15, 2026
gemini-code-assist[bot]
gemini-code-assist Bot reviewed Oct 15, 2026
View reviewed changes

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request removes the testGetProductsWithStoreKitError and testFailedNetworkErrorPurchase tests from InAppPurchaseStoreKit2PluginTests.swift, adds a line to clear simulated purchase errors, and resets the test session to its default state in StoreKit2TranslatorTests.swift. Review feedback recommends moving the simulated error cleanup and session resetting to tearDown methods in both test classes to prevent memory leaks and state pollution across tests.

Comment thread .../in_app_purchase_storekit/example/shared/RunnerTests/InAppPurchaseStoreKit2PluginTests.swift Resolved
Comment thread ..._purchase/in_app_purchase_storekit/example/shared/RunnerTests/StoreKit2TranslatorTests.swift Resolved
@jmagman

jmagman commented Oct 15, 2026

Copy link
Copy Markdown
Member

if a test that purchases a product succeeds, if we later hit setSimulatedError(..., forAPI: .loadProducts), the cached product will be returned, causing the test to fail.

Does setting setSimulatedError to nil in the tearDown work?

packages/packages/in_app_purchase/in_app_purchase_storekit/example/shared/RunnerTests/InAppPurchaseStoreKit2PluginTests.swift

Line 60 in 916d4c9

self.session.clearTransactions()

https://developer.apple.com/videos/play/wwdc2023/10140/?time=1337

To disable the simulated failure, use the same setter API and pass nil in place of the error type.

@LouiseHsu

LouiseHsu commented Oct 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

if a test that purchases a product succeeds, if we later hit setSimulatedError(..., forAPI: .loadProducts), the cached product will be returned, causing the test to fail.

Does setting setSimulatedError to nil in the tearDown work?

packages/packages/in_app_purchase/in_app_purchase_storekit/example/shared/RunnerTests/InAppPurchaseStoreKit2PluginTests.swift

Line 60 in 916d4c9

self.session.clearTransactions()

https://developer.apple.com/videos/play/wwdc2023/10140/?time=1337

To disable the simulated failure, use the same setter API and pass nil in place of the error type.

So what I think (pls correct if im wrong) what youre saying is in regards to how the old tests turned the fake error on, but never turned it off with setSimulatedError(nil, ...) and u are guessing that we just need to turn em off. The issue isnt the errors, actually.

The issue is like if ANY test like testGetProducts runs first, it will load a product like, subscription_silver normally, and Storekit will cache that product in memory for the rest of the tests. Notably, the product cache here is unaffected by any of the normal resetting methods.

Next, when testGetProductsWithStoreKitError runs later, it sets setSimulatedError(..., forAPI: .loadProducts), Product.products(for:) will just returns the cached product and skips .loadProducts entirely. Because .loadProducts is skipped, the simulated error never fires, the call succeeds instead of failing, and the test fails.

This means the test is order-dependent: it will pass if it runs before a product is cached, and it will fail if another test cached that product first.

@jmagman
jmagman removed the request for review from okorohelijah October 21, 2026 18:20
@LouiseHsu LouiseHsu added the autosubmit Merge PR when tree becomes green via auto submit App label Oct 27, 2026
@auto-submit
auto-submit Bot merged commit 559aeed into flutter:main Oct 27, 2026
13 checks passed
@engine-flutter-autoroll engine-flutter-autoroll mentioned this pull request Oct 28, 2026
Merged
flutter-zl pushed a commit to flutter-zl/flutter that referenced this pull request Oct 28, 2026
@engine-flutter-autoroll
Roll Packages from a8549ae to 13d38e0 (9 revisions) (flutter#195488)
0c2473c 
flutter/packages@a8549ae...13d38e0

2026-10-28 [email protected] [material_ui] Unskip long press drag test
on web (flutter/packages#13358)
2026-10-28 [email protected] [google_maps_flutter_ios, pigeon,
webview_flutter_wkwebview] Migrate remaining example apps to uiscene
(flutter/packages#13280)
2026-10-27 [email protected] manually roll flutter and fix flaky
video player test that is preventing roll from landing
(flutter/packages#13465)
2026-10-27 [email protected] [material_ui]
Unskip web tests that pass on CanvasKit and Skwasm: data_table_test,
floating_action_button_test, text_field_cursor_test
(flutter/packages#13460)
2026-10-27 [email protected] [material_ui] Decouple
disabled CheckboxListTile state in a11y_assessments
(flutter/packages#13177)
2026-10-27 [email protected] [camera_web] Skip video input
devices that cannot be started in availableCameras
(flutter/packages#13196)
2026-10-27 [email protected] [camera] Add support for
zero-shutter-lag capture (flutter/packages#12374)
2026-10-27 [email protected] [in_app_purchase_storekit} Fix more test
flakes (flutter/packages#13279)
2026-10-27 [email protected] [material_ui]
Refactor `typography.dart` and `text_theme.dart` in preparation for M3E
(flutter/packages#13371)

If this roll has caused a breakage, revert this CL and set the roller
to dry run mode using the controls here:
https://autoroll.skia.org/r/flutter-packages-flutter-autoroll
Please CC [email protected] on the revert to ensure that a
human
is aware of the problem.

To file a bug in Flutter:
https://github.com/flutter/flutter/issues/new/choose

To report a problem with the AutoRoller itself, please file a bug:
https://issues.skia.org/issues/new?component=1389291&template=1850622

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
suryaKommana2662 pushed a commit to suryaKommana2662/flutter that referenced this pull request Oct 29, 2026
@engine-flutter-autoroll @suryaKommana2662
Roll Packages from a8549ae to 13d38e0 (9 revisions) (flutter#195488)
0842dfd 
flutter/packages@a8549ae...13d38e0

2026-10-28 [email protected] [material_ui] Unskip long press drag test
on web (flutter/packages#13358)
2026-10-28 [email protected] [google_maps_flutter_ios, pigeon,
webview_flutter_wkwebview] Migrate remaining example apps to uiscene
(flutter/packages#13280)
2026-10-27 [email protected] manually roll flutter and fix flaky
video player test that is preventing roll from landing
(flutter/packages#13465)
2026-10-27 [email protected] [material_ui]
Unskip web tests that pass on CanvasKit and Skwasm: data_table_test,
floating_action_button_test, text_field_cursor_test
(flutter/packages#13460)
2026-10-27 [email protected] [material_ui] Decouple
disabled CheckboxListTile state in a11y_assessments
(flutter/packages#13177)
2026-10-27 [email protected] [camera_web] Skip video input
devices that cannot be started in availableCameras
(flutter/packages#13196)
2026-10-27 [email protected] [camera] Add support for
zero-shutter-lag capture (flutter/packages#12374)
2026-10-27 [email protected] [in_app_purchase_storekit} Fix more test
flakes (flutter/packages#13279)
2026-10-27 [email protected] [material_ui]
Refactor `typography.dart` and `text_theme.dart` in preparation for M3E
(flutter/packages#13371)

If this roll has caused a breakage, revert this CL and set the roller
to dry run mode using the controls here:
https://autoroll.skia.org/r/flutter-packages-flutter-autoroll
Please CC [email protected] on the revert to ensure that a
human
is aware of the problem.

To file a bug in Flutter:
https://github.com/flutter/flutter/issues/new/choose

To report a problem with the AutoRoller itself, please file a bug:
https://issues.skia.org/issues/new?component=1389291&template=1850622

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@hellohuanlin hellohuanlin hellohuanlin approved these changes
@LongCatIsLooong LongCatIsLooong Awaiting requested review from LongCatIsLooong
+1 more reviewer
@gemini-code-assist gemini-code-assist[bot] gemini-code-assist[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

autosubmit Merge PR when tree becomes green via auto submit App CICD Run CI/CD p: in_app_purchase platform-ios platform-macos

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[Xcode 27] in_app_purchase_storekit StoreKit 2 native tests fail on the iOS 27 simulator

3 participants

@LouiseHsu @jmagman @hellohuanlin