Skip to content

Commit 31bd0fb

Browse files
dkwingsmtDanTup
authored andcommitted
Allow resetting test invariants in addTearDown (flutter#192082)
Fixes flutter#110488. ### Background: The `testWidgets` Lifecycle Understanding this change requires distinguishing between two layers of `Future`s and two distinct verification steps in `testWidgets`: #### 1. Two Layers of `Future`s * **`testBody()` Future**: The closure containing the user's test logic. * **`runTest()` Future**: The overall async harness returned by `binding.runTest()`. It manages the test zone, unmounts the widget tree, and resolves only after `testBody()` and test-specific verifications finish. #### 2. Two Distinct Verification Phases * **Test-specific verifications (`invariantTester`)**: Passed as a callback to `runTest` and executed synchronously inside `_runTestBody` immediately after the `testBody()` Future completes and the widget tree is unmounted. In `testWidgets`, this runs `tester._endOfTestVerifications` to verify that tickers and semantics handles were disposed. * **Framework-level invariant checks**: Verifies global framework state, including ensuring foundation/rendering/widgets debug flags were reset and no non-periodic timers or animations remain. #### 3. The Teardown Lifecycle & Root Cause of flutter#110488 When `testWidgets` runs: 1. `testWidgets` registers `test_package.addTearDown(binding.postTest)` with `package:test`. 2. `testWidgets` calls and returns `binding.runTest(...)`. 3. Inside `testBody()`, users may register cleanups using `addTearDown` (for example, `addTearDown(() => debugDefaultTargetPlatformOverride = null)`). 4. `package:test` executes teardowns in **LIFO (reverse registration) order** only *after* the Future returned by `runTest` completes. Previously, framework-level invariant checks ran at the end of `_runTestBody` *before* the `runTest` Future completed. As a result, they ran before `package:test` executed any user-registered `addTearDown` callbacks, causing tests that clean up debug flags via `addTearDown` to fail invariant verification (flutter#110488). The following diagram shows the code flow after this PR: ``` testWidgets(description, (tester) async { ... }) │ ├── 1. Registration Phase │ └── test_package.addTearDown(binding.postTest) [Teardown Stack: #1] │ ├── 2. Test Execution Phase: binding.runTest(...) │ │ │ │ [ Future 1: testBody() ] │ ├──► Run testBody(tester) │ │ └── User calls addTearDown(...) [Teardown Stack: #2] │ │ │ │ [ Post-testBody cleanup (inside runTest) ] │ ├──► Unmount widget tree (runApp) │ ├──► Verification 1: invariantTester() (tickers, semantics) │ └──► Set _shouldVerifyInvariants = true │ │ [ Future 2: runTest() completes ] │ └── 3. Teardown Phase: package:test executes in LIFO order │ ├──► [Step 1] User Teardowns (Stack #2) │ └── e.g., debugDefaultTargetPlatformOverride = null │ └──► [Step 2] binding.postTest() (Stack #1) ├── Verification 2: Framework Invariants (debug vars, timers) ├── Reset Binding State (_currentFakeAsync, etc.) └── reportTestException(...) (if Verification 2 failed) ``` ### Changes & Design Rationale #### 1. Defer Framework Invariant Checks to `postTest` Framework invariant checks (`_verifyInvariants()`, `_verifyAutoUpdateGoldensUnset`, etc.) are moved from `_runTestBody` into `TestWidgetsFlutterBinding.postTest`. Because `testWidgets` registers `binding.postTest` before running the test body, `package:test`'s LIFO teardown ordering guarantees that all user-registered `addTearDown` callbacks execute before `binding.postTest`. #### 2. Clarifying the `postTest` Error Contract Previously, `postTest` only performed state cleanup and did not check invariants. Moving invariant checks into `postTest` introduced the question of how invariant failures should be propagated: allowing `postTest` to throw, or absorbing errors and reporting them via `reportTestException`. We explicitly define the contract of `postTest` to **not throw exceptions**, reporting any invariant failures via `reportTestException`: * **Rationale & Trade-offs**: * *Alternative considered (throwing)*: If `super.postTest()` threw on an invariant failure, `package:test` would still capture the failure. However, subclasses (`AutomatedTestWidgetsFlutterBinding`, `LiveTestWidgetsFlutterBinding`, or custom bindings) override `postTest` to reset their own state (e.g. nulling `_currentFakeAsync` / `_clock`, or setting `_inTest = false`). Allowing `super.postTest()` to throw would require every subclass to defensively wrap `super.postTest()` in `try ... finally`. If any subclass omitted this, an invariant failure would leave the binding in a dirty state and trigger cascading `!inTest` failures in subsequent tests. * *Chosen approach (non-throwing)*: Absorbing errors in `postTest` and forwarding them to `reportTestException` guarantees that all cleanup in base and subclass implementations executes sequentially without defensive `try ... finally` boilerplate, while still cleanly failing the test. **Why this does not break existing behavior:** In the previous implementation, when invariant assertions failed inside `_runTestBody`, they were caught by `_testZone.handleUncaughtError` and forwarded to `reportTestException` (which delegates to `package:test.registerException`). They never escaped `runTest` as unhandled thrown exceptions. Catching invariant failures in `postTest` and routing them to `reportTestException` preserves this exact error reporting pipeline, diagnostic output, and failure semantics without disrupting teardown execution. #### 3. Preserving Existing Behavior on Failed Tests (`_shouldVerifyInvariants`) Historically, `_runTestBody` only verified invariants if the test body did not fail: ```dart if (_pendingExceptionDetails == null) { // We only try to clean up and verify invariants if we didn't already // fail. If we got an exception already, then we instead leave everything // alone so that we don't cause more spurious errors. ... _verifyInvariants(); } ``` When invariant checks are moved to `postTest`, `_pendingExceptionDetails` is no longer available because it has already been reported and cleared by `testCompletionHandler` when `runTest` completed. To preserve this exact existing behavior across the lifecycle boundary, `_shouldVerifyInvariants` is introduced. It is set to `true` at the end of `_runTestBody` only when `_pendingExceptionDetails == null`. If a test fails in its body, `_shouldVerifyInvariants` remains `false`, and `postTest` skips invariant checks—ensuring no behavior change and preventing spurious errors (e.g. active animations on an unmounted tree) from masking the real failure. #### 4. Documentation Updates Updated the doc comments on: * `TestWidgetsFlutterBinding.postTest`: Documents the non-throwing contract, the `@mustCallSuper` requirement, and error reporting via `reportTestException`. * `TestWidgetsFlutterBinding.runTest`: Clarifies the distinction between `invariantTester` (which runs before `runTest` completes) and framework invariant checks (which run in `postTest` after `runTest` and user teardowns complete). ### Tests - Added unit tests in `packages/flutter_test/test/bindings_invariants_test.dart` verifying that foundation variables (`debugDefaultTargetPlatformOverride`, `debugDoublePrecision`, `debugBrightnessOverride`), `autoUpdateGoldenFiles`, `ErrorWidget.builder`, and `shouldPropagateDevicePointerEvents` can all be reset using `addTearDown`. - Added test cases verifying invariant failures in `postTest` and ensuring invariant checks are skipped when the test body throws. ## Pre-launch Checklist - [ ] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [ ] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [ ] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [ ] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [ ] I signed the [CLA]. - [ ] I listed at least one issue that this PR fixes in the description above. - [ ] I updated/added relevant documentation (doc comments with `///`). - [ ] I added new tests to check the change I am making, or this PR is [test-exempt]. - [ ] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [ ] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#ai-contribution-guidelines [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
1 parent 1e9a2ce commit 31bd0fb

3 files changed

Lines changed: 275 additions & 44 deletions

File tree

‎packages/flutter_test/lib/src/binding.dart‎

Lines changed: 98 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -1688,8 +1688,15 @@ abstract class TestWidgetsFlutterBinding extends BindingBase
16881688
/// Called by the [testWidgets] and [benchmarkWidgets] functions to
16891689
/// run a test.
16901690
///
1691-
/// The `invariantTester` argument is called after the `testBody`'s [Future]
1692-
/// completes. If it throws, then the test is marked as failed.
1691+
/// The `invariantTester` argument is called inside this method immediately
1692+
/// after the `testBody` completes and the widget tree is unmounted (for
1693+
/// example, to verify that tickers and semantics handles were disposed).
1694+
///
1695+
/// This should not be confused with framework-level invariant checks (such as
1696+
/// verifying that debug flags were reset and pending timers disposed) and
1697+
/// binding state cleanups, which are performed in [postTest] after the
1698+
/// [Future] returned by this method completes and after any per-test
1699+
/// `addTearDown` callbacks have executed.
16931700
///
16941701
/// The `description` is used by the [LiveTestWidgetsFlutterBinding] to
16951702
/// show a label on the screen during the test. The description comes from
@@ -1944,18 +1951,18 @@ abstract class TestWidgetsFlutterBinding extends BindingBase
19441951
assert(inTest);
19451952
// So that we can assert that it remains the same after the test finishes.
19461953
_beforeTestCheckIntrinsicSizes = debugCheckIntrinsicSizes;
1954+
_beforeTestAutoUpdateGoldens = autoUpdateGoldenFiles && !isBrowser;
1955+
_beforeTestReportTestException = reportTestException;
1956+
_beforeTestErrorWidgetBuilder = ErrorWidget.builder;
1957+
_beforeTestShouldPropagateDevicePointerEvents = shouldPropagateDevicePointerEvents;
1958+
_shouldVerifyInvariants = false;
19471959

19481960
runApp(Container(key: UniqueKey(), child: _preTestMessage)); // Reset the tree to a known state.
19491961
await pump();
19501962
// Pretend that the first frame produced in the test body is the first frame
19511963
// sent to the engine.
19521964
resetFirstFrameSent();
19531965

1954-
final bool autoUpdateGoldensBeforeTest = autoUpdateGoldenFiles && !isBrowser;
1955-
final TestExceptionReporter reportTestExceptionBeforeTest = reportTestException;
1956-
final ErrorWidgetBuilder errorWidgetBuilderBeforeTest = ErrorWidget.builder;
1957-
final bool shouldPropagateDevicePointerEventsBeforeTest = shouldPropagateDevicePointerEvents;
1958-
19591966
// run the test
19601967
await testBody();
19611968
asyncBarrier(); // drains the microtasks in `flutter test` mode (when using AutomatedTestWidgetsFlutterBinding)
@@ -1974,19 +1981,57 @@ abstract class TestWidgetsFlutterBinding extends BindingBase
19741981
_testTextInput.unregister();
19751982
}
19761983
invariantTester();
1977-
_verifyAutoUpdateGoldensUnset(autoUpdateGoldensBeforeTest && !isBrowser);
1978-
_verifyReportTestExceptionUnset(reportTestExceptionBeforeTest);
1979-
_verifyErrorWidgetBuilderUnset(errorWidgetBuilderBeforeTest);
1980-
_verifyShouldPropagateDevicePointerEventsUnset(shouldPropagateDevicePointerEventsBeforeTest);
1981-
_verifyInvariants();
1984+
_verifyPostPumpInvariants();
1985+
_shouldVerifyInvariants = true;
19821986
}
19831987

19841988
assert(inTest);
19851989
asyncBarrier(); // When using AutomatedTestWidgetsFlutterBinding, this flushes the microtasks.
19861990
}
19871991

1992+
bool _beforeTestAutoUpdateGoldens = false;
1993+
late TestExceptionReporter _beforeTestReportTestException;
1994+
late ErrorWidgetBuilder _beforeTestErrorWidgetBuilder;
1995+
bool _beforeTestShouldPropagateDevicePointerEvents = false;
19881996
late bool _beforeTestCheckIntrinsicSizes;
19891997

1998+
// Whether post-test invariant verifications should run in [postTest].
1999+
//
2000+
// Set to true only if the test body completed without exceptions and the
2001+
// widget tree was unmounted. If the test encountered an exception, invariant
2002+
// checks are skipped to avoid spurious errors (e.g., active animations or
2003+
// unreset debug flags from aborted tests) from obscuring the real failure.
2004+
bool _shouldVerifyInvariants = false;
2005+
2006+
/// Verifies invariants that must hold immediately after the widget tree is
2007+
/// unmounted and pumped at the end of the test body, before teardowns run.
2008+
///
2009+
/// This is an exceptional hook called inside `_runTestBody` right after
2010+
/// the widget tree is replaced with `_postTestMessage` and pumped. Almost all
2011+
/// invariant checks belong in `_verifyInvariants` instead of here.
2012+
///
2013+
/// Use this method only for rare, specialized checks that strictly depend on
2014+
/// the immediate synchronous state of the test zone right after the final
2015+
/// pump (such as asserting that no unflushed microtasks remain in `FakeAsync`).
2016+
/// Checks placed here cannot be reset or cleaned up by user `addTearDown`
2017+
/// callbacks.
2018+
void _verifyPostPumpInvariants() {}
2019+
2020+
/// Verifies framework-level invariants after the test body and all user
2021+
/// `addTearDown` callbacks have finished executing.
2022+
///
2023+
/// This method is called in [postTest] and is the default place for invariant
2024+
/// verifications. The vast majority of invariants belong here, including
2025+
/// persistent framework state, global debug flags (such as
2026+
/// [debugDefaultTargetPlatformOverride] or [timeDilation]), and resources
2027+
/// (such as pending [Timer]s or active animations). Running these checks in
2028+
/// [postTest] allows tests to cleanly reset state using `addTearDown`.
2029+
///
2030+
/// See also:
2031+
///
2032+
/// * `_verifyPostPumpInvariants`, which is used in rare cases where an
2033+
/// invariant strictly requires the immediate synchronous state following
2034+
/// widget disposal (such as `FakeAsync.microtaskCount` being zero).
19902035
void _verifyInvariants() {
19912036
assert(
19922037
debugAssertNoTransientCallbacks(
@@ -2042,13 +2087,7 @@ abstract class TestWidgetsFlutterBinding extends BindingBase
20422087
void _verifyAutoUpdateGoldensUnset(bool valueBeforeTest) {
20432088
assert(() {
20442089
if (autoUpdateGoldenFiles != valueBeforeTest) {
2045-
FlutterError.reportError(
2046-
FlutterErrorDetails(
2047-
exception: FlutterError('The value of autoUpdateGoldenFiles was changed by the test.'),
2048-
stack: StackTrace.current,
2049-
library: 'Flutter test framework',
2050-
),
2051-
);
2090+
throw FlutterError('The value of autoUpdateGoldenFiles was changed by the test.');
20522091
}
20532092
return true;
20542093
}());
@@ -2062,13 +2101,7 @@ abstract class TestWidgetsFlutterBinding extends BindingBase
20622101
// So we reset the error reporter to its initial value and then report
20632102
// this error.
20642103
reportTestException = valueBeforeTest;
2065-
FlutterError.reportError(
2066-
FlutterErrorDetails(
2067-
exception: FlutterError('The value of reportTestException was changed by the test.'),
2068-
stack: StackTrace.current,
2069-
library: 'Flutter test framework',
2070-
),
2071-
);
2104+
throw FlutterError('The value of reportTestException was changed by the test.');
20722105
}
20732106
return true;
20742107
}());
@@ -2077,13 +2110,7 @@ abstract class TestWidgetsFlutterBinding extends BindingBase
20772110
void _verifyErrorWidgetBuilderUnset(ErrorWidgetBuilder valueBeforeTest) {
20782111
assert(() {
20792112
if (ErrorWidget.builder != valueBeforeTest) {
2080-
FlutterError.reportError(
2081-
FlutterErrorDetails(
2082-
exception: FlutterError('The value of ErrorWidget.builder was changed by the test.'),
2083-
stack: StackTrace.current,
2084-
library: 'Flutter test framework',
2085-
),
2086-
);
2113+
throw FlutterError('The value of ErrorWidget.builder was changed by the test.');
20872114
}
20882115
return true;
20892116
}());
@@ -2092,14 +2119,8 @@ abstract class TestWidgetsFlutterBinding extends BindingBase
20922119
void _verifyShouldPropagateDevicePointerEventsUnset(bool valueBeforeTest) {
20932120
assert(() {
20942121
if (shouldPropagateDevicePointerEvents != valueBeforeTest) {
2095-
FlutterError.reportError(
2096-
FlutterErrorDetails(
2097-
exception: FlutterError(
2098-
'The value of shouldPropagateDevicePointerEvents was changed by the test.',
2099-
),
2100-
stack: StackTrace.current,
2101-
library: 'Flutter test framework',
2102-
),
2122+
throw FlutterError(
2123+
'The value of shouldPropagateDevicePointerEvents was changed by the test.',
21032124
);
21042125
}
21052126
return true;
@@ -2137,8 +2158,34 @@ abstract class TestWidgetsFlutterBinding extends BindingBase
21372158
}
21382159

21392160
/// Called by the [testWidgets] function after a test is executed.
2161+
///
2162+
/// Subclasses that override this method must call `super.postTest()` and must
2163+
/// not throw exceptions. This method will not throw errors; any invariant
2164+
/// failures detected during verification are reported via
2165+
/// [reportTestException].
2166+
@mustCallSuper
21402167
void postTest() {
21412168
assert(inTest);
2169+
FlutterErrorDetails? invariantError;
2170+
if (_shouldVerifyInvariants) {
2171+
_shouldVerifyInvariants = false;
2172+
try {
2173+
_verifyAutoUpdateGoldensUnset(_beforeTestAutoUpdateGoldens && !isBrowser);
2174+
_verifyReportTestExceptionUnset(_beforeTestReportTestException);
2175+
_verifyErrorWidgetBuilderUnset(_beforeTestErrorWidgetBuilder);
2176+
_verifyShouldPropagateDevicePointerEventsUnset(
2177+
_beforeTestShouldPropagateDevicePointerEvents,
2178+
);
2179+
_verifyInvariants();
2180+
} catch (error, stack) {
2181+
invariantError = FlutterErrorDetails(
2182+
exception: error,
2183+
stack: stack,
2184+
context: ErrorDescription('running invariant verifications after a test'),
2185+
library: 'Flutter test framework',
2186+
);
2187+
}
2188+
}
21422189
FlutterError.onError = _oldExceptionHandler;
21432190
FlutterError.demangleStackTrace = _oldStackTraceDemangler;
21442191
_pendingExceptionDetails = null;
@@ -2174,6 +2221,10 @@ abstract class TestWidgetsFlutterBinding extends BindingBase
21742221
assert(ServicesBinding.instance == WidgetsBinding.instance);
21752222
// ignore: invalid_use_of_visible_for_testing_member
21762223
ServicesBinding.instance.resetInternalState();
2224+
2225+
if (invariantError != null) {
2226+
reportTestException(invariantError, '');
2227+
}
21772228
}
21782229
}
21792230

@@ -2528,6 +2579,12 @@ class AutomatedTestWidgetsFlutterBinding extends TestWidgetsFlutterBinding {
25282579
super.asyncBarrier();
25292580
}
25302581

2582+
@override
2583+
void _verifyPostPumpInvariants() {
2584+
super._verifyPostPumpInvariants();
2585+
assert(_currentFakeAsync!.microtaskCount == 0); // Shouldn't be possible.
2586+
}
2587+
25312588
@override
25322589
void _verifyInvariants() {
25332590
super._verifyInvariants();
@@ -2549,7 +2606,6 @@ class AutomatedTestWidgetsFlutterBinding extends TestWidgetsFlutterBinding {
25492606
timersPending = true;
25502607
}
25512608
assert(!timersPending, 'A Timer is still pending even after the widget tree was disposed.');
2552-
assert(_currentFakeAsync!.microtaskCount == 0); // Shouldn't be possible.
25532609
}
25542610

25552611
@override

0 commit comments

Comments
 (0)