Fail tests on exceptions raised after test completed #144706

goderbauer · 2024-03-06T19:20:56Z

A test was failing silently because of this (see #144353 and fixed in #144709). The failure went undetected for months. Ideally, this should have been a regular non-silent failure. This change makes that so. package:test can properly handle reported exceptions outside of test cases. With this change, the test fails as follows:

00:03 +82: Smoke test material/color_scheme/dynamic_content_color.0.dart
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following assertion was thrown running a test (but after the test had completed):
setState() called after dispose(): _DynamicColorExampleState#1cd37(lifecycle state: defunct, not
mounted)
This error happens if you call setState() on a State object for a widget that no longer appears in
the widget tree (e.g., whose parent widget no longer includes the widget in its build). This error
can occur when code calls setState() from a timer or an animation callback.
The preferred solution is to cancel the timer or stop listening to the animation in the dispose()
callback. Another solution is to check the "mounted" property of this object before calling
setState() to ensure the object is still in the tree.
This error might indicate a memory leak if setState() is being called because another object is
retaining a reference to this State object after it has been removed from the tree. To avoid memory
leaks, consider breaking the reference to this object during dispose().

When the exception was thrown, this was the stack:
#0      State.setState.<anonymous closure> (package:flutter/src/widgets/framework.dart:1167:9)
#1      State.setState (package:flutter/src/widgets/framework.dart:1202:6)
#2      _DynamicColorExampleState._updateImage (package:flutter_api_samples/material/color_scheme/dynamic_content_color.0.dart:191:5)
<asynchronous suspension>
════════════════════════════════════════════════════════════════════════════════════════════════════
00:03 +81 -1: Smoke test material/context_menu/context_menu_controller.0.dart
00:03 +81 -1: Smoke test material/color_scheme/dynamic_content_color.0.dart [E]
  Test failed. See exception logs above.
  The test description was: Smoke test material/color_scheme/dynamic_content_color.0.dart
  
  This test failed after it had already completed.
  Make sure to use a matching library which informs the test runner
  of pending async work.

goderbauer · 2024-03-06T19:57:08Z

The failing checks are the reason for his PR, they used to be silent and went undetected. The failures will go away with the fix in #144709.

packages/flutter_test/lib/src/binding.dart

This fixes the silent failure reported in #144353. I am experimenting in #144706 with whether the failure should have been non-silent.

goderbauer · 2024-03-07T21:46:59Z

dev/automated_tests/test_smoke_test/fail_test_on_exception_after_test.dart

Verified in #144785 that the new test is indeed failing on current tip of tree without these changes.

Piinks

Awesome!

auto-submit · 2024-03-11T23:28:46Z

A reason for requesting a revert of flutter/flutter/144706 could
not be found or the reason was not properly formatted. Begin a comment with 'Reason for revert:' to tell the bot why
this issue is being reverted.

gspencergoog · 2024-03-11T23:29:30Z

Reason for revert: This has broken the tree because some tests are still failing post completion. This particular one looks like it might have to do with a gold image not existing.

This reverts commit 91cccc8.

…" (#144970) Reverts: #144706 Initiated by: gspencergoog Reason for reverting: This has broken the tree because some tests are still failing post completion. This particular one looks like it might have to do with a gold image not existing. Original PR Author: goderbauer Reviewed By: {Piinks} This change reverts the following previous change: A test was failing silently because of this (see #144353 and fixed in #144709). The failure went undetected for months. Ideally, this should have been a regular non-silent failure. This change makes that so. `package:test` can properly handle reported exceptions outside of test cases. With this change, the test fails as follows: ``` 00:03 +82: Smoke test material/color_scheme/dynamic_content_color.0.dart ══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════ The following assertion was thrown running a test (but after the test had completed): setState() called after dispose(): _DynamicColorExampleState#1cd37(lifecycle state: defunct, not mounted) This error happens if you call setState() on a State object for a widget that no longer appears in the widget tree (e.g., whose parent widget no longer includes the widget in its build). This error can occur when code calls setState() from a timer or an animation callback. The preferred solution is to cancel the timer or stop listening to the animation in the dispose() callback. Another solution is to check the "mounted" property of this object before calling setState() to ensure the object is still in the tree. This error might indicate a memory leak if setState() is being called because another object is retaining a reference to this State object after it has been removed from the tree. To avoid memory leaks, consider breaking the reference to this object during dispose(). When the exception was thrown, this was the stack: #0 State.setState.<anonymous closure> (package:flutter/src/widgets/framework.dart:1167:9) #1 State.setState (package:flutter/src/widgets/framework.dart:1202:6) #2 _DynamicColorExampleState._updateImage (package:flutter_api_samples/material/color_scheme/dynamic_content_color.0.dart:191:5) <asynchronous suspension> ════════════════════════════════════════════════════════════════════════════════════════════════════ 00:03 +81 -1: Smoke test material/context_menu/context_menu_controller.0.dart 00:03 +81 -1: Smoke test material/color_scheme/dynamic_content_color.0.dart [E] Test failed. See exception logs above. The test description was: Smoke test material/color_scheme/dynamic_content_color.0.dart This test failed after it had already completed. Make sure to use a matching library which informs the test runner of pending async work. ```

Hixie · 2024-03-12T00:04:16Z

@gspencergoog Can you list the failing tests here so @goderbauer (or people doing archeology) can follow the breadcrumb trail? Thanks!

goderbauer · 2024-03-12T05:06:34Z

Failure for prosperity included below. Looks like the test is missing an await, which is fixed by #144978.

03:59 +598 ~29: test/material/chip_test.dart: Delete button is visible on disabled RawChip                                                                                                             [+9618 ms] Preparing to send command: {"imageFile":"/b/s/w/ir/x/t/flutter_tools.LOFYQB/flutter_web_platform.FXXMZU/imageCVLAFW/image","key":"raw_chip.disabled.delete_button.png","update":false}
[ +237 ms] <<< {"success":false,"message":"SkiaException: Skia Gold received an unapproved image in post-submit \ntesting. Golden file images in flutter/flutter are triaged \nin pre-submit during code review for the given PR.\n\nVisit https://flutter-gold.skia.org// to view and approve \nthe image(s), or revert the associated change. For more \ninformation, visit the wiki: \nhttps://github.com/flutter/flutter/wiki/Writing-a-golden-file-test-for-package:flutter\n\nDebug information for Gold --------------------------------\nstdout: Given image with hash 355e4280d8ca02fbdcf5fe321f0af129 for test material.raw_chip.disabled.delete_button\nExpectation for test: 155a76b26e049ebc408f50458bc714d7 (positive)\nExpectation for test: 3632280d51c05370fb39a943f3ef92e5 (positive)\nExpectation for test: 821c91b7a6b1de219c7f1cea3f5e7920 (positive)\nExpectation for test: 8ca29dc308c415cd31d233ef250007be (positive)\nExpectation for test: dba77e2843dfcb0d2ec5803a3665f2d1 (positive)\nExpectation for test: f9501bce78e84a55ce4c8f86d6ea1def (positive)\nExpectation for test: 0133f6382f3179df5f87efe0d1d95899 (positive)\nExpectation for test: 121961c0a6a411327dd796c86dec8846 (positive)\nExpectation for test: 194b1f256d05dac7284056d4633ee069 (positive)\nExpectation for test: 314510ea64798e865d148ed65433424b (positive)\nExpectation for test: 72c68746eb3e2255800d0aa2946ee581 (positive)\nExpectation for test: ccd17ed15bcb9643d7a9f55146161773 (positive)\nExpectation for test: 1ea0cd337497310f4c7299e29dcd245d (positive)\nExpectation for test: 3a1a7b2a497a4c9184cfe7df37218b6a (positive)\nExpectation for test: 863bc286a8cb7637706c48a3a70ea0ec (positive)\nExpectation for test: 9fc8003bfff820a37313963b517b6b33 (positive)\nExpectation for test: a655c4207515fedf8115512420341221 (positive)\nExpectation for test: 317bd7e3a5400ffec160c3a10b2ac25c (negative)\nExpectation for test: 355e4280d8ca02fbdcf5fe321f0af129 (negative)\nExpectation for test: 5272bb7a944f7b9efb46345a80bc936b (negative)\nExpectation for test: 811facfffeb0a530eb63c5e03c4629ad (positive)\nUntriaged or negative image: [https://flutter-gold.skia.org/detail?grouping=name%3Dmaterial.raw_chip.disabled.delete_button%26source_type%3Dflutter&digest=355e4280d8ca02fbdcf5fe321f0af129\n\n\nstderr](https://flutter-gold.skia.org/detail?grouping=name%3Dmaterial.raw_chip.disabled.delete_button%26source_type%3Dflutter&digest=355e4280d8ca02fbdcf5fe321f0af129%5cn%5cn%5cnstderr): Test: material.raw_chip.disabled.delete_button FAIL\n\n\nresult-state.json: No result file found.\n"}

03:59 +598 ~29: test/material/chip_test.dart: Delete button is visible on disabled RawChip                                                                                                             
══╡ EXCEPTION CAUGHT BY FLUTTER TEST FRAMEWORK ╞════════════════════════════════════════════════════
The following TestFailure was thrown running a test (but after the test had completed):
Expected: one widget whose rasterized image matches golden image
"raw_chip.disabled.delete_button.png"
  Actual: _TypeWidgetFinder:<Found 1 widget with type "RawChip": [
            StatefulElement#f34d0(DEFUNCT)(no widget),
          ]>
   Which: SkiaException: Skia Gold received an unapproved image in post-submit
          testing. Golden file images in flutter/flutter are triaged
          in pre-submit during code review for the given PR.

          Visit https://flutter-gold.skia.org// to view and approve
          the image(s), or revert the associated change. For more
          information, visit the wiki:
          https://github.com/flutter/flutter/wiki/Writing-a-golden-file-test-for-package:flutter

          Debug information for Gold --------------------------------
          stdout: Given image with hash 355e4280d8ca02fbdcf5fe321f0af129 for test
material.raw_chip.disabled.delete_button
          Expectation for test: 155a76b26e049ebc408f50458bc714d7 (positive)
          Expectation for test: 3632280d51c05370fb39a943f3ef92e5 (positive)
          Expectation for test: 821c91b7a6b1de219c7f1cea3f5e7920 (positive)
          Expectation for test: 8ca29dc308c415cd31d233ef250007be (positive)
          Expectation for test: dba77e2843dfcb0d2ec5803a3665f2d1 (positive)
          Expectation for test: f9501bce78e84a55ce4c8f86d6ea1def (positive)
          Expectation for test: 0133f6382f3179df5f87efe0d1d95899 (positive)
          Expectation for test: 121961c0a6a411327dd796c86dec8846 (positive)
          Expectation for test: 194b1f256d05dac7284056d4633ee069 (positive)
          Expectation for test: 314510ea64798e865d148ed65433424b (positive)
          Expectation for test: 72c68746eb3e2255800d0aa2946ee581 (positive)
          Expectation for test: ccd17ed15bcb9643d7a9f55146161773 (positive)
          Expectation for test: 1ea0cd337497310f4c7299e29dcd245d (positive)
          Expectation for test: 3a1a7b2a497a4c9184cfe7df37218b6a (positive)
          Expectation for test: 863bc286a8cb7637706c48a3a70ea0ec (positive)
          Expectation for test: 9fc8003bfff820a37313963b517b6b33 (positive)
          Expectation for test: a655c4207515fedf8115512420341221 (positive)
          Expectation for test: 317bd7e3a5400ffec160c3a10b2ac25c (negative)
          Expectation for test: 355e4280d8ca02fbdcf5fe321f0af129 (negative)
          Expectation for test: 5272bb7a944f7b9efb46345a80bc936b (negative)
          Expectation for test: 811facfffeb0a530eb63c5e03c4629ad (positive)
          Untriaged or negative image:
https://flutter-gold.skia.org/detail?grouping=name%3Dmaterial.raw_chip.disabled.delete_button%26source_type%3Dflutter&digest=355e4280d8ca02fbdcf5fe321f0af129


          stderr: Test: material.raw_chip.disabled.delete_button FAIL


          result-state.json: No result file found.

When the exception was thrown, this was the stack:
../dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/errors.dart 296:3  throw_
../packages/matcher/src/expect/prints_matcher.dart.js 453:22                    fail
../packages/matcher/src/expect/prints_matcher.dart.js 430:18                    <fn>
../packages/stack_trace/src/stack_zone_specification.dart.js 241:98             <fn>
../packages/stack_trace/src/stack_zone_specification.dart.js 293:16             [_run]
../packages/stack_trace/src/stack_zone_specification.dart.js 241:80             <fn>
../dart-sdk/lib/async/zone.dart 1407:47                                         _rootRunUnary
../dart-sdk/lib/async/zone.dart 1308:19                                         runUnary
../dart-sdk/lib/async/future_impl.dart 163:18                                   handleValue
../dart-sdk/lib/async/future_impl.dart 847:44                                   handleValueCallback
../dart-sdk/lib/async/future_impl.dart 876:13                                   _propagateToListeners
../dart-sdk/lib/async/future_impl.dart 652:5                                    [_completeWithValue]
../dart-sdk/lib/async/future_impl.dart 722:7                                    callback
../dart-sdk/lib/async/schedule_microtask.dart 40:11                             _microtaskLoop
../dart-sdk/lib/async/schedule_microtask.dart 49:5                              _startMicrotaskLoop
../dart-sdk/lib/_internal/js_dev_runtime/patch/async_patch.dart 181:7           <fn>
════════════════════════════════════════════════════════════════════════════════════════════════════

03:59 +598 ~29 -1: test/material/chip_test.dart: Delete button is visible on disabled RawChip [E]                                                                                                      
  Test failed. See exception logs above.
  The test description was: Delete button is visible on disabled RawChip
  

To run this test again: /b/s/w/ir/x/w/flutter/bin/cache/dart-sdk/bin/dart test test/material/chip_test.dart -p chrome --plain-name 'Delete button is visible on disabled RawChip'

…/flutter#144706)

Manual roll requested by [email protected] flutter/flutter@3bb2e59...1ca8873 2024-03-12 [email protected] Update integration tests regexes. (flutter/flutter#144847) 2024-03-11 98614782+auto-submit[bot]@users.noreply.github.com Reverts "Fail tests on exceptions raised after test completed (#144706)" (flutter/flutter#144970) 2024-03-11 [email protected] Make TabController communicating creation in constructor. (flutter/flutter#144912) 2024-03-11 [email protected] Fail tests on exceptions raised after test completed (flutter/flutter#144706) 2024-03-11 [email protected] Refactoring `if` chains into `switch` statements (flutter/flutter#144905) 2024-03-11 98614782+auto-submit[bot]@users.noreply.github.com Reverts "Expose build mode in environment of asset transformer processes (#144752)" (flutter/flutter#144957) 2024-03-11 [email protected] Expose build mode in environment of asset transformer processes (flutter/flutter#144752) 2024-03-11 [email protected] Roll Flutter Engine from 9196947bc687 to 6745955bb49e (2 revisions) (flutter/flutter#144946) 2024-03-11 [email protected] Skip test temporarily until headingLevel is added in engine (issue 41â�¦ (flutter/flutter#135077) 2024-03-11 [email protected] Roll Flutter Engine from 3b0b59bb224d to 9196947bc687 (1 revision) (flutter/flutter#144934) 2024-03-11 [email protected] Roll Packages from 0badb43 to d489d84 (3 revisions) (flutter/flutter#144931) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages Please CC [email protected],[email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

gspencergoog · 2024-03-12T15:34:25Z

@gspencergoog Can you list the failing tests here so @goderbauer (or people doing archeology) can follow the breadcrumb trail? Thanks!

I did actually list them in the revert PR at #144970 (comment) But I suppose I could have posted them here too.

This was uncovered as part of #144706.

…" (#144980) Reverts #144970 No changes in this PR compared to the original. The test failure was fixed by adding missing awaits in #144978. Fixes #144353.

…/flutter#144706)

github-actions bot added a: tests "flutter test", flutter_test, or one of our tests framework flutter/packages/flutter repository. See also f: labels. labels Mar 6, 2024

goderbauer mentioned this pull request Mar 6, 2024

Fix code sample failing in smoke test #144709

Merged

goderbauer commented Mar 6, 2024

View reviewed changes

packages/flutter_test/lib/src/binding.dart Outdated Show resolved Hide resolved

auto-submit bot pushed a commit that referenced this pull request Mar 6, 2024

Fix code sample failing in smoke test (#144709)

9e46c67

This fixes the silent failure reported in #144353. I am experimenting in #144706 with whether the failure should have been non-silent.

goderbauer force-pushed the failTestsAfterTheyCompleted branch 3 times, most recently from 039638b to aa371fe Compare March 7, 2024 18:06

goderbauer commented Mar 7, 2024

View reviewed changes

goderbauer marked this pull request as ready for review March 7, 2024 21:47

goderbauer requested a review from Piinks March 7, 2024 21:47

Piinks approved these changes Mar 8, 2024

View reviewed changes

goderbauer force-pushed the failTestsAfterTheyCompleted branch from 27f00b8 to 4c4804d Compare March 8, 2024 21:40

goderbauer added 4 commits March 11, 2024 09:46

Fail tests on exceptions raised after test completed

9a2d3c3

test

8c647ff

fix other test

5fdbfb8

++

a8b03bd

goderbauer force-pushed the failTestsAfterTheyCompleted branch from 4c4804d to a8b03bd Compare March 11, 2024 16:46

goderbauer added autosubmit Merge PR when tree becomes green via auto submit App and removed autosubmit Merge PR when tree becomes green via auto submit App labels Mar 11, 2024

goderbauer merged commit 91cccc8 into flutter:master Mar 11, 2024

goderbauer deleted the failTestsAfterTheyCompleted branch March 11, 2024 23:05

gspencergoog added the revert Autorevert PR (with "Reason for revert:" comment) label Mar 11, 2024

auto-submit bot removed the revert Autorevert PR (with "Reason for revert:" comment) label Mar 11, 2024

gspencergoog added the revert Autorevert PR (with "Reason for revert:" comment) label Mar 11, 2024

auto-submit bot pushed a commit that referenced this pull request Mar 11, 2024

Revert "Fail tests on exceptions raised after test completed (#144706)"

cb2bf9b

This reverts commit 91cccc8.

auto-submit bot mentioned this pull request Mar 11, 2024

Reverts "Fail tests on exceptions raised after test completed (#144706)" #144970

Merged

auto-submit bot removed the revert Autorevert PR (with "Reason for revert:" comment) label Mar 11, 2024

goderbauer mentioned this pull request Mar 12, 2024

add missing awaits #144978

Merged

engine-flutter-autoroll mentioned this pull request Mar 12, 2024

Manual roll Flutter from 3bb2e5948e7a to 1ca88730a0c1 (11 revisions) flutter/packages#6304

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 12, 2024

91cccc8 Fail tests on exceptions raised after test completed (flutter…

8a49acc

…/flutter#144706)

auto-submit bot pushed a commit that referenced this pull request Mar 12, 2024

add missing awaits (#144978)

e277549

This was uncovered as part of #144706.

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request May 14, 2024

91cccc8 Fail tests on exceptions raised after test completed (flutter…

a0d271e

…/flutter#144706)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Fail tests on exceptions raised after test completed #144706

Fail tests on exceptions raised after test completed #144706

Uh oh!

goderbauer commented Mar 6, 2024 •

edited

Loading

Uh oh!

goderbauer commented Mar 6, 2024

Uh oh!

Uh oh!

goderbauer Mar 7, 2024

Uh oh!

Piinks left a comment

Uh oh!

auto-submit bot commented Mar 11, 2024

Uh oh!

gspencergoog commented Mar 11, 2024

Uh oh!

Hixie commented Mar 12, 2024

Uh oh!

goderbauer commented Mar 12, 2024

Uh oh!

gspencergoog commented Mar 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Fail tests on exceptions raised after test completed #144706

Fail tests on exceptions raised after test completed #144706

Uh oh!

Conversation

goderbauer commented Mar 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

goderbauer commented Mar 6, 2024

Uh oh!

Uh oh!

goderbauer Mar 7, 2024

Choose a reason for hiding this comment

Uh oh!

Piinks left a comment

Choose a reason for hiding this comment

Uh oh!

auto-submit bot commented Mar 11, 2024

Uh oh!

gspencergoog commented Mar 11, 2024

Uh oh!

Hixie commented Mar 12, 2024

Uh oh!

goderbauer commented Mar 12, 2024

Uh oh!

gspencergoog commented Mar 12, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

goderbauer commented Mar 6, 2024 •

edited

Loading