Refactor `android_engine_test`, make it easier to debug/deflake locally. #161534

matanlurey · 2025-01-13T19:14:38Z

The goal here is to have a great standalone android_engine_test suite that replaces scenario_app/android.

No test is functionally changed in this PR, but overview of changes:

Finished renaming the suite android_engine_tests instead of flutter_driver_android
Added instructions and an environment variable for local generation of golden-files (UPDATE_GOLDENS=1)
Added explanations of the individual tests, where they live, and how to run them locally
Added a hybrid-composition (HC, not TLHC, which is already tested) test
Renamed "other_smiley" to "surface_texture_smiley" (and renamed the original to "surface_producer_smiley")
Removed unnecessary ".android" suffix (we will not run this on anything but Android)
Added a tool/deflake.dart to run a test suite 10x (or custom) times locally to try and determine flakiness

After this PR, I'll add flags to let you control variants and name the screenshots accordingly, i.e.:

API v34 or v35
OpenGLES or Vulkan (will require an AndroidManifest.xml edit during the test instrumentation)

…gine-test

…engine-test

…gine-test

flutter-dashboard · 2025-01-14T00:02:16Z

Golden file changes have been found for this pull request. Click here to view and triage (e.g. because this is an intentional change).

If you are still iterating on this change and are not ready to resolve the images on the Flutter Gold dashboard, consider marking this PR as a draft pull request above. You will still be able to view image results on the dashboard, commenting will be silenced, and the check will not try to resolve itself until marked ready for review.

For more guidance, visit Writing a golden file test for package:flutter.

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

Changes reported for pull request #161534 at sha f239cb8

jonahwilliams

LGTM

reidbaker · 2025-01-14T15:30:43Z

dev/integration_tests/android_engine_test/README.md

+$ flutter drive lib/platform_view_tap_color_change_main_test.dart
+```
+
+Files of significance:


Non blocking, I am not sure files of significance adds much.

reidbaker · 2025-01-14T15:32:08Z

...android_engine_test/test_driver/external_texture/surface_producer_smiley_face_main_test.dart

    await expectLater(
      nativeDriver.screenshot(),
-      matchesGoldenFile('external_texture_smiley_face.android.png'),
+      matchesGoldenFile('external_texture_surface_producer_smiley_face.png'),


For my own education what controls the name of this file?

The literal string. That is, we could call this poop.png and it would work, so it's just a convention.

reidbaker · 2025-01-14T15:33:07Z

...ndroid_engine_test/test_driver/platform_view/hybrid_composition_platform_view_main_test.dart

+  test('should screenshot and match a blue -> orange gradient', () async {
+    await expectLater(
+      nativeDriver.screenshot(),
+      matchesGoldenFile('$goldenPrefix.blue_orange_gradient_portrait.png'),


Why do these tests need a golden prefix but the blue rectangle ones do not?

This file has multiple goldens, the other one does not.

reidbaker · 2025-01-14T15:33:33Z

...ndroid_engine_test/test_driver/platform_view/hybrid_composition_platform_view_main_test.dart

+      nativeDriver.screenshot(),
+      matchesGoldenFile('$goldenPrefix.blue_orange_gradient_portrait.png'),
+    );
+  }, timeout: Timeout.none);


For my own education why do these tests get no timeout?

It's a requirement of our infra (I don't agree with it):

flutter/dev/bots/analyze.dart

Line 1238 in 818133b

final int timeoutNoneCount = 'timeout: Timeout.none'.allMatches(contents).length;

reidbaker · 2025-01-14T15:38:15Z

dev/integration_tests/android_engine_test/tool/deflake.dart

+  }
+}
+
+final ArgParser _argParser =


nonblocking nit: I would prefer this to be at the top of main. My inital read I didn't realize there were flags that could be passed and thought your documentation was future looking.

Sounds reasonable. Done!

…gine-test

flutter-dashboard · 2025-01-14T23:05:26Z

Golden file changes are available for triage from new commit, Click here to view.

For more guidance, visit Writing a golden file test for package:flutter.

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

Changes reported for pull request #161534 at sha 0fd61e3

flutter-dashboard · 2025-01-15T00:21:16Z

Golden file changes are available for triage from new commit, Click here to view.

For more guidance, visit Writing a golden file test for package:flutter.

Reviewers: Read the Tree Hygiene page and make sure this patch meets those guidelines before LGTMing.

Changes reported for pull request #161534 at sha d462077

…ke locally. (flutter/flutter#161534)

flutter/flutter@40c2b86...5517cc9 2025-01-15 [email protected] feat: Change default value of keyboardDismissBehavior (flutter/flutter#158580) 2025-01-15 [email protected] Roll pub packages (flutter/flutter#161680) 2025-01-15 [email protected] Revert "Autocomplete Options Width" (flutter/flutter#161666) 2025-01-15 [email protected] Update two_dimensional_scrollables triage routing (flutter/flutter#161667) 2025-01-15 [email protected] [DisplayList] Migrate from SkRSXform to Impeller RSTransform (flutter/flutter#161652) 2025-01-15 [email protected] Roll Packages from d1fd623 to f73cb00 (2 revisions) (flutter/flutter#161672) 2025-01-15 [email protected] Fix DropdownMenu isCollapsed decoration does not Reduce height (flutter/flutter#161427) 2025-01-15 [email protected] Manual roll of Skia to e7b8d078851f (flutter/flutter#161609) 2025-01-15 [email protected] Fix `TabBar` glitchy elastic `Tab` animation (flutter/flutter#161514) 2025-01-15 [email protected] Roll pub packages (flutter/flutter#161643) 2025-01-15 [email protected] Exclude the top-level `engine` directory from `generate_gradle_lockfiles`. (flutter/flutter#161635) 2025-01-15 [email protected] Roll pub packages (flutter/flutter#161632) 2025-01-15 [email protected] Refactor `android_engine_test`, make it easier to debug/deflake locally. (flutter/flutter#161534) 2025-01-15 [email protected] [Impeller] null check device buffer in image encoding. (flutter/flutter#161194) 2025-01-15 [email protected] Feature/twitter keyboard (flutter/flutter#161025) 2025-01-15 [email protected] Fixed XiaoMi statusBar Bug (flutter/flutter#161271) 2025-01-15 [email protected] Clean up engine's analysis_options.yaml (flutter/flutter#161554) 2025-01-14 [email protected] Remove `gradle_deprecated_settings` test app, and remove reference from lockfile exclusion yaml (flutter/flutter#161622) 2025-01-14 [email protected] [deps] remove no-longer-used repo deps (flutter/flutter#161605) 2025-01-14 [email protected] [DisplayList] remove obsolete use of Skia goemetry objects in DL utils (flutter/flutter#161553) 2025-01-14 [email protected] [Engine] Support asymmetrical rounded superellipses (flutter/flutter#161409) 2025-01-14 [email protected] [SwiftPM] Make 'flutter build ios-framework' generate an empty Package.swift (flutter/flutter#161464) 2025-01-14 [email protected] [canvaskit] Fix GIF decode failure (flutter/flutter#161536) 2025-01-14 [email protected] [Impeller] fixes for AHB swapchains. (flutter/flutter#161562) 2025-01-14 [email protected] Last Engine<>Framework lint sync (flutter/flutter#161560) 2025-01-14 [email protected] Check that localization files of stocks app are up-to-date (flutter/flutter#161608) 2025-01-14 [email protected] [Android] Actually remove dev dependencies from release builds (flutter/flutter#161343) 2025-01-14 [email protected] Update package revisions to latest (flutter/flutter#161525) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages Please CC [email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

…ke locally. (flutter/flutter#161534)

flutter/flutter@40c2b86...5517cc9 2025-01-15 [email protected] feat: Change default value of keyboardDismissBehavior (flutter/flutter#158580) 2025-01-15 [email protected] Roll pub packages (flutter/flutter#161680) 2025-01-15 [email protected] Revert "Autocomplete Options Width" (flutter/flutter#161666) 2025-01-15 [email protected] Update two_dimensional_scrollables triage routing (flutter/flutter#161667) 2025-01-15 [email protected] [DisplayList] Migrate from SkRSXform to Impeller RSTransform (flutter/flutter#161652) 2025-01-15 [email protected] Roll Packages from d1fd623 to f73cb00 (2 revisions) (flutter/flutter#161672) 2025-01-15 [email protected] Fix DropdownMenu isCollapsed decoration does not Reduce height (flutter/flutter#161427) 2025-01-15 [email protected] Manual roll of Skia to e7b8d078851f (flutter/flutter#161609) 2025-01-15 [email protected] Fix `TabBar` glitchy elastic `Tab` animation (flutter/flutter#161514) 2025-01-15 [email protected] Roll pub packages (flutter/flutter#161643) 2025-01-15 [email protected] Exclude the top-level `engine` directory from `generate_gradle_lockfiles`. (flutter/flutter#161635) 2025-01-15 [email protected] Roll pub packages (flutter/flutter#161632) 2025-01-15 [email protected] Refactor `android_engine_test`, make it easier to debug/deflake locally. (flutter/flutter#161534) 2025-01-15 [email protected] [Impeller] null check device buffer in image encoding. (flutter/flutter#161194) 2025-01-15 [email protected] Feature/twitter keyboard (flutter/flutter#161025) 2025-01-15 [email protected] Fixed XiaoMi statusBar Bug (flutter/flutter#161271) 2025-01-15 [email protected] Clean up engine's analysis_options.yaml (flutter/flutter#161554) 2025-01-14 [email protected] Remove `gradle_deprecated_settings` test app, and remove reference from lockfile exclusion yaml (flutter/flutter#161622) 2025-01-14 [email protected] [deps] remove no-longer-used repo deps (flutter/flutter#161605) 2025-01-14 [email protected] [DisplayList] remove obsolete use of Skia goemetry objects in DL utils (flutter/flutter#161553) 2025-01-14 [email protected] [Engine] Support asymmetrical rounded superellipses (flutter/flutter#161409) 2025-01-14 [email protected] [SwiftPM] Make 'flutter build ios-framework' generate an empty Package.swift (flutter/flutter#161464) 2025-01-14 [email protected] [canvaskit] Fix GIF decode failure (flutter/flutter#161536) 2025-01-14 [email protected] [Impeller] fixes for AHB swapchains. (flutter/flutter#161562) 2025-01-14 [email protected] Last Engine<>Framework lint sync (flutter/flutter#161560) 2025-01-14 [email protected] Check that localization files of stocks app are up-to-date (flutter/flutter#161608) 2025-01-14 [email protected] [Android] Actually remove dev dependencies from release builds (flutter/flutter#161343) 2025-01-14 [email protected] Update package revisions to latest (flutter/flutter#161525) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages Please CC [email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

matanlurey added 5 commits January 10, 2025 11:26

Refactor and document android_engine_test.

f543b9b

Merge remote-tracking branch 'upstream/main' into refactor-android-en…

73848c0

…gine-test

More WIP

ccc7b89

Merge remote-tracking branch 'upstream/master' into refactor-android-…

cbc5f03

…engine-test

Remove .android.

fd1c891

matanlurey mentioned this pull request Jan 13, 2025

Remove scenario_app/android and rename to ios_scenario_app. #160992

Merged

matanlurey added 5 commits January 13, 2025 13:24

Merge remote-tracking branch 'upstream/main' into refactor-android-en…

843e5be

…gine-test

Finish refactoring the tests themselves.

cf35064

++

386318f

Tweak, update suite_runner.

54b72f3

Finish updating README and doc comments.

f239cb8

matanlurey marked this pull request as ready for review January 13, 2025 22:45

matanlurey changed the title ~~WIP: Refactor android_engine_test.~~ Refactor android_engine_test, make it easier to debug/deflake locally. Jan 13, 2025

matanlurey requested review from jonahwilliams and reidbaker January 13, 2025 22:50

flutter-dashboard bot added the will affect goldens Changes to golden files label Jan 14, 2025

Fix prefix.

8d35144

jonahwilliams approved these changes Jan 14, 2025

View reviewed changes

reidbaker approved these changes Jan 14, 2025

View reviewed changes

matanlurey added 2 commits January 14, 2025 13:02

Merge remote-tracking branch 'upstream/main' into refactor-android-en…

84aa1a9

…gine-test

Tweaks and format.

0fd61e3

Fix double ..

d462077

matanlurey added the autosubmit Merge PR when tree becomes green via auto submit App label Jan 14, 2025

auto-submit bot added this pull request to the merge queue Jan 15, 2025

Merged via the queue into flutter:master with commit e517ae3 Jan 15, 2025
177 of 178 checks passed

flutter-dashboard bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Jan 15, 2025

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 15, 2025

e517ae3 Refactor android_engine_test, make it easier to debug/defla…

b42f187

…ke locally. (flutter/flutter#161534)

reidbaker mentioned this pull request Jan 15, 2025

Misconfigured android engine test "Invalid shard: android_engine_tests" #161674

Open

engine-flutter-autoroll mentioned this pull request Jan 15, 2025

Roll Flutter from 40c2b86fdf65 to 1580a3d3fbe7 (23 revisions) flutter/packages#8437

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 15, 2025

e517ae3 Refactor android_engine_test, make it easier to debug/defla…

3ebb300

…ke locally. (flutter/flutter#161534)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 16, 2025

e517ae3 Refactor android_engine_test, make it easier to debug/defla…

6fdcd3f

…ke locally. (flutter/flutter#161534)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 16, 2025

e517ae3 Refactor android_engine_test, make it easier to debug/defla…

f3001ba

…ke locally. (flutter/flutter#161534)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 16, 2025

e517ae3 Refactor android_engine_test, make it easier to debug/defla…

f110fe0

…ke locally. (flutter/flutter#161534)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 16, 2025

e517ae3 Refactor android_engine_test, make it easier to debug/defla…

277c4a8

…ke locally. (flutter/flutter#161534)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 16, 2025

e517ae3 Refactor android_engine_test, make it easier to debug/defla…

af64395

…ke locally. (flutter/flutter#161534)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 16, 2025

e517ae3 Refactor android_engine_test, make it easier to debug/defla…

bcaae92

…ke locally. (flutter/flutter#161534)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 16, 2025

e517ae3 Refactor android_engine_test, make it easier to debug/defla…

cbb08f5

…ke locally. (flutter/flutter#161534)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 16, 2025

e517ae3 Refactor android_engine_test, make it easier to debug/defla…

f08d279

…ke locally. (flutter/flutter#161534)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Jan 16, 2025

e517ae3 Refactor android_engine_test, make it easier to debug/defla…

7ac99e3

…ke locally. (flutter/flutter#161534)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Feb 12, 2025

e517ae3 Refactor android_engine_test, make it easier to debug/defla…

bd28cd7

…ke locally. (flutter/flutter#161534)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Feb 13, 2025

e517ae3 Refactor android_engine_test, make it easier to debug/defla…

d1e180e

…ke locally. (flutter/flutter#161534)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Feb 13, 2025

e517ae3 Refactor android_engine_test, make it easier to debug/defla…

698a2b0

…ke locally. (flutter/flutter#161534)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 6, 2025

e517ae3 Refactor android_engine_test, make it easier to debug/defla…

3d40780

…ke locally. (flutter/flutter#161534)

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Mar 7, 2025

e517ae3 Refactor android_engine_test, make it easier to debug/defla…

bead208

…ke locally. (flutter/flutter#161534)

Refactor android_engine_test, make it easier to debug/deflake locally. #161534

Refactor android_engine_test, make it easier to debug/deflake locally. #161534

Uh oh!

Conversation

matanlurey commented Jan 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

flutter-dashboard bot commented Jan 14, 2025

Uh oh!

jonahwilliams left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

flutter-dashboard bot commented Jan 14, 2025

Uh oh!

flutter-dashboard bot commented Jan 15, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Refactor `android_engine_test`, make it easier to debug/deflake locally. #161534

Refactor `android_engine_test`, make it easier to debug/deflake locally. #161534

matanlurey commented Jan 13, 2025 •

edited

Loading