Support using lightweight Flutter Engines to run tests #141726

derekxu16 · 2024-01-17T22:37:25Z

This PR implements the functionality described above and hides it behind the --experimental-faster-testing flag of flutter test.

The following are some performance measurements from test runs conducted on GitHub Actions

run 1 logs: https://github.com/derekxu16/flutter_test_ci/actions/runs/8008029772/attempts/1
run 2 logs: https://github.com/derekxu16/flutter_test_ci/actions/runs/8008029772/attempts/2
run 3 logs: https://github.com/derekxu16/flutter_test_ci/actions/runs/8008029772/attempts/3

length of flutter test --reporter=expanded test/animation test/foundation step

run 1: 54s
run 2: 52s
run 3: 56s

average: 54s

length of flutter test --experimental-faster-testing --reporter=expanded test/animation test/foundation step

run 1: 27s
run 2: 27s
run 3: 29s

average: 27.67s (~48.77% shorter than 54s)

length of flutter test --reporter=expanded test/animation test/foundation test/gestures test/painting test/physics test/rendering test/scheduler test/semantics test/services step

run 1: 260s
run 2: 270s
run 3: 305s

average: 278.33s

length of flutter test --experimental-faster-testing --reporter=expanded test/animation test/foundation test/gestures test/painting test/physics test/rendering test/scheduler test/semantics test/services step

from a clean build (right after deleting the build folder):

run 1: 215s
run 2: 227s
run 3: 245s

average: 229s (~17.72% shorter than 278.33s)

Note that in reality, test/material was not passed to flutter test in the trials below. All of the test files under test/material except for test/material/icons_test.dart were listed out individually

length of flutter test --reporter=expanded test/material step

run 1: 408s
run 2: 421s
run 3: 451s

average: 426.67s

length of flutter test --experimental-faster-testing --reporter=expanded test/material step

run 1: 382s
run 2: 373s
run 3: 400s

average: 385s (~9.77% shorter than 426.67s)

packages/flutter_test/pubspec.yaml

packages/flutter_tools/lib/src/test/runner.dart

packages/flutter_tools/test/integration.shard/test_test.dart

derekxu16 · 2024-01-19T22:09:35Z

cc @a-siva

packages/flutter_tools/lib/src/commands/test.dart

derekxu16 · 2024-02-05T18:51:19Z

I just tried running flutter test --experimental-faster-testing on Windows to investigate the Windows failures. The problem is that when running root_test_isolate_spawner.dart, the _loadLibraryFromKernel call here

https://github.com/flutter/flutter/blob/3380abbb68a1b84d0c17823602fc5f632ce60925/packages/flutter_tools/lib/src/test/runner.dart#L519-L521

is returning null, giving the error:

[ERROR:flutter/shell/testing/tester_main.cc(666)] Unhandled exception
Exception: type 'Null' is not a subtype of type '(SendPort) => void' in type cast
Stack trace: #0      main (file:///C:/Users/Derek/flutter/packages/flutter/build/isolate_spawning_tester/root_test_isolate_spawner.dart:70:32)

I tried changing '${childTestIsolateSpawnerDillFile.absolute.path}' to 'file://${childTestIsolateSpawnerDillFile.absolute.path}', but that had no effect. Any ideas @dnfield?

dnfield · 2024-02-12T20:22:01Z

The windows version of this needs a windows-style path. File URIs won't work.

derekxu16 · 2024-02-14T20:51:41Z

The checks are now all passing.

packages/flutter_tools/pubspec.yaml

dnfield

What's the result of using this flag locally to run tests compared to the "default" behavior?

What's the plan for implementing this in Flutter CI?

dnfield

This LGTM to get things rolling. I'd appreciate it if you remove the comment I tagged in the pubspec. It also needs a merge conflict fix.

It would be very helpful if you add some numbers even if just locally to this PR to compare how it's running on e.g. the framework tests with and without the flag.

…ines

…on Windows

derekxu16 · 2024-02-15T22:00:31Z

Running the material tests with --experimental-faster-testing produces some Golden pixel test failures that don't occur when running without the flag. I'm trying to fix these now.

…er/flutter#141726)

flutter/flutter@41581c9...39585e6 2024-02-23 [email protected] Re-enable Impeller goldens blocking. (flutter/flutter#144023) 2024-02-23 [email protected] Roll Flutter Engine from 0f3ad23b84ed to 5f99a6c3289e (16 revisions) (flutter/flutter#144028) 2024-02-23 [email protected] Roll Packages from 078c2a3 to 7df2085 (3 revisions) (flutter/flutter#144030) 2024-02-23 [email protected] Run tests on iOS 17 exclusively (flutter/flutter#144022) 2024-02-23 [email protected] Retry button tap in [FlutterUITests testFlutterViewWarm] (flutter/flutter#143967) 2024-02-23 [email protected] Remove deprecated `InteractiveViewer.alignPanAxis` (flutter/flutter#142500) 2024-02-23 [email protected] Remove deprecated `KeepAliveHandle.release` (flutter/flutter#143961) 2024-02-23 [email protected] Remove deprecated AnimatedListItemBuilder, AnimatedListRemovedItemBuilder (flutter/flutter#143974) 2024-02-23 [email protected] Manual roll Flutter Engine from 06448ee8165f to 0f3ad23b84ed (16 revisions) (flutter/flutter#143989) 2024-02-23 [email protected] Remove deprecated TimelineSummary.writeSummaryToFile (flutter/flutter#143983) 2024-02-23 [email protected] Remove unused drone_dimension field (flutter/flutter#143984) 2024-02-23 [email protected] Remove deprecated MediaQuery.boldTextOverride (flutter/flutter#143960) 2024-02-22 [email protected] Revert engine back to 06448ee8165f4ff6579ef9510ac69322eda896ac (flutter/flutter#143986) 2024-02-22 [email protected] Remove deprecated FlutterDriver.enableAccessibility (flutter/flutter#143979) 2024-02-22 [email protected] Roll Flutter Engine from cb6115d3a6ab to c5d2e011652b (3 revisions) (flutter/flutter#143980) 2024-02-22 [email protected] Revert "Introduce tone-based surfaces and accent color add-ons - Part 2" (flutter/flutter#143973) 2024-02-22 [email protected] Add CloseButtonTooltip to the 'X' button on a SnackBar (flutter/flutter#143934) 2024-02-22 49699333+dependabot[bot]@users.noreply.github.com Bump github/codeql-action from 3.24.3 to 3.24.4 (flutter/flutter#143971) 2024-02-22 [email protected] Roll Flutter Engine from 06448ee8165f to cb6115d3a6ab (10 revisions) (flutter/flutter#143972) 2024-02-22 [email protected] Use inlining annotations on important methods for all targets (flutter/flutter#143923) 2024-02-22 [email protected] Add missing `TextPainter.strutStyle` to paragraph style (flutter/flutter#143771) 2024-02-22 [email protected] Shift Moto G4 staging tests to A02s (flutter/flutter#143957) 2024-02-22 [email protected] Add CurvedAnimation disposals in some widgets (flutter/flutter#143790) 2024-02-22 [email protected] Fix documentation typo in basic.dart (flutter/flutter#143887) 2024-02-22 [email protected] Fixed Small Typo in Emulators Test Name (flutter/flutter#143578) 2024-02-22 [email protected] Support using lightweight Flutter Engines to run tests (flutter/flutter#141726) 2024-02-22 [email protected] Roll Flutter Engine from 9250bfdb38c5 to 06448ee8165f (1 revision) (flutter/flutter#143943) 2024-02-22 98614782+auto-submit[bot]@users.noreply.github.com Reverts "Reland Add UI Benchmarks (#143542) (#143799)" (flutter/flutter#143946) 2024-02-22 [email protected] Roll Packages from 48048f6 to 078c2a3 (11 revisions) (flutter/flutter#143935) 2024-02-22 [email protected] Roll Flutter Engine from 7777480eb94d to 9250bfdb38c5 (8 revisions) (flutter/flutter#143933) 2024-02-22 [email protected] Reland Add UI Benchmarks (#143542) (flutter/flutter#143799) If this roll has caused a breakage, revert this CL and stop the roller using the controls here: https://autoroll.skia.org/r/flutter-packages Please CC [email protected],[email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

…er/flutter#141726)

github-actions bot added a: tests "flutter test", flutter_test, or one of our tests tool Affects the "flutter" command-line tool. See also t: labels. framework flutter/packages/flutter repository. See also f: labels. labels Jan 17, 2024

derekxu16 force-pushed the experimental-faster-testing branch from 8622538 to 0746b51 Compare January 17, 2024 22:37

derekxu16 changed the title ~~Support using lightweight Flutter Engines to run test~~ Support using lightweight Flutter Engines to run tests Jan 17, 2024

derekxu16 force-pushed the experimental-faster-testing branch from 0746b51 to 8c6e6dd Compare January 18, 2024 19:22

derekxu16 commented Jan 18, 2024

View reviewed changes

packages/flutter_test/pubspec.yaml Outdated Show resolved Hide resolved

derekxu16 commented Jan 18, 2024

View reviewed changes

packages/flutter_tools/lib/src/test/runner.dart Outdated Show resolved Hide resolved

derekxu16 commented Jan 18, 2024

View reviewed changes

packages/flutter_tools/lib/src/test/runner.dart Outdated Show resolved Hide resolved

derekxu16 force-pushed the experimental-faster-testing branch 3 times, most recently from 61c7a6f to 6b68eb8 Compare January 19, 2024 21:48

derekxu16 commented Jan 19, 2024

View reviewed changes

packages/flutter_tools/test/integration.shard/test_test.dart Outdated Show resolved Hide resolved

derekxu16 requested review from christopherfujino and dnfield January 19, 2024 22:09

derekxu16 marked this pull request as ready for review January 19, 2024 22:09

dnfield reviewed Jan 22, 2024

View reviewed changes

packages/flutter_tools/lib/src/commands/test.dart Outdated Show resolved Hide resolved

github-actions bot removed a: tests "flutter test", flutter_test, or one of our tests framework flutter/packages/flutter repository. See also f: labels. labels Jan 29, 2024

derekxu16 force-pushed the experimental-faster-testing branch from 976e2c4 to d9fbd9d Compare January 29, 2024 19:14

derekxu16 force-pushed the experimental-faster-testing branch 2 times, most recently from 5c682fa to 32da351 Compare February 14, 2024 19:38

derekxu16 requested a review from dnfield February 14, 2024 20:51

dnfield reviewed Feb 14, 2024

View reviewed changes

packages/flutter_tools/pubspec.yaml Outdated Show resolved Hide resolved

dnfield reviewed Feb 14, 2024

View reviewed changes

dnfield approved these changes Feb 15, 2024

View reviewed changes

derekxu16 added 3 commits February 15, 2024 15:42

Fix generated paths on Windows

6f8fd8c

Add nativeAssetsBuilder parameter to runTestsBySpawningLightweightEng…

7c4d712

…ines

Fix uses of globals.fs.path.join that were not producing valid paths …

c29f084

…on Windows

derekxu16 force-pushed the experimental-faster-testing branch from 32da351 to d234512 Compare February 15, 2024 20:46

github-actions bot added a: tests "flutter test", flutter_test, or one of our tests framework flutter/packages/flutter repository. See also f: labels. labels Feb 15, 2024

derekxu16 force-pushed the experimental-faster-testing branch from d234512 to 2c9e2c6 Compare February 15, 2024 20:48

Address comments

30c2ef8

derekxu16 force-pushed the experimental-faster-testing branch from 2c9e2c6 to 30c2ef8 Compare February 16, 2024 19:18

dnfield and others added 2 commits February 17, 2024 11:47

Merge branch 'master' into experimental-faster-testing

f260b22

Add concurrency paramter to runTestsBySpawningLightweightEngines

b89c157

derekxu16 merged commit dfb5888 into flutter:master Feb 22, 2024

derekxu16 deleted the experimental-faster-testing branch February 22, 2024 18:32

engine-flutter-autoroll mentioned this pull request Feb 23, 2024

Roll Flutter from 41581c9a1b90 to fcd154bd247f (26 revisions) flutter/packages#6188

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Feb 23, 2024

dfb5888 Support using lightweight Flutter Engines to run tests (flutt…

cd3367b

…er/flutter#141726)

engine-flutter-autoroll mentioned this pull request Feb 23, 2024

Roll Flutter from 41581c9a1b90 to 00229827ab7c (27 revisions) flutter/packages#6189

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Feb 23, 2024

dfb5888 Support using lightweight Flutter Engines to run tests (flutt…

a9cc5dd

…er/flutter#141726)

engine-flutter-autoroll mentioned this pull request Feb 23, 2024

Roll Flutter from 41581c9a1b90 to bb856079c401 (28 revisions) flutter/packages#6190

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Feb 23, 2024

dfb5888 Support using lightweight Flutter Engines to run tests (flutt…

d9d6165

…er/flutter#141726)

engine-flutter-autoroll mentioned this pull request Feb 23, 2024

Roll Flutter from 41581c9a1b90 to bab5f85df560 (29 revisions) flutter/packages#6193

Closed

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Feb 23, 2024

dfb5888 Support using lightweight Flutter Engines to run tests (flutt…

697d79c

…er/flutter#141726)

engine-flutter-autoroll mentioned this pull request Feb 23, 2024

Roll Flutter from 41581c9a1b90 to 39585e66c11f (31 revisions) flutter/packages#6194

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request Feb 23, 2024

dfb5888 Support using lightweight Flutter Engines to run tests (flutt…

ec9a5d5

…er/flutter#141726)

derekxu16 mentioned this pull request Mar 8, 2024

Upgrade vm_service. #144845

Merged

engine-flutter-autoroll added a commit to engine-flutter-autoroll/packages that referenced this pull request May 14, 2024

dfb5888 Support using lightweight Flutter Engines to run tests (flutt…

252c5f4

…er/flutter#141726)

derekxu16 mentioned this pull request May 13, 2025

Make flutter_test depend on the packages needed for --experimental-faster-testing #168735

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Support using lightweight Flutter Engines to run tests #141726

Support using lightweight Flutter Engines to run tests #141726

Uh oh!

derekxu16 commented Jan 17, 2024 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

derekxu16 commented Jan 19, 2024

Uh oh!

Uh oh!

derekxu16 commented Feb 5, 2024

Uh oh!

dnfield commented Feb 12, 2024

Uh oh!

derekxu16 commented Feb 14, 2024

Uh oh!

Uh oh!

dnfield left a comment

Uh oh!

dnfield left a comment

Uh oh!

derekxu16 commented Feb 15, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Support using lightweight Flutter Engines to run tests #141726

Support using lightweight Flutter Engines to run tests #141726

Uh oh!

Conversation

derekxu16 commented Jan 17, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

The following are some performance measurements from test runs conducted on GitHub Actions

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

derekxu16 commented Jan 19, 2024

Uh oh!

Uh oh!

derekxu16 commented Feb 5, 2024

Uh oh!

dnfield commented Feb 12, 2024

Uh oh!

derekxu16 commented Feb 14, 2024

Uh oh!

Uh oh!

dnfield left a comment

Choose a reason for hiding this comment

Uh oh!

dnfield left a comment

Choose a reason for hiding this comment

Uh oh!

derekxu16 commented Feb 15, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

derekxu16 commented Jan 17, 2024 •

edited

Loading