Run flutter tests in preview-dart-2 mode on travis flutter builds. #14728

aam · 2018-02-15T17:32:25Z

a-siva · 2018-02-16T02:37:39Z

The changes look good but I am not sure why you are getting test errors in all the shards. Maybe you should sync up and try the tests again to see what state they end up in.

Hixie · 2018-02-16T02:51:38Z

.travis.yml

  - os: osx
    env: SHARD=docs
+  - os: osx
+    env: SHARD=tests_dart2


this excludes it from osx. i think we want it enabled for osx, no?

Was not sure we want to run them on osx, but enabled now.

Hixie · 2018-02-16T02:52:14Z

dev/bots/test.dart

  await _pubRunTest(path.join(flutterRoot, 'packages', 'flutter_tools'));
  await _pubRunTest(path.join(flutterRoot, 'dev', 'bots'));

  await _runAllDartTests(path.join(flutterRoot, 'dev', 'devicelab'));


can we run dart tests with preview2 too?

Yes, that should be possible.

Hixie · 2018-02-16T02:52:30Z

dev/bots/test.dart

+  await _runFlutterTest(path.join(flutterRoot, 'packages', 'flutter_driver'), options: options);
+  await _runFlutterTest(path.join(flutterRoot, 'packages', 'flutter_test'), options: options);
  await _pubRunTest(path.join(flutterRoot, 'packages', 'flutter_tools'));
  await _pubRunTest(path.join(flutterRoot, 'dev', 'bots'));


how about these?

Not sure how pub run can run in dart2 mode.

We need to file an issue against pub for this.
Currently it has the following options for pub run
Usage: pub run [args...]
-h, --help Print this usage information.
-c, --[no-]checked Enable runtime type checks and assertions.
--mode Mode to run transformers in.
(defaults to "release" for dependencies, "debug" for entrypoint)

it needs another option here '--preview-dart-2' to be consistent that would run in dart2 mode.

Opened dart-lang/sdk#32214

a-siva · 2018-02-17T00:42:42Z

Maybe we should move running under dart2 mode to the Allowed Failure section so that we can land this PR and then fix up the tests.

aam · 2018-02-17T00:47:32Z

Maybe we should move running under dart2 mode to the Allowed Failure section so that we can land this PR and then fix up the tests.

I would have been happy to do that, but I'm seeing some weird timeout issues with running dart2 tests all at once. It looks like machine performance tanks when you run ~30 frontend tests in parallel(15 tests work).

aam · 2018-02-17T01:43:14Z

Maybe we should move running under dart2 mode to the Allowed Failure section so that we can land this PR and then fix up the tests.

I would have been happy to do that, but I'm seeing some weird timeout issues with running dart2 tests all at once. It looks like machine performance tanks when you run ~30 frontend tests in parallel(15 tests work

Limiting concurrency to 4 seems to work fine.

mraleph · 2018-02-17T12:29:45Z

I think we need to investigate reusing a single frontend_server instance for running multiple tests. Spawning a new one per test is likely way too much overhead.

mraleph · 2018-02-17T13:11:38Z

FYI: flutter_driver test failures should be fixed by dart-lang/sdk@eb997b2, I also filed dart-lang/sdk#32185 and dart-lang/sdk#32183 for compilation errors in file and vmservice_client packages.

Actually use single compiler to incrementally recompile all tests executed by 'flutter test'.

a-siva · 2018-02-23T18:28:28Z

packages/flutter_tools/lib/src/test/flutter_platform.dart

+      bool isEmpty = compilationQueue.isEmpty;
+      compilationQueue.add(request);
+      // Only trigger processing if queue was empty - i.e. no other requests
+      // we being processed. This effectively enforces "one compilation request at a time".


no other requests are currently being processed

Thanks, Done.

a-siva · 2018-02-23T18:38:31Z

packages/flutter_tools/lib/src/test/flutter_platform.dart

+          compiler.accept();
+          compiler.reset();
+          request.result.complete(outputPath);
+          // Only remove now when we finished processing the element


in what circumstance would we try recompiling the same test again ? I am wondering why is is necessary to remove the request only after we are done with the compilation.

It's not that we are going to recompile the same test again. We only remove the test from the queue after we are done compiling because that way we know that no other compilation request can start before we exit(no await statements from this point on).

a-siva · 2018-03-01T01:46:23Z

What is the final outcome of this, do we run the Travis tests using the batch compiler? I thought this would cause timeouts as we would exceed the Travis time budgets.

aam · 2018-03-01T01:58:10Z

What is the final outcome of this, do we run the Travis tests using the batch compiler? I thought this would cause timeouts as we would exceed the Travis time budgets.

Correct, this is now blocked on incremental compiler issue revealed in #14931.

aam · 2018-03-01T05:00:26Z

Blocked on dart-lang/sdk#32368

aam · 2018-03-01T16:50:22Z

With dart-lang/sdk#32368 fixed, this PR that shares frontend server between tests travis build in preview-dart-2 takes 26m47s, "only" 20s more than non-preview-dart-2 test, so is ready to be submitted I believe.

mraleph · 2018-03-02T16:33:32Z

It looks more like 3 minutes slower on Mac, but 8 minutes slower on Linux if I am not misreading the output.

I think this warrants investigation, to see if we can do something about it /cc @jensjoha @kmillikin

aam · 2018-03-02T19:26:54Z

as @Hixie pointed out comparing numbers from Build Jobs dashboard does not paint full picture as preview-dart-2 tests don't run all the way to the completion.
Better comparison at this point would be to look at execution of flutter test in packages/flutter which completes successfully in both configurations:

06:13 +2327 ~16: All tests passed! without preview-dart-2 vs
18:28 +2327 ~16: All tests passed! with preview-dart-2

tvolkert · 2018-03-07T00:38:12Z

Looks like this broke internal Google tests - /cc @keertip

tvolkert · 2018-03-07T00:42:03Z

And looks like @jason-simmons already has a fix internally

jensjoha · 2018-03-08T13:02:23Z

For now I have a few comments:

Running flutter test --preview-dart-2 in packages/flutter doesn't work for me at all. The VM crashes because of a deleted file after which all tests time out (after 12 minutes!). I have a fix for that though (fix in dart sdk/pkg/vm; still not uploaded).
When running flutter test my computer uses 100% CPU on all (12) cores, when running flutter test --preview-dart-2 it uses 100% CPU on 1 core and "a little bit" on the other ones.
The tests basically does this: recompile, accept, reset, where reset throws away the compiler inside the frontend_server and creates a new one. That doesn't seem ideal.

After fixing whats talked about in bullet-point #1 above I get

01:10 +2334 ~16: All tests passed! (flutter test --local-engine=host_debug)
08:34 +2333 ~16 -1: Some tests failed. (flutter test --preview-dart-2 --local-engine=host_debug)

…lutter#14728) * Run flutter tests in preview-dart-2 mode on travis flutter builds. * Run dart2 tests on osx. Run dart tests in dart2. * Fix name camelCase * Default options to empty array, rather than null * Troubleshoot failures * More logging * Troubleshoot: run single test * Troubleshoot: run 15 tests * Troubleshoot: run 15 tests with fix * Try limit concurrency to 1 * Limit concurrency for preview-dart-2 tests to 4 * Move dart2 tests to allow_failures section * Reinstate tests_dart_2 shard * Raise concurrency to 8 * Reuse compiler across multiple test runs * Allow to switch entry points when recompiling. Actually use single compiler to incrementally recompile all tests executed by 'flutter test'. * Remove leftover commented code * Fix comment * Lints

googlebot added the cla: yes label Feb 15, 2018

Run flutter tests in preview-dart-2 mode on travis flutter builds.

6fbd363

aam force-pushed the run-test-dart2 branch from 3961fda to 6fbd363 Compare February 15, 2018 17:37

Hixie reviewed Feb 16, 2018

View reviewed changes

aam added 11 commits February 15, 2018 19:54

Merge branch 'master' into run-test-dart2

3220369

Run dart2 tests on osx. Run dart tests in dart2.

c06257f

Fix name camelCase

fcd39f6

Default options to empty array, rather than null

231b355

Merge branch 'master' into run-test-dart2

a049b40

Troubleshoot failures

f755c59

More logging

80b4ac7

Merge branch 'master' into run-test-dart2

cce584a

Troubleshoot: run single test

e17ca07

Troubleshoot: run 15 tests

ada1d8e

Troubleshoot: run 15 tests with fix

01c716e

aam added 2 commits February 16, 2018 17:03

Try limit concurrency to 1

7c12db4

Limit concurrency for preview-dart-2 tests to 4

34384be

aam mentioned this pull request Feb 17, 2018

Implement --preview-dart-2 option for pub run dart-lang/sdk#32214

Closed

aam added 3 commits February 16, 2018 17:43

Move dart2 tests to allow_failures section

af42f57

Reinstate tests_dart_2 shard

1112b2f

Raise concurrency to 8

2c1ef90

Merge branch 'master' into run-test-dart2

3423dde

This was referenced Feb 20, 2018

Add the ability to spawn an isolate in Dart 2 preview mode dart-lang/sdk#32253

Closed

Implement --preview-dart-2 option for pub run dart-lang/pub#1807

Closed

aam added 4 commits February 22, 2018 16:26

Merge

bbbe9c9

Reuse compiler across multiple test runs

7adf1d2

Allow to switch entry points when recompiling.

8338b7b

Actually use single compiler to incrementally recompile all tests executed by 'flutter test'.

Remove leftover commented code

651bf67

a-siva reviewed Feb 23, 2018

View reviewed changes

Fix comment

9eafd36

a-siva approved these changes Feb 23, 2018

View reviewed changes

Merge branch 'master' into run-test-dart2

2cacbe3

This was referenced Feb 28, 2018

When doing flutter test use batch compiler and link platform in. #14983

Merged

Another try to run tests in --preview-dart-2 mode. #14989

Closed

aam added 3 commits February 28, 2018 21:04

Merge

0082ee4

Lints

12eaab4

Merge branch 'master' into run-test-dart2

e647d16

aam merged commit d379762 into flutter:master Mar 1, 2018

aam deleted the run-test-dart2 branch March 1, 2018 17:04

JekCharlsonYu mentioned this pull request Mar 1, 2018

Meta Issue for Flutter Test Run (14728) dart-lang/sdk#32381

Closed

github-actions bot locked as resolved and limited conversation to collaborators Aug 11, 2021

Run flutter tests in preview-dart-2 mode on travis flutter builds. #14728

Run flutter tests in preview-dart-2 mode on travis flutter builds. #14728

Uh oh!

Conversation

aam commented Feb 15, 2018

Uh oh!

a-siva commented Feb 16, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

a-siva commented Feb 17, 2018

Uh oh!

aam commented Feb 17, 2018

Uh oh!

aam commented Feb 17, 2018

Uh oh!

mraleph commented Feb 17, 2018

Uh oh!

mraleph commented Feb 17, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

a-siva commented Mar 1, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aam commented Mar 1, 2018

Uh oh!

aam commented Mar 1, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

aam commented Mar 1, 2018

Uh oh!

mraleph commented Mar 2, 2018

Uh oh!

aam commented Mar 2, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tvolkert commented Mar 7, 2018

Uh oh!

tvolkert commented Mar 7, 2018

Uh oh!

jensjoha commented Mar 8, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

a-siva commented Mar 1, 2018 •

edited

Loading

aam commented Mar 1, 2018 •

edited

Loading

aam commented Mar 2, 2018 •

edited

Loading