Skip to content
This repository was archived by the owner on Feb 25, 2025. It is now read-only.

Conversation

@yjbanov
Copy link
Contributor

@yjbanov yjbanov commented Sep 9, 2019

  • Add a custom PlatformPlugin that spins up a server waiting for test to request a screenshot.
    • When a screenshot is requested the plugin talks to Chrome via the debug port, captures a screenshot, and compares it with a golden file
  • This PR also adds proper CLI for dev/test.dart with an ArgParser for future extension into a proper developer tool for the team.
    • As a first couple of features, it adds --debug option to launch Chrome in debug mode, a --target option to choose a single test to run rather than all tests, and --shard option to choose a subset of tests to run.

Limitations:

  • While screenshot test will run on Cirrus, they are configured to not fail on Cirrus. Need to solve Chrome version skew.
  • Stack maps do not work yet (you get stacks, but they are nonsensical)
  • When requesting a single test, build_runner builds all tests anyway
  • Nothing but desktop Chrome is supported
  • Nothing but Linux is supported
  • There's no Chrome version pinning; currently assumed stable Chrome channel

@yjbanov yjbanov added the Work in progress (WIP) Not ready (yet) for review! label Sep 9, 2019
io.stderr.writeln('Command-line options not passed to Environment.');
io.exit(1);
}
if (_environment == null) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

return _environment ??= Environment(); if you wanna be concise :)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


/// Contains various environment variables, such as common file paths and command-line options.
Environment get environment {
if (Environment.commandLineArguments == null) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This check already exists in the Environment constructor. Is there a reason for duplicating it here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

final bool isDebug = options['debug'];
final List<String> targets = options['target'];

final io.File self = io.File.fromUri(io.Platform.script);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks to me like all you need to compute here is the engine src dir. Everything else can be a getter that's derived from engine src.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! That cleaned it up significantly.

_expect(allSourceFiles.isNotEmpty, 'Dart source listing of ${environment.webUiRootDir.path} must not be empty.');

final List<String> allDartPaths = allSourceFiles.map((f) => f.path).toList();
print(allDartPaths.join('\n'));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

.whereType<io.File>()
.where((f) => f.path.endsWith('.dart') || f.path.endsWith('.js'))
.where((f) {
if (!f.path.endsWith('.dart') || f.path.endsWith('.js')) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the .js check necessary?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch! Copy'n'paste error. The condition should've been fully inverted here. Fixed.

// Run screenshot tests one at a time.
for (String testFilePath in screenshotTestFiles) {
await _runTestBatch(<String>[testFilePath], concurrency: 1, expectFailure: false);
_checkExitCode();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we use the return value of _runTestBatch instead of relying on the indirect communication between the two functions through io.exitCode?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Unfortunately no. This relies on the package:test's test runner API, which communicates test success and failures by setting the exit code. It kind of makes sense, since on CI you communicate via process exit codes as well. Otherwise, if some place in the code accidentally catches an exception or ignores the returned exit code the CI will incorrectly pass.

@yjbanov yjbanov changed the title [WIP]: Add custom test plugin that supports screenshot tests Add custom test plugin that supports screenshot tests Sep 9, 2019
@yjbanov yjbanov removed the Work in progress (WIP) Not ready (yet) for review! label Sep 9, 2019
@yjbanov
Copy link
Contributor Author

yjbanov commented Sep 9, 2019

Ok, this is good to review. In the meantime I'll fix the licenses.


// TODO(yjbanov): do not fail Cirrus builds. They currently fail because Chrome produces
// different pictures. We need to pin Chrome versions and use a fuzzy image
// comparator.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would look at what flutter_goldens is doing with package:image

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea. Looking at it right now.

Copy link
Contributor

@jonahwilliams jonahwilliams left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if it works, then LGTM

@yjbanov yjbanov requested a review from mdebbar September 10, 2019 19:58
engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Sep 11, 2019
engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Sep 11, 2019
engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Sep 11, 2019
engine-flutter-autoroll added a commit to flutter/flutter that referenced this pull request Sep 11, 2019
[email protected]:flutter/engine.git/compare/7ea9884ab00e...5b94c8a

git log 7ea9884..5b94c8a --no-merges --oneline
2019-09-11 [email protected] Revert "Roll src/third_party/dart be66176534..ec7ec4ecf7 (37 commits)" (flutter/engine#12223)
2019-09-11 [email protected] Do not generate kernel platform files on topaz tree (flutter/engine#12222)
2019-09-11 [email protected] Don't disable toString in release mode for dart:ui classes (flutter/engine#12204)
2019-09-11 [email protected] Roll fuchsia/sdk/core/linux-amd64 from 7gDBN... to u7Q31... (flutter/engine#12221)
2019-09-11 [email protected] Roll src/third_party/skia d96ef09317d6..120e7d6766e4 (2 commits) (flutter/engine#12220)
2019-09-11 [email protected] Roll fuchsia/sdk/core/mac-amd64 from vDk46... to _nS67... (flutter/engine#12219)
2019-09-11 [email protected] Namespace patched SDK names to not conflict with Topaz (flutter/engine#12218)
2019-09-11 [email protected] Roll src/third_party/dart ca7baa4013..4d5e15abde (29 commits)
2019-09-11 [email protected] Roll src/third_party/skia 14318c140949..d96ef09317d6 (2 commits) (flutter/engine#12216)
2019-09-11 [email protected] Roll src/third_party/skia b23a4f9d9442..14318c140949 (2 commits) (flutter/engine#12215)
2019-09-11 [email protected] Roll src/third_party/skia 26ac0467cb4c..b23a4f9d9442 (2 commits) (flutter/engine#12214)
2019-09-11 [email protected] Roll src/third_party/dart 7bbfd532de..ca7baa4013 (3 commits)
2019-09-11 [email protected] Roll fuchsia/sdk/core/linux-amd64 from R1yqu... to 7gDBN... (flutter/engine#12212)
2019-09-11 [email protected] Roll fuchsia/sdk/core/mac-amd64 from spUG2... to vDk46... (flutter/engine#12210)
2019-09-11 [email protected] Roll buildroot and Fuchsia toolchain and unblock the Fuchsia/Linux autoroller manually. (flutter/engine#12209)
2019-09-11 [email protected] Roll src/third_party/skia 4fe30e15c06c..26ac0467cb4c (2 commits) (flutter/engine#12207)
2019-09-11 [email protected] Only build the x64 variant of Fuchsia on the try-jobs. (flutter/engine#12206)
2019-09-10 [email protected] Don't load Roboto by default (flutter/engine#12205)
2019-09-10 [email protected] Roll src/third_party/dart 300c3333d1..7bbfd532de (5 commits)
2019-09-10 [email protected] Roll src/third_party/skia 66d8006c2bb1..4fe30e15c06c (11 commits) (flutter/engine#12202)
2019-09-10 [email protected] add a convenience CLI tool for building and testing web engine (flutter/engine#12197)
2019-09-10 [email protected] [flutter_runner] Generate symbols for the Dart VM profiler (flutter/engine#12048)
2019-09-10 [email protected] Add custom test plugin that supports screenshot tests (flutter/engine#12079)
2019-09-10 [email protected] Move the Fuchsia tryjob into a its own step and disable LTO. (flutter/engine#12190)
2019-09-10 [email protected] Roll src/third_party/dart 62f78a7abb..300c3333d1 (6 commits)
2019-09-10 [email protected] Roll src/third_party/skia b88894c8811b..66d8006c2bb1 (5 commits) (flutter/engine#12178)
2019-09-10 [email protected] [flutter_runner] Port the accessibility bridge from Topaz (flutter/engine#12054)
2019-09-10 [email protected] Smooth out iOS irregular input events delivery (flutter/engine#11817)
2019-09-10 [email protected] Roll src/third_party/dart ccb6ba948b..62f78a7abb (3 commits)
2019-09-10 [email protected] Make ImageShader implement Shader for web ui (flutter/engine#12161)
2019-09-10 [email protected] Roll src/third_party/dart 2e8d912848..ccb6ba948b (30 commits)
2019-09-10 [email protected] Roll src/third_party/skia 9e5c47936b17..b88894c8811b (3 commits) (flutter/engine#12151)
2019-09-10 [email protected] Roll src/third_party/skia 1bf30ce852e0..9e5c47936b17 (2 commits) (flutter/engine#12129)
2019-09-10 [email protected] Roll src/third_party/skia 8cae1e95a23b..1bf30ce852e0 (2 commits) (flutter/engine#12106)
2019-09-10 [email protected] Roll fuchsia/clang/mac-amd64 from H1Qjc... to HfPKR... (flutter/engine#12088)
2019-09-10 [email protected] Don't launch the observatory by default on each embedder unit-test invocation. (flutter/engine#12087)
2019-09-10 [email protected] Roll src/third_party/skia c2d84bfa7421..8cae1e95a23b (4 commits) (flutter/engine#12086)
2019-09-10 [email protected] Roll src/third_party/dart fb14babf59..2e8d912848 (65 commits)
2019-09-10 [email protected] Guard availability of user notification related methods to iOS 10.0 (flutter/engine#12084)
2019-09-09 [email protected] Add capability to add AppDelegate as UNUserNotificationCenterDelegate (flutter/engine#9864)
2019-09-09 [email protected] Add GradientRadial paintStyle implementation (flutter/engine#12081)
2019-09-09 [email protected] Don't quote generic font families (flutter/engine#12080)
2019-09-09 [email protected] Roll src/third_party/skia 28d40b2e7ade..c2d84bfa7421 (3 commits) (flutter/engine#12082)
2019-09-09 [email protected] Remove ENABLE_BITCODE from Scenarios test app (flutter/engine#11839)
2019-09-09 [email protected] Roll src/third_party/skia 4f2674da4bbc..28d40b2e7ade (4 commits) (flutter/engine#12077)
...
engine-flutter-autoroll added a commit to engine-flutter-autoroll/flutter that referenced this pull request Sep 12, 2019
engine-flutter-autoroll added a commit to flutter/flutter that referenced this pull request Sep 12, 2019
[email protected]:flutter/engine.git/compare/7ea9884ab00e...bbb1f12

git log 7ea9884..bbb1f12 --no-merges --oneline
2019-09-12 [email protected] Adjust iOS frame start times to match the platform info (flutter/engine#11802)
2019-09-12 [email protected] Roll src/third_party/skia 50f377e275c3..7c47d41067d4 (3 commits) (flutter/engine#12231)
2019-09-12 [email protected] Revert "Manage iOS contexts separately (#12078)" (flutter/engine#12233)
2019-09-11 [email protected] Manage iOS contexts separately (flutter/engine#12078)
2019-09-11 [email protected] Roll src/third_party/skia 120e7d6766e4..50f377e275c3 (7 commits) (flutter/engine#12224)
2019-09-11 [email protected] Revert "Roll src/third_party/dart be66176534..ec7ec4ecf7 (37 commits)" (flutter/engine#12223)
2019-09-11 [email protected] Do not generate kernel platform files on topaz tree (flutter/engine#12222)
2019-09-11 [email protected] Don't disable toString in release mode for dart:ui classes (flutter/engine#12204)
2019-09-11 [email protected] Roll fuchsia/sdk/core/linux-amd64 from 7gDBN... to u7Q31... (flutter/engine#12221)
2019-09-11 [email protected] Roll src/third_party/skia d96ef09317d6..120e7d6766e4 (2 commits) (flutter/engine#12220)
2019-09-11 [email protected] Roll fuchsia/sdk/core/mac-amd64 from vDk46... to _nS67... (flutter/engine#12219)
2019-09-11 [email protected] Namespace patched SDK names to not conflict with Topaz (flutter/engine#12218)
2019-09-11 [email protected] Roll src/third_party/dart ca7baa4013..4d5e15abde (29 commits)
2019-09-11 [email protected] Roll src/third_party/skia 14318c140949..d96ef09317d6 (2 commits) (flutter/engine#12216)
2019-09-11 [email protected] Roll src/third_party/skia b23a4f9d9442..14318c140949 (2 commits) (flutter/engine#12215)
2019-09-11 [email protected] Roll src/third_party/skia 26ac0467cb4c..b23a4f9d9442 (2 commits) (flutter/engine#12214)
2019-09-11 [email protected] Roll src/third_party/dart 7bbfd532de..ca7baa4013 (3 commits)
2019-09-11 [email protected] Roll fuchsia/sdk/core/linux-amd64 from R1yqu... to 7gDBN... (flutter/engine#12212)
2019-09-11 [email protected] Roll fuchsia/sdk/core/mac-amd64 from spUG2... to vDk46... (flutter/engine#12210)
2019-09-11 [email protected] Roll buildroot and Fuchsia toolchain and unblock the Fuchsia/Linux autoroller manually. (flutter/engine#12209)
2019-09-11 [email protected] Roll src/third_party/skia 4fe30e15c06c..26ac0467cb4c (2 commits) (flutter/engine#12207)
2019-09-11 [email protected] Only build the x64 variant of Fuchsia on the try-jobs. (flutter/engine#12206)
2019-09-10 [email protected] Don't load Roboto by default (flutter/engine#12205)
2019-09-10 [email protected] Roll src/third_party/dart 300c3333d1..7bbfd532de (5 commits)
2019-09-10 [email protected] Roll src/third_party/skia 66d8006c2bb1..4fe30e15c06c (11 commits) (flutter/engine#12202)
2019-09-10 [email protected] add a convenience CLI tool for building and testing web engine (flutter/engine#12197)
2019-09-10 [email protected] [flutter_runner] Generate symbols for the Dart VM profiler (flutter/engine#12048)
2019-09-10 [email protected] Add custom test plugin that supports screenshot tests (flutter/engine#12079)
2019-09-10 [email protected] Move the Fuchsia tryjob into a its own step and disable LTO. (flutter/engine#12190)
2019-09-10 [email protected] Roll src/third_party/dart 62f78a7abb..300c3333d1 (6 commits)
2019-09-10 [email protected] Roll src/third_party/skia b88894c8811b..66d8006c2bb1 (5 commits) (flutter/engine#12178)
2019-09-10 [email protected] [flutter_runner] Port the accessibility bridge from Topaz (flutter/engine#12054)
2019-09-10 [email protected] Smooth out iOS irregular input events delivery (flutter/engine#11817)
2019-09-10 [email protected] Roll src/third_party/dart ccb6ba948b..62f78a7abb (3 commits)
2019-09-10 [email protected] Make ImageShader implement Shader for web ui (flutter/engine#12161)
2019-09-10 [email protected] Roll src/third_party/dart 2e8d912848..ccb6ba948b (30 commits)
2019-09-10 [email protected] Roll src/third_party/skia 9e5c47936b17..b88894c8811b (3 commits) (flutter/engine#12151)
2019-09-10 [email protected] Roll src/third_party/skia 1bf30ce852e0..9e5c47936b17 (2 commits) (flutter/engine#12129)
2019-09-10 [email protected] Roll src/third_party/skia 8cae1e95a23b..1bf30ce852e0 (2 commits) (flutter/engine#12106)
2019-09-10 [email protected] Roll fuchsia/clang/mac-amd64 from H1Qjc... to HfPKR... (flutter/engine#12088)
2019-09-10 [email protected] Don't launch the observatory by default on each embedder unit-test invocation. (flutter/engine#12087)
2019-09-10 [email protected] Roll src/third_party/skia c2d84bfa7421..8cae1e95a23b (4 commits) (flutter/engine#12086)
2019-09-10 [email protected] Roll src/third_party/dart fb14babf59..2e8d912848 (65 commits)
2019-09-10 [email protected] Guard availability of user notification related methods to iOS 10.0 (flutter/engine#12084)
2019-09-09 [email protected] Add capability to add AppDelegate as UNUserNotificationCenterDelegate (flutter/engine#9864)
...
Inconnu08 pushed a commit to Inconnu08/flutter that referenced this pull request Sep 30, 2019
[email protected]:flutter/engine.git/compare/7ea9884ab00e...bbb1f12

git log 7ea9884..bbb1f12 --no-merges --oneline
2019-09-12 [email protected] Adjust iOS frame start times to match the platform info (flutter/engine#11802)
2019-09-12 [email protected] Roll src/third_party/skia 50f377e275c3..7c47d41067d4 (3 commits) (flutter/engine#12231)
2019-09-12 [email protected] Revert "Manage iOS contexts separately (flutter#12078)" (flutter/engine#12233)
2019-09-11 [email protected] Manage iOS contexts separately (flutter/engine#12078)
2019-09-11 [email protected] Roll src/third_party/skia 120e7d6766e4..50f377e275c3 (7 commits) (flutter/engine#12224)
2019-09-11 [email protected] Revert "Roll src/third_party/dart be66176534..ec7ec4ecf7 (37 commits)" (flutter/engine#12223)
2019-09-11 [email protected] Do not generate kernel platform files on topaz tree (flutter/engine#12222)
2019-09-11 [email protected] Don't disable toString in release mode for dart:ui classes (flutter/engine#12204)
2019-09-11 [email protected] Roll fuchsia/sdk/core/linux-amd64 from 7gDBN... to u7Q31... (flutter/engine#12221)
2019-09-11 [email protected] Roll src/third_party/skia d96ef09317d6..120e7d6766e4 (2 commits) (flutter/engine#12220)
2019-09-11 [email protected] Roll fuchsia/sdk/core/mac-amd64 from vDk46... to _nS67... (flutter/engine#12219)
2019-09-11 [email protected] Namespace patched SDK names to not conflict with Topaz (flutter/engine#12218)
2019-09-11 [email protected] Roll src/third_party/dart ca7baa4013..4d5e15abde (29 commits)
2019-09-11 [email protected] Roll src/third_party/skia 14318c140949..d96ef09317d6 (2 commits) (flutter/engine#12216)
2019-09-11 [email protected] Roll src/third_party/skia b23a4f9d9442..14318c140949 (2 commits) (flutter/engine#12215)
2019-09-11 [email protected] Roll src/third_party/skia 26ac0467cb4c..b23a4f9d9442 (2 commits) (flutter/engine#12214)
2019-09-11 [email protected] Roll src/third_party/dart 7bbfd532de..ca7baa4013 (3 commits)
2019-09-11 [email protected] Roll fuchsia/sdk/core/linux-amd64 from R1yqu... to 7gDBN... (flutter/engine#12212)
2019-09-11 [email protected] Roll fuchsia/sdk/core/mac-amd64 from spUG2... to vDk46... (flutter/engine#12210)
2019-09-11 [email protected] Roll buildroot and Fuchsia toolchain and unblock the Fuchsia/Linux autoroller manually. (flutter/engine#12209)
2019-09-11 [email protected] Roll src/third_party/skia 4fe30e15c06c..26ac0467cb4c (2 commits) (flutter/engine#12207)
2019-09-11 [email protected] Only build the x64 variant of Fuchsia on the try-jobs. (flutter/engine#12206)
2019-09-10 [email protected] Don't load Roboto by default (flutter/engine#12205)
2019-09-10 [email protected] Roll src/third_party/dart 300c3333d1..7bbfd532de (5 commits)
2019-09-10 [email protected] Roll src/third_party/skia 66d8006c2bb1..4fe30e15c06c (11 commits) (flutter/engine#12202)
2019-09-10 [email protected] add a convenience CLI tool for building and testing web engine (flutter/engine#12197)
2019-09-10 [email protected] [flutter_runner] Generate symbols for the Dart VM profiler (flutter/engine#12048)
2019-09-10 [email protected] Add custom test plugin that supports screenshot tests (flutter/engine#12079)
2019-09-10 [email protected] Move the Fuchsia tryjob into a its own step and disable LTO. (flutter/engine#12190)
2019-09-10 [email protected] Roll src/third_party/dart 62f78a7abb..300c3333d1 (6 commits)
2019-09-10 [email protected] Roll src/third_party/skia b88894c8811b..66d8006c2bb1 (5 commits) (flutter/engine#12178)
2019-09-10 [email protected] [flutter_runner] Port the accessibility bridge from Topaz (flutter/engine#12054)
2019-09-10 [email protected] Smooth out iOS irregular input events delivery (flutter/engine#11817)
2019-09-10 [email protected] Roll src/third_party/dart ccb6ba948b..62f78a7abb (3 commits)
2019-09-10 [email protected] Make ImageShader implement Shader for web ui (flutter/engine#12161)
2019-09-10 [email protected] Roll src/third_party/dart 2e8d912848..ccb6ba948b (30 commits)
2019-09-10 [email protected] Roll src/third_party/skia 9e5c47936b17..b88894c8811b (3 commits) (flutter/engine#12151)
2019-09-10 [email protected] Roll src/third_party/skia 1bf30ce852e0..9e5c47936b17 (2 commits) (flutter/engine#12129)
2019-09-10 [email protected] Roll src/third_party/skia 8cae1e95a23b..1bf30ce852e0 (2 commits) (flutter/engine#12106)
2019-09-10 [email protected] Roll fuchsia/clang/mac-amd64 from H1Qjc... to HfPKR... (flutter/engine#12088)
2019-09-10 [email protected] Don't launch the observatory by default on each embedder unit-test invocation. (flutter/engine#12087)
2019-09-10 [email protected] Roll src/third_party/skia c2d84bfa7421..8cae1e95a23b (4 commits) (flutter/engine#12086)
2019-09-10 [email protected] Roll src/third_party/dart fb14babf59..2e8d912848 (65 commits)
2019-09-10 [email protected] Guard availability of user notification related methods to iOS 10.0 (flutter/engine#12084)
2019-09-09 [email protected] Add capability to add AppDelegate as UNUserNotificationCenterDelegate (flutter/engine#9864)
...
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants