-
Notifications
You must be signed in to change notification settings - Fork 6k
Add custom test plugin that supports screenshot tests #12079
Conversation
lib/web_ui/dev/environment.dart
Outdated
| io.stderr.writeln('Command-line options not passed to Environment.'); | ||
| io.exit(1); | ||
| } | ||
| if (_environment == null) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
return _environment ??= Environment(); if you wanna be concise :)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
lib/web_ui/dev/environment.dart
Outdated
|
|
||
| /// Contains various environment variables, such as common file paths and command-line options. | ||
| Environment get environment { | ||
| if (Environment.commandLineArguments == null) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This check already exists in the Environment constructor. Is there a reason for duplicating it here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Done
| final bool isDebug = options['debug']; | ||
| final List<String> targets = options['target']; | ||
|
|
||
| final io.File self = io.File.fromUri(io.Platform.script); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It looks to me like all you need to compute here is the engine src dir. Everything else can be a getter that's derived from engine src.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks! That cleaned it up significantly.
| _expect(allSourceFiles.isNotEmpty, 'Dart source listing of ${environment.webUiRootDir.path} must not be empty.'); | ||
|
|
||
| final List<String> allDartPaths = allSourceFiles.map((f) => f.path).toList(); | ||
| print(allDartPaths.join('\n')); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
lib/web_ui/dev/test.dart
Outdated
| .whereType<io.File>() | ||
| .where((f) => f.path.endsWith('.dart') || f.path.endsWith('.js')) | ||
| .where((f) { | ||
| if (!f.path.endsWith('.dart') || f.path.endsWith('.js')) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is the .js check necessary?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good catch! Copy'n'paste error. The condition should've been fully inverted here. Fixed.
| // Run screenshot tests one at a time. | ||
| for (String testFilePath in screenshotTestFiles) { | ||
| await _runTestBatch(<String>[testFilePath], concurrency: 1, expectFailure: false); | ||
| _checkExitCode(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we use the return value of _runTestBatch instead of relying on the indirect communication between the two functions through io.exitCode?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Unfortunately no. This relies on the package:test's test runner API, which communicates test success and failures by setting the exit code. It kind of makes sense, since on CI you communicate via process exit codes as well. Otherwise, if some place in the code accidentally catches an exception or ignores the returned exit code the CI will incorrectly pass.
|
Ok, this is good to review. In the meantime I'll fix the licenses. |
|
|
||
| // TODO(yjbanov): do not fail Cirrus builds. They currently fail because Chrome produces | ||
| // different pictures. We need to pin Chrome versions and use a fuzzy image | ||
| // comparator. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would look at what flutter_goldens is doing with package:image
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good idea. Looking at it right now.
jonahwilliams
left a comment
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if it works, then LGTM
[email protected]:flutter/engine.git/compare/7ea9884ab00e...5b94c8a git log 7ea9884..5b94c8a --no-merges --oneline 2019-09-11 [email protected] Revert "Roll src/third_party/dart be66176534..ec7ec4ecf7 (37 commits)" (flutter/engine#12223) 2019-09-11 [email protected] Do not generate kernel platform files on topaz tree (flutter/engine#12222) 2019-09-11 [email protected] Don't disable toString in release mode for dart:ui classes (flutter/engine#12204) 2019-09-11 [email protected] Roll fuchsia/sdk/core/linux-amd64 from 7gDBN... to u7Q31... (flutter/engine#12221) 2019-09-11 [email protected] Roll src/third_party/skia d96ef09317d6..120e7d6766e4 (2 commits) (flutter/engine#12220) 2019-09-11 [email protected] Roll fuchsia/sdk/core/mac-amd64 from vDk46... to _nS67... (flutter/engine#12219) 2019-09-11 [email protected] Namespace patched SDK names to not conflict with Topaz (flutter/engine#12218) 2019-09-11 [email protected] Roll src/third_party/dart ca7baa4013..4d5e15abde (29 commits) 2019-09-11 [email protected] Roll src/third_party/skia 14318c140949..d96ef09317d6 (2 commits) (flutter/engine#12216) 2019-09-11 [email protected] Roll src/third_party/skia b23a4f9d9442..14318c140949 (2 commits) (flutter/engine#12215) 2019-09-11 [email protected] Roll src/third_party/skia 26ac0467cb4c..b23a4f9d9442 (2 commits) (flutter/engine#12214) 2019-09-11 [email protected] Roll src/third_party/dart 7bbfd532de..ca7baa4013 (3 commits) 2019-09-11 [email protected] Roll fuchsia/sdk/core/linux-amd64 from R1yqu... to 7gDBN... (flutter/engine#12212) 2019-09-11 [email protected] Roll fuchsia/sdk/core/mac-amd64 from spUG2... to vDk46... (flutter/engine#12210) 2019-09-11 [email protected] Roll buildroot and Fuchsia toolchain and unblock the Fuchsia/Linux autoroller manually. (flutter/engine#12209) 2019-09-11 [email protected] Roll src/third_party/skia 4fe30e15c06c..26ac0467cb4c (2 commits) (flutter/engine#12207) 2019-09-11 [email protected] Only build the x64 variant of Fuchsia on the try-jobs. (flutter/engine#12206) 2019-09-10 [email protected] Don't load Roboto by default (flutter/engine#12205) 2019-09-10 [email protected] Roll src/third_party/dart 300c3333d1..7bbfd532de (5 commits) 2019-09-10 [email protected] Roll src/third_party/skia 66d8006c2bb1..4fe30e15c06c (11 commits) (flutter/engine#12202) 2019-09-10 [email protected] add a convenience CLI tool for building and testing web engine (flutter/engine#12197) 2019-09-10 [email protected] [flutter_runner] Generate symbols for the Dart VM profiler (flutter/engine#12048) 2019-09-10 [email protected] Add custom test plugin that supports screenshot tests (flutter/engine#12079) 2019-09-10 [email protected] Move the Fuchsia tryjob into a its own step and disable LTO. (flutter/engine#12190) 2019-09-10 [email protected] Roll src/third_party/dart 62f78a7abb..300c3333d1 (6 commits) 2019-09-10 [email protected] Roll src/third_party/skia b88894c8811b..66d8006c2bb1 (5 commits) (flutter/engine#12178) 2019-09-10 [email protected] [flutter_runner] Port the accessibility bridge from Topaz (flutter/engine#12054) 2019-09-10 [email protected] Smooth out iOS irregular input events delivery (flutter/engine#11817) 2019-09-10 [email protected] Roll src/third_party/dart ccb6ba948b..62f78a7abb (3 commits) 2019-09-10 [email protected] Make ImageShader implement Shader for web ui (flutter/engine#12161) 2019-09-10 [email protected] Roll src/third_party/dart 2e8d912848..ccb6ba948b (30 commits) 2019-09-10 [email protected] Roll src/third_party/skia 9e5c47936b17..b88894c8811b (3 commits) (flutter/engine#12151) 2019-09-10 [email protected] Roll src/third_party/skia 1bf30ce852e0..9e5c47936b17 (2 commits) (flutter/engine#12129) 2019-09-10 [email protected] Roll src/third_party/skia 8cae1e95a23b..1bf30ce852e0 (2 commits) (flutter/engine#12106) 2019-09-10 [email protected] Roll fuchsia/clang/mac-amd64 from H1Qjc... to HfPKR... (flutter/engine#12088) 2019-09-10 [email protected] Don't launch the observatory by default on each embedder unit-test invocation. (flutter/engine#12087) 2019-09-10 [email protected] Roll src/third_party/skia c2d84bfa7421..8cae1e95a23b (4 commits) (flutter/engine#12086) 2019-09-10 [email protected] Roll src/third_party/dart fb14babf59..2e8d912848 (65 commits) 2019-09-10 [email protected] Guard availability of user notification related methods to iOS 10.0 (flutter/engine#12084) 2019-09-09 [email protected] Add capability to add AppDelegate as UNUserNotificationCenterDelegate (flutter/engine#9864) 2019-09-09 [email protected] Add GradientRadial paintStyle implementation (flutter/engine#12081) 2019-09-09 [email protected] Don't quote generic font families (flutter/engine#12080) 2019-09-09 [email protected] Roll src/third_party/skia 28d40b2e7ade..c2d84bfa7421 (3 commits) (flutter/engine#12082) 2019-09-09 [email protected] Remove ENABLE_BITCODE from Scenarios test app (flutter/engine#11839) 2019-09-09 [email protected] Roll src/third_party/skia 4f2674da4bbc..28d40b2e7ade (4 commits) (flutter/engine#12077) ...
[email protected]:flutter/engine.git/compare/7ea9884ab00e...bbb1f12 git log 7ea9884..bbb1f12 --no-merges --oneline 2019-09-12 [email protected] Adjust iOS frame start times to match the platform info (flutter/engine#11802) 2019-09-12 [email protected] Roll src/third_party/skia 50f377e275c3..7c47d41067d4 (3 commits) (flutter/engine#12231) 2019-09-12 [email protected] Revert "Manage iOS contexts separately (#12078)" (flutter/engine#12233) 2019-09-11 [email protected] Manage iOS contexts separately (flutter/engine#12078) 2019-09-11 [email protected] Roll src/third_party/skia 120e7d6766e4..50f377e275c3 (7 commits) (flutter/engine#12224) 2019-09-11 [email protected] Revert "Roll src/third_party/dart be66176534..ec7ec4ecf7 (37 commits)" (flutter/engine#12223) 2019-09-11 [email protected] Do not generate kernel platform files on topaz tree (flutter/engine#12222) 2019-09-11 [email protected] Don't disable toString in release mode for dart:ui classes (flutter/engine#12204) 2019-09-11 [email protected] Roll fuchsia/sdk/core/linux-amd64 from 7gDBN... to u7Q31... (flutter/engine#12221) 2019-09-11 [email protected] Roll src/third_party/skia d96ef09317d6..120e7d6766e4 (2 commits) (flutter/engine#12220) 2019-09-11 [email protected] Roll fuchsia/sdk/core/mac-amd64 from vDk46... to _nS67... (flutter/engine#12219) 2019-09-11 [email protected] Namespace patched SDK names to not conflict with Topaz (flutter/engine#12218) 2019-09-11 [email protected] Roll src/third_party/dart ca7baa4013..4d5e15abde (29 commits) 2019-09-11 [email protected] Roll src/third_party/skia 14318c140949..d96ef09317d6 (2 commits) (flutter/engine#12216) 2019-09-11 [email protected] Roll src/third_party/skia b23a4f9d9442..14318c140949 (2 commits) (flutter/engine#12215) 2019-09-11 [email protected] Roll src/third_party/skia 26ac0467cb4c..b23a4f9d9442 (2 commits) (flutter/engine#12214) 2019-09-11 [email protected] Roll src/third_party/dart 7bbfd532de..ca7baa4013 (3 commits) 2019-09-11 [email protected] Roll fuchsia/sdk/core/linux-amd64 from R1yqu... to 7gDBN... (flutter/engine#12212) 2019-09-11 [email protected] Roll fuchsia/sdk/core/mac-amd64 from spUG2... to vDk46... (flutter/engine#12210) 2019-09-11 [email protected] Roll buildroot and Fuchsia toolchain and unblock the Fuchsia/Linux autoroller manually. (flutter/engine#12209) 2019-09-11 [email protected] Roll src/third_party/skia 4fe30e15c06c..26ac0467cb4c (2 commits) (flutter/engine#12207) 2019-09-11 [email protected] Only build the x64 variant of Fuchsia on the try-jobs. (flutter/engine#12206) 2019-09-10 [email protected] Don't load Roboto by default (flutter/engine#12205) 2019-09-10 [email protected] Roll src/third_party/dart 300c3333d1..7bbfd532de (5 commits) 2019-09-10 [email protected] Roll src/third_party/skia 66d8006c2bb1..4fe30e15c06c (11 commits) (flutter/engine#12202) 2019-09-10 [email protected] add a convenience CLI tool for building and testing web engine (flutter/engine#12197) 2019-09-10 [email protected] [flutter_runner] Generate symbols for the Dart VM profiler (flutter/engine#12048) 2019-09-10 [email protected] Add custom test plugin that supports screenshot tests (flutter/engine#12079) 2019-09-10 [email protected] Move the Fuchsia tryjob into a its own step and disable LTO. (flutter/engine#12190) 2019-09-10 [email protected] Roll src/third_party/dart 62f78a7abb..300c3333d1 (6 commits) 2019-09-10 [email protected] Roll src/third_party/skia b88894c8811b..66d8006c2bb1 (5 commits) (flutter/engine#12178) 2019-09-10 [email protected] [flutter_runner] Port the accessibility bridge from Topaz (flutter/engine#12054) 2019-09-10 [email protected] Smooth out iOS irregular input events delivery (flutter/engine#11817) 2019-09-10 [email protected] Roll src/third_party/dart ccb6ba948b..62f78a7abb (3 commits) 2019-09-10 [email protected] Make ImageShader implement Shader for web ui (flutter/engine#12161) 2019-09-10 [email protected] Roll src/third_party/dart 2e8d912848..ccb6ba948b (30 commits) 2019-09-10 [email protected] Roll src/third_party/skia 9e5c47936b17..b88894c8811b (3 commits) (flutter/engine#12151) 2019-09-10 [email protected] Roll src/third_party/skia 1bf30ce852e0..9e5c47936b17 (2 commits) (flutter/engine#12129) 2019-09-10 [email protected] Roll src/third_party/skia 8cae1e95a23b..1bf30ce852e0 (2 commits) (flutter/engine#12106) 2019-09-10 [email protected] Roll fuchsia/clang/mac-amd64 from H1Qjc... to HfPKR... (flutter/engine#12088) 2019-09-10 [email protected] Don't launch the observatory by default on each embedder unit-test invocation. (flutter/engine#12087) 2019-09-10 [email protected] Roll src/third_party/skia c2d84bfa7421..8cae1e95a23b (4 commits) (flutter/engine#12086) 2019-09-10 [email protected] Roll src/third_party/dart fb14babf59..2e8d912848 (65 commits) 2019-09-10 [email protected] Guard availability of user notification related methods to iOS 10.0 (flutter/engine#12084) 2019-09-09 [email protected] Add capability to add AppDelegate as UNUserNotificationCenterDelegate (flutter/engine#9864) ...
[email protected]:flutter/engine.git/compare/7ea9884ab00e...bbb1f12 git log 7ea9884..bbb1f12 --no-merges --oneline 2019-09-12 [email protected] Adjust iOS frame start times to match the platform info (flutter/engine#11802) 2019-09-12 [email protected] Roll src/third_party/skia 50f377e275c3..7c47d41067d4 (3 commits) (flutter/engine#12231) 2019-09-12 [email protected] Revert "Manage iOS contexts separately (flutter#12078)" (flutter/engine#12233) 2019-09-11 [email protected] Manage iOS contexts separately (flutter/engine#12078) 2019-09-11 [email protected] Roll src/third_party/skia 120e7d6766e4..50f377e275c3 (7 commits) (flutter/engine#12224) 2019-09-11 [email protected] Revert "Roll src/third_party/dart be66176534..ec7ec4ecf7 (37 commits)" (flutter/engine#12223) 2019-09-11 [email protected] Do not generate kernel platform files on topaz tree (flutter/engine#12222) 2019-09-11 [email protected] Don't disable toString in release mode for dart:ui classes (flutter/engine#12204) 2019-09-11 [email protected] Roll fuchsia/sdk/core/linux-amd64 from 7gDBN... to u7Q31... (flutter/engine#12221) 2019-09-11 [email protected] Roll src/third_party/skia d96ef09317d6..120e7d6766e4 (2 commits) (flutter/engine#12220) 2019-09-11 [email protected] Roll fuchsia/sdk/core/mac-amd64 from vDk46... to _nS67... (flutter/engine#12219) 2019-09-11 [email protected] Namespace patched SDK names to not conflict with Topaz (flutter/engine#12218) 2019-09-11 [email protected] Roll src/third_party/dart ca7baa4013..4d5e15abde (29 commits) 2019-09-11 [email protected] Roll src/third_party/skia 14318c140949..d96ef09317d6 (2 commits) (flutter/engine#12216) 2019-09-11 [email protected] Roll src/third_party/skia b23a4f9d9442..14318c140949 (2 commits) (flutter/engine#12215) 2019-09-11 [email protected] Roll src/third_party/skia 26ac0467cb4c..b23a4f9d9442 (2 commits) (flutter/engine#12214) 2019-09-11 [email protected] Roll src/third_party/dart 7bbfd532de..ca7baa4013 (3 commits) 2019-09-11 [email protected] Roll fuchsia/sdk/core/linux-amd64 from R1yqu... to 7gDBN... (flutter/engine#12212) 2019-09-11 [email protected] Roll fuchsia/sdk/core/mac-amd64 from spUG2... to vDk46... (flutter/engine#12210) 2019-09-11 [email protected] Roll buildroot and Fuchsia toolchain and unblock the Fuchsia/Linux autoroller manually. (flutter/engine#12209) 2019-09-11 [email protected] Roll src/third_party/skia 4fe30e15c06c..26ac0467cb4c (2 commits) (flutter/engine#12207) 2019-09-11 [email protected] Only build the x64 variant of Fuchsia on the try-jobs. (flutter/engine#12206) 2019-09-10 [email protected] Don't load Roboto by default (flutter/engine#12205) 2019-09-10 [email protected] Roll src/third_party/dart 300c3333d1..7bbfd532de (5 commits) 2019-09-10 [email protected] Roll src/third_party/skia 66d8006c2bb1..4fe30e15c06c (11 commits) (flutter/engine#12202) 2019-09-10 [email protected] add a convenience CLI tool for building and testing web engine (flutter/engine#12197) 2019-09-10 [email protected] [flutter_runner] Generate symbols for the Dart VM profiler (flutter/engine#12048) 2019-09-10 [email protected] Add custom test plugin that supports screenshot tests (flutter/engine#12079) 2019-09-10 [email protected] Move the Fuchsia tryjob into a its own step and disable LTO. (flutter/engine#12190) 2019-09-10 [email protected] Roll src/third_party/dart 62f78a7abb..300c3333d1 (6 commits) 2019-09-10 [email protected] Roll src/third_party/skia b88894c8811b..66d8006c2bb1 (5 commits) (flutter/engine#12178) 2019-09-10 [email protected] [flutter_runner] Port the accessibility bridge from Topaz (flutter/engine#12054) 2019-09-10 [email protected] Smooth out iOS irregular input events delivery (flutter/engine#11817) 2019-09-10 [email protected] Roll src/third_party/dart ccb6ba948b..62f78a7abb (3 commits) 2019-09-10 [email protected] Make ImageShader implement Shader for web ui (flutter/engine#12161) 2019-09-10 [email protected] Roll src/third_party/dart 2e8d912848..ccb6ba948b (30 commits) 2019-09-10 [email protected] Roll src/third_party/skia 9e5c47936b17..b88894c8811b (3 commits) (flutter/engine#12151) 2019-09-10 [email protected] Roll src/third_party/skia 1bf30ce852e0..9e5c47936b17 (2 commits) (flutter/engine#12129) 2019-09-10 [email protected] Roll src/third_party/skia 8cae1e95a23b..1bf30ce852e0 (2 commits) (flutter/engine#12106) 2019-09-10 [email protected] Roll fuchsia/clang/mac-amd64 from H1Qjc... to HfPKR... (flutter/engine#12088) 2019-09-10 [email protected] Don't launch the observatory by default on each embedder unit-test invocation. (flutter/engine#12087) 2019-09-10 [email protected] Roll src/third_party/skia c2d84bfa7421..8cae1e95a23b (4 commits) (flutter/engine#12086) 2019-09-10 [email protected] Roll src/third_party/dart fb14babf59..2e8d912848 (65 commits) 2019-09-10 [email protected] Guard availability of user notification related methods to iOS 10.0 (flutter/engine#12084) 2019-09-09 [email protected] Add capability to add AppDelegate as UNUserNotificationCenterDelegate (flutter/engine#9864) ...
PlatformPluginthat spins up a server waiting for test to request a screenshot.dev/test.dartwith anArgParserfor future extension into a proper developer tool for the team.--debugoption to launch Chrome in debug mode, a--targetoption to choose a single test to run rather than all tests, and--shardoption to choose a subset of tests to run.Limitations: