Repository navigation
Commit 7e4332b
authored
Web Benchmarks Part 4: Standalone Local CLI Runner (flutter#194782)
Adds a standalone local CLI runner
(`dev/devicelab/bin/run_local_web_benchmark.dart`) for running Flutter
Web macrobenchmarks locally and in automated workflows without full
Devicelab task boilerplate:
- Supports `--renderer` (`canvaskit`, `skwasm`, `skwasm_st`),
`--benchmark`, `--filter`, `--smoke-test` (the three selectors are
mutually exclusive), `--[no-]clean`, `--no-build`, `--local-web-sdk`,
`--local-engine-src-path`, `--results-file`, `--baseline`,
`--regression-threshold`, and `--verbose`.
- Extends `runWebBenchmark` in
`dev/devicelab/lib/tasks/web_benchmarks.dart` with optional `noBuild`,
`localWebSdk`, `localEngineSrcPath`, and `verbose` parameters while
preserving the `WebBenchmarkOptions` record typedef.
- Validates CLI arguments, prebuilt artifacts (`--no-build`), and
baseline JSON files upfront with distinct exit codes (`0` pass, `1`
regression threshold exceeded, `2` usage/validation error, `3`
benchmark/infrastructure failure).
- Whenever `--baseline` is provided, compares the `.average` metrics
present in both the current run and the baseline using `compareResults`,
and prints the comparison table, colored only when stdout supports ANSI
escapes. It exits `2` if the two runs have no metrics in common or a
compared value is invalid.
- Uses `kSmokeTestBenchmarks` for `--smoke-test`, and adds hermetic CLI
unit tests in `dev/devicelab/test/tasks/web_benchmarks_test.dart`.
## Local verification
I ran the CLI on Linux (headless Chrome 155, CanvasKit, profile build)
from `dev/devicelab`. The numbers below come from an uncalibrated
workstation and show only that the CLI works. They are not meaningful
performance data.
**1. Build, run the smoke-test benchmarks and save a baseline** (exit
code `0`, about 65s including the web build):
```
$ dart bin/run_local_web_benchmark.dart --smoke-test --results-file /tmp/wb/run1.json
...
✓ Built build/web
Launching Chrome.
Launching benchmark "draw_rect"
Extracted 100 measured frames.
Launching benchmark "bench_card_infinite_scroll"
Extracted 100 measured frames.
bench_card_infinite_scroll.canvaskit.apply_frame.average: 67.17708333333333
bench_card_infinite_scroll.canvaskit.drawFrameDuration.average: 1338.2291666666667
bench_card_infinite_scroll.canvaskit.preroll_frame.average: 185.81720430107526
bench_card_infinite_scroll.canvaskit.totalUiFrame.average: 1772
draw_rect.canvaskit.drawFrameDuration.average: 4915.694736842105
draw_rect.canvaskit.sceneBuildDuration.average: 2.2142857142857144
draw_rect.canvaskit.totalUiFrame.average: 5218
draw_rect.canvaskit.windowRenderDuration.average: 1564.59375
```
`run1.json` holds all 26 metrics that `TaskResult.data` reports,
including the `.noise` and `.outlierAverage` metrics.
**2. Reuse the build and compare against the baseline** (exit code `0`,
about 15s):
```
$ dart bin/run_local_web_benchmark.dart --smoke-test --no-build --baseline /tmp/wb/run1.json --results-file /tmp/wb/run5.json
...
Metric Baseline Current Delta Status
-------------------------------------------------------------- -------- ------- ------ ------
bench_card_infinite_scroll.canvaskit.apply_frame.average 67.2 67.4 0.3% PASS
bench_card_infinite_scroll.canvaskit.drawFrameDuration.average 1338.2 1265.6 -5.4% PASS
bench_card_infinite_scroll.canvaskit.preroll_frame.average 185.8 181.1 -2.5% PASS
bench_card_infinite_scroll.canvaskit.totalUiFrame.average 1772.0 2310.0 30.4% PASS
draw_rect.canvaskit.drawFrameDuration.average 4915.7 4984.0 1.4% PASS
draw_rect.canvaskit.sceneBuildDuration.average 2.2 1.9 -13.3% PASS
draw_rect.canvaskit.totalUiFrame.average 5218.0 5174.0 -0.8% PASS
draw_rect.canvaskit.windowRenderDuration.average 1564.6 1601.1 2.3% PASS
```
Without `--regression-threshold`, the table is for information only and
the command exits `0`.
**3. Regression gate with a deliberately tight threshold** (exit code
`1`). This run happened while the machine was busy, so the deltas are
large:
```
$ dart bin/run_local_web_benchmark.dart --smoke-test --no-build --baseline /tmp/wb/run1.json --regression-threshold 1
...
Metric Baseline Current Delta Status
-------------------------------------------------------------- -------- ------- ------ ------
bench_card_infinite_scroll.canvaskit.apply_frame.average 67.2 65.6 -2.4% PASS
bench_card_infinite_scroll.canvaskit.drawFrameDuration.average 1338.2 1191.3 -11.0% PASS
bench_card_infinite_scroll.canvaskit.preroll_frame.average 185.8 176.8 -4.9% PASS
bench_card_infinite_scroll.canvaskit.totalUiFrame.average 1772.0 2244.0 26.6% FAIL
draw_rect.canvaskit.drawFrameDuration.average 4915.7 6985.4 42.1% FAIL
draw_rect.canvaskit.sceneBuildDuration.average 2.2 7.2 226.7% FAIL
draw_rect.canvaskit.totalUiFrame.average 5218.0 6414.0 22.9% FAIL
draw_rect.canvaskit.windowRenderDuration.average 1564.6 2595.2 65.9% FAIL
FAILED: 1 or more metrics exceeded the regression threshold of 1.0%.
```
**4. Usage error** (exit code `2`):
```
$ dart bin/run_local_web_benchmark.dart --regression-threshold 5
--regression-threshold requires --baseline.
```
Part of flutter#191316
## Pre-launch Checklist
- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [AI contribution guidelines] and understand my
responsibilities, or I am not using AI tools.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant in-code documentation (doc comments with
`///`).
- [x] If this PR introduces a new feature or capability, I created and
linked a website documentation issue or PR in [flutter/website] (or
verified none is needed).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.
<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[AI contribution guidelines]:
https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[flutter/website]: https://github.com/flutter/website
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md1 parent a5128c7 commit 7e4332b
3 files changed
Lines changed: 538 additions & 4 deletions
File tree
- dev/devicelab
- bin
- lib/tasks
- test/tasks
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
0 commit comments