Web Benchmarks Part 4: Standalone Local CLI Runner by harryterkelsen · Pull Request #194782 · flutter/flutter

harryterkelsen · 2026-10-19T18:42:36Z

Adds a standalone local CLI runner (dev/devicelab/bin/run_local_web_benchmark.dart) for running Flutter Web macrobenchmarks locally and in automated workflows without full Devicelab task boilerplate:

Supports --renderer (canvaskit, skwasm, skwasm_st), --benchmark, --filter, --smoke-test (the three selectors are mutually exclusive), --[no-]clean, --no-build, --local-web-sdk, --local-engine-src-path, --results-file, --baseline, --regression-threshold, and --verbose.
Extends runWebBenchmark in dev/devicelab/lib/tasks/web_benchmarks.dart with optional noBuild, localWebSdk, localEngineSrcPath, and verbose parameters while preserving the WebBenchmarkOptions record typedef.
Validates CLI arguments, prebuilt artifacts (--no-build), and baseline JSON files upfront with distinct exit codes (0 pass, 1 regression threshold exceeded, 2 usage/validation error, 3 benchmark/infrastructure failure).
Whenever --baseline is provided, compares the .average metrics present in both the current run and the baseline using compareResults, and prints the comparison table, colored only when stdout supports ANSI escapes. It exits 2 if the two runs have no metrics in common or a compared value is invalid.
Uses kSmokeTestBenchmarks for --smoke-test, and adds hermetic CLI unit tests in dev/devicelab/test/tasks/web_benchmarks_test.dart.

Local verification

I ran the CLI on Linux (headless Chrome 155, CanvasKit, profile build) from dev/devicelab. The numbers below come from an uncalibrated workstation and show only that the CLI works. They are not meaningful performance data.

1. Build, run the smoke-test benchmarks and save a baseline (exit code 0, about 65s including the web build):

$ dart bin/run_local_web_benchmark.dart --smoke-test --results-file /tmp/wb/run1.json
...
✓ Built build/web
Launching Chrome.
Launching benchmark "draw_rect"
Extracted 100 measured frames.
Launching benchmark "bench_card_infinite_scroll"
Extracted 100 measured frames.
bench_card_infinite_scroll.canvaskit.apply_frame.average: 67.17708333333333
bench_card_infinite_scroll.canvaskit.drawFrameDuration.average: 1338.2291666666667
bench_card_infinite_scroll.canvaskit.preroll_frame.average: 185.81720430107526
bench_card_infinite_scroll.canvaskit.totalUiFrame.average: 1772
draw_rect.canvaskit.drawFrameDuration.average: 4915.694736842105
draw_rect.canvaskit.sceneBuildDuration.average: 2.2142857142857144
draw_rect.canvaskit.totalUiFrame.average: 5218
draw_rect.canvaskit.windowRenderDuration.average: 1564.59375

run1.json holds all 26 metrics that TaskResult.data reports, including the .noise and .outlierAverage metrics.

2. Reuse the build and compare against the baseline (exit code 0, about 15s):

$ dart bin/run_local_web_benchmark.dart --smoke-test --no-build --baseline /tmp/wb/run1.json --results-file /tmp/wb/run5.json
...
Metric                                                          Baseline  Current   Delta  Status
--------------------------------------------------------------  --------  -------  ------  ------
bench_card_infinite_scroll.canvaskit.apply_frame.average            67.2     67.4    0.3%  PASS
bench_card_infinite_scroll.canvaskit.drawFrameDuration.average    1338.2   1265.6   -5.4%  PASS
bench_card_infinite_scroll.canvaskit.preroll_frame.average         185.8    181.1   -2.5%  PASS
bench_card_infinite_scroll.canvaskit.totalUiFrame.average         1772.0   2310.0   30.4%  PASS
draw_rect.canvaskit.drawFrameDuration.average                     4915.7   4984.0    1.4%  PASS
draw_rect.canvaskit.sceneBuildDuration.average                       2.2      1.9  -13.3%  PASS
draw_rect.canvaskit.totalUiFrame.average                          5218.0   5174.0   -0.8%  PASS
draw_rect.canvaskit.windowRenderDuration.average                  1564.6   1601.1    2.3%  PASS

Without --regression-threshold, the table is for information only and the command exits 0.

3. Regression gate with a deliberately tight threshold (exit code 1). This run happened while the machine was busy, so the deltas are large:

$ dart bin/run_local_web_benchmark.dart --smoke-test --no-build --baseline /tmp/wb/run1.json --regression-threshold 1
...
Metric                                                          Baseline  Current   Delta  Status
--------------------------------------------------------------  --------  -------  ------  ------
bench_card_infinite_scroll.canvaskit.apply_frame.average            67.2     65.6   -2.4%  PASS
bench_card_infinite_scroll.canvaskit.drawFrameDuration.average    1338.2   1191.3  -11.0%  PASS
bench_card_infinite_scroll.canvaskit.preroll_frame.average         185.8    176.8   -4.9%  PASS
bench_card_infinite_scroll.canvaskit.totalUiFrame.average         1772.0   2244.0   26.6%  FAIL
draw_rect.canvaskit.drawFrameDuration.average                     4915.7   6985.4   42.1%  FAIL
draw_rect.canvaskit.sceneBuildDuration.average                       2.2      7.2  226.7%  FAIL
draw_rect.canvaskit.totalUiFrame.average                          5218.0   6414.0   22.9%  FAIL
draw_rect.canvaskit.windowRenderDuration.average                  1564.6   2595.2   65.9%  FAIL

FAILED: 1 or more metrics exceeded the regression threshold of 1.0%.

4. Usage error (exit code 2):

$ dart bin/run_local_web_benchmark.dart --regression-threshold 5
--regression-threshold requires --baseline.

Part of #191316

Pre-launch Checklist

gemini-code-assist

Code Review

This pull request introduces a standalone local web benchmark runner CLI with comprehensive argument parsing, validation, and execution logic, alongside corresponding unit tests. The review feedback suggests enhancing verbose logging and error handling by printing errors and stack traces when they occur, and improving test validity by using existing files instead of non-existent paths to ensure validation logic is correctly tested.

- Drop the unclear -b, -f, and -r flag abbreviations. - Remove the test-only injection parameters from runBenchmarkCli and the tests that depended on them. - Reuse the target benchmark parsing and BenchmarkFilter validation instead of duplicating them. - Allow --filter to be combined with --benchmark or --smoke-test. - Clarify the error when --no-build cannot find build artifacts.

The benchmark filter now rejects combining target benchmarks with a filter pattern, so --filter can no longer be used together with --benchmark or --smoke-test. Update the CLI help text and cover the rejected combinations in tests.

…NSI escapes

…files in CLI validation tests

…markCli tests

…b benchmark CLI

…tion and --no-build check

Adds a standalone local CLI runner (`dev/devicelab/bin/run_local_web_benchmark.dart`) for running Flutter Web macrobenchmarks locally and in automated workflows without full Devicelab task boilerplate: - Supports `--renderer` (`canvaskit`, `skwasm`, `skwasm_st`), `--benchmark`, `--filter`, `--smoke-test` (the three selectors are mutually exclusive), `--[no-]clean`, `--no-build`, `--local-web-sdk`, `--local-engine-src-path`, `--results-file`, `--baseline`, `--regression-threshold`, and `--verbose`. - Extends `runWebBenchmark` in `dev/devicelab/lib/tasks/web_benchmarks.dart` with optional `noBuild`, `localWebSdk`, `localEngineSrcPath`, and `verbose` parameters while preserving the `WebBenchmarkOptions` record typedef. - Validates CLI arguments, prebuilt artifacts (`--no-build`), and baseline JSON files upfront with distinct exit codes (`0` pass, `1` regression threshold exceeded, `2` usage/validation error, `3` benchmark/infrastructure failure). - Whenever `--baseline` is provided, compares the `.average` metrics present in both the current run and the baseline using `compareResults`, and prints the comparison table, colored only when stdout supports ANSI escapes. It exits `2` if the two runs have no metrics in common or a compared value is invalid. - Uses `kSmokeTestBenchmarks` for `--smoke-test`, and adds hermetic CLI unit tests in `dev/devicelab/test/tasks/web_benchmarks_test.dart`. ## Local verification I ran the CLI on Linux (headless Chrome 155, CanvasKit, profile build) from `dev/devicelab`. The numbers below come from an uncalibrated workstation and show only that the CLI works. They are not meaningful performance data. **1. Build, run the smoke-test benchmarks and save a baseline** (exit code `0`, about 65s including the web build): ``` $ dart bin/run_local_web_benchmark.dart --smoke-test --results-file /tmp/wb/run1.json ... ✓ Built build/web Launching Chrome. Launching benchmark "draw_rect" Extracted 100 measured frames. Launching benchmark "bench_card_infinite_scroll" Extracted 100 measured frames. bench_card_infinite_scroll.canvaskit.apply_frame.average: 67.17708333333333 bench_card_infinite_scroll.canvaskit.drawFrameDuration.average: 1338.2291666666667 bench_card_infinite_scroll.canvaskit.preroll_frame.average: 185.81720430107526 bench_card_infinite_scroll.canvaskit.totalUiFrame.average: 1772 draw_rect.canvaskit.drawFrameDuration.average: 4915.694736842105 draw_rect.canvaskit.sceneBuildDuration.average: 2.2142857142857144 draw_rect.canvaskit.totalUiFrame.average: 5218 draw_rect.canvaskit.windowRenderDuration.average: 1564.59375 ``` `run1.json` holds all 26 metrics that `TaskResult.data` reports, including the `.noise` and `.outlierAverage` metrics. **2. Reuse the build and compare against the baseline** (exit code `0`, about 15s): ``` $ dart bin/run_local_web_benchmark.dart --smoke-test --no-build --baseline /tmp/wb/run1.json --results-file /tmp/wb/run5.json ... Metric Baseline Current Delta Status -------------------------------------------------------------- -------- ------- ------ ------ bench_card_infinite_scroll.canvaskit.apply_frame.average 67.2 67.4 0.3% PASS bench_card_infinite_scroll.canvaskit.drawFrameDuration.average 1338.2 1265.6 -5.4% PASS bench_card_infinite_scroll.canvaskit.preroll_frame.average 185.8 181.1 -2.5% PASS bench_card_infinite_scroll.canvaskit.totalUiFrame.average 1772.0 2310.0 30.4% PASS draw_rect.canvaskit.drawFrameDuration.average 4915.7 4984.0 1.4% PASS draw_rect.canvaskit.sceneBuildDuration.average 2.2 1.9 -13.3% PASS draw_rect.canvaskit.totalUiFrame.average 5218.0 5174.0 -0.8% PASS draw_rect.canvaskit.windowRenderDuration.average 1564.6 1601.1 2.3% PASS ``` Without `--regression-threshold`, the table is for information only and the command exits `0`. **3. Regression gate with a deliberately tight threshold** (exit code `1`). This run happened while the machine was busy, so the deltas are large: ``` $ dart bin/run_local_web_benchmark.dart --smoke-test --no-build --baseline /tmp/wb/run1.json --regression-threshold 1 ... Metric Baseline Current Delta Status -------------------------------------------------------------- -------- ------- ------ ------ bench_card_infinite_scroll.canvaskit.apply_frame.average 67.2 65.6 -2.4% PASS bench_card_infinite_scroll.canvaskit.drawFrameDuration.average 1338.2 1191.3 -11.0% PASS bench_card_infinite_scroll.canvaskit.preroll_frame.average 185.8 176.8 -4.9% PASS bench_card_infinite_scroll.canvaskit.totalUiFrame.average 1772.0 2244.0 26.6% FAIL draw_rect.canvaskit.drawFrameDuration.average 4915.7 6985.4 42.1% FAIL draw_rect.canvaskit.sceneBuildDuration.average 2.2 7.2 226.7% FAIL draw_rect.canvaskit.totalUiFrame.average 5218.0 6414.0 22.9% FAIL draw_rect.canvaskit.windowRenderDuration.average 1564.6 2595.2 65.9% FAIL FAILED: 1 or more metrics exceeded the regression threshold of 1.0%. ``` **4. Usage error** (exit code `2`): ``` $ dart bin/run_local_web_benchmark.dart --regression-threshold 5 --regression-threshold requires --baseline. ``` Part of flutter#191316 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing.  [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md

github-actions Bot added the team-web Owned by Web platform team label Oct 19, 2026

harryterkelsen force-pushed the web-benchmarks-part-4-standalone-local-cli-runner branch from 5973712 to 4cc2878 Compare October 19, 2026 19:43

harryterkelsen requested a review from mdebbar October 19, 2026 20:51

harryterkelsen marked this pull request as ready for review October 19, 2026 20:51

harryterkelsen added the CICD Run CI/CD label Oct 19, 2026

gemini-code-assist Bot reviewed Oct 19, 2026

View reviewed changes

Comment thread dev/devicelab/lib/tasks/web_benchmarks.dart Resolved

Comment thread dev/devicelab/lib/tasks/web_benchmarks.dart Resolved

Comment thread dev/devicelab/test/tasks/web_benchmarks_test.dart Resolved

Comment thread dev/devicelab/test/tasks/web_benchmarks_test.dart Resolved

Piinks added the platform-web Web applications specifically label Oct 19, 2026

harryterkelsen force-pushed the web-benchmarks-part-4-standalone-local-cli-runner branch from 997d2cb to 79eb5c5 Compare October 19, 2026 22:06

github-actions Bot removed the platform-web Web applications specifically label Oct 19, 2026

harryterkelsen added the platform-web Web applications specifically label Oct 19, 2026

harryterkelsen force-pushed the web-benchmarks-part-4-standalone-local-cli-runner branch from 79eb5c5 to 060e303 Compare October 19, 2026 23:11

github-actions Bot removed the platform-web Web applications specifically label Oct 19, 2026

harryterkelsen added 12 commits October 19, 2026 16:21

test: add failing reproduction tests for standalone local cli runner

9e29aae

fix: implement standalone local web benchmark cli

444b245

fix: handle badly typed benchmark JSON properly

a68c185

fix: wire standalone local web benchmark CLI runner to runWebBenchmark

e7660b5

refactor: update import to web_benchmarks_comparison.dart

188515f

refactor: use plain catch for benchmark run failure

78514f0

refactor: adapt local web benchmark CLI to trimmed comparison helper

f1d75e5

fix: color the benchmark comparison table only when stdout supports A…

d358b4b

…NSI escapes

fix: log errors and stack traces in runBenchmarkCli and use existing …

53716bf

…files in CLI validation tests

test: simplify path interpolation and clean up temp files in runBench…

181d818

…markCli tests

harryterkelsen force-pushed the web-benchmarks-part-4-standalone-local-cli-runner branch from 060e303 to 181d818 Compare October 19, 2026 23:25

mdebbar requested changes Oct 20, 2026

View reviewed changes

harryterkelsen added 3 commits October 20, 2026 10:01

test: reject --local-web-sdk combined with --no-build in the local we…

3262039

…b benchmark CLI

fix: harden local web benchmark CLI flag validation, repo root resolu…

76a32a4

…tion and --no-build check

style/docs: auto-resolve pre-flight review swarm findings

a5379ac

harryterkelsen requested a review from mdebbar October 20, 2026 17:25

mdebbar approved these changes Oct 21, 2026

View reviewed changes

harryterkelsen added the autosubmit Merge PR when tree becomes green via auto submit App label Oct 21, 2026

auto-submit Bot added this pull request to the merge queue Oct 21, 2026

Merged via the queue into flutter:master with commit 7e4332b Oct 21, 2026
23 checks passed

engine-flutter-autoroll mentioned this pull request Oct 21, 2026

Roll Flutter from a3d71d82160a to 7e4332ba502b (75 revisions) flutter/packages#13364

Closed

flutter-dashboard Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Oct 21, 2026

This was referenced Oct 28, 2026

[Web Benchmarks] Headless single-command runner for local web benchmarks #191316

Closed

[Umbrella] Improve developer and agent feedback loops for Flutter Web #191313

Open

[web] Add support for --local-web-sdk in benchmarks #123781

Closed

Navigation Menu

Repository navigation

Web Benchmarks Part 4: Standalone Local CLI Runner - #194782

Web Benchmarks Part 4: Standalone Local CLI Runner#194782
auto-submit[bot] merged 15 commits into
flutter:masterfrom
harryterkelsen:web-benchmarks-part-4-standalone-local-cli-runner

harryterkelsen commented Oct 19, 2026

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

harryterkelsen commented Oct 19, 2026

Local verification

Pre-launch Checklist

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants