Skip to content

Commit 7e4332b

Browse files
Web Benchmarks Part 4: Standalone Local CLI Runner (flutter#194782)
Adds a standalone local CLI runner (`dev/devicelab/bin/run_local_web_benchmark.dart`) for running Flutter Web macrobenchmarks locally and in automated workflows without full Devicelab task boilerplate: - Supports `--renderer` (`canvaskit`, `skwasm`, `skwasm_st`), `--benchmark`, `--filter`, `--smoke-test` (the three selectors are mutually exclusive), `--[no-]clean`, `--no-build`, `--local-web-sdk`, `--local-engine-src-path`, `--results-file`, `--baseline`, `--regression-threshold`, and `--verbose`. - Extends `runWebBenchmark` in `dev/devicelab/lib/tasks/web_benchmarks.dart` with optional `noBuild`, `localWebSdk`, `localEngineSrcPath`, and `verbose` parameters while preserving the `WebBenchmarkOptions` record typedef. - Validates CLI arguments, prebuilt artifacts (`--no-build`), and baseline JSON files upfront with distinct exit codes (`0` pass, `1` regression threshold exceeded, `2` usage/validation error, `3` benchmark/infrastructure failure). - Whenever `--baseline` is provided, compares the `.average` metrics present in both the current run and the baseline using `compareResults`, and prints the comparison table, colored only when stdout supports ANSI escapes. It exits `2` if the two runs have no metrics in common or a compared value is invalid. - Uses `kSmokeTestBenchmarks` for `--smoke-test`, and adds hermetic CLI unit tests in `dev/devicelab/test/tasks/web_benchmarks_test.dart`. ## Local verification I ran the CLI on Linux (headless Chrome 155, CanvasKit, profile build) from `dev/devicelab`. The numbers below come from an uncalibrated workstation and show only that the CLI works. They are not meaningful performance data. **1. Build, run the smoke-test benchmarks and save a baseline** (exit code `0`, about 65s including the web build): ``` $ dart bin/run_local_web_benchmark.dart --smoke-test --results-file /tmp/wb/run1.json ... ✓ Built build/web Launching Chrome. Launching benchmark "draw_rect" Extracted 100 measured frames. Launching benchmark "bench_card_infinite_scroll" Extracted 100 measured frames. bench_card_infinite_scroll.canvaskit.apply_frame.average: 67.17708333333333 bench_card_infinite_scroll.canvaskit.drawFrameDuration.average: 1338.2291666666667 bench_card_infinite_scroll.canvaskit.preroll_frame.average: 185.81720430107526 bench_card_infinite_scroll.canvaskit.totalUiFrame.average: 1772 draw_rect.canvaskit.drawFrameDuration.average: 4915.694736842105 draw_rect.canvaskit.sceneBuildDuration.average: 2.2142857142857144 draw_rect.canvaskit.totalUiFrame.average: 5218 draw_rect.canvaskit.windowRenderDuration.average: 1564.59375 ``` `run1.json` holds all 26 metrics that `TaskResult.data` reports, including the `.noise` and `.outlierAverage` metrics. **2. Reuse the build and compare against the baseline** (exit code `0`, about 15s): ``` $ dart bin/run_local_web_benchmark.dart --smoke-test --no-build --baseline /tmp/wb/run1.json --results-file /tmp/wb/run5.json ... Metric Baseline Current Delta Status -------------------------------------------------------------- -------- ------- ------ ------ bench_card_infinite_scroll.canvaskit.apply_frame.average 67.2 67.4 0.3% PASS bench_card_infinite_scroll.canvaskit.drawFrameDuration.average 1338.2 1265.6 -5.4% PASS bench_card_infinite_scroll.canvaskit.preroll_frame.average 185.8 181.1 -2.5% PASS bench_card_infinite_scroll.canvaskit.totalUiFrame.average 1772.0 2310.0 30.4% PASS draw_rect.canvaskit.drawFrameDuration.average 4915.7 4984.0 1.4% PASS draw_rect.canvaskit.sceneBuildDuration.average 2.2 1.9 -13.3% PASS draw_rect.canvaskit.totalUiFrame.average 5218.0 5174.0 -0.8% PASS draw_rect.canvaskit.windowRenderDuration.average 1564.6 1601.1 2.3% PASS ``` Without `--regression-threshold`, the table is for information only and the command exits `0`. **3. Regression gate with a deliberately tight threshold** (exit code `1`). This run happened while the machine was busy, so the deltas are large: ``` $ dart bin/run_local_web_benchmark.dart --smoke-test --no-build --baseline /tmp/wb/run1.json --regression-threshold 1 ... Metric Baseline Current Delta Status -------------------------------------------------------------- -------- ------- ------ ------ bench_card_infinite_scroll.canvaskit.apply_frame.average 67.2 65.6 -2.4% PASS bench_card_infinite_scroll.canvaskit.drawFrameDuration.average 1338.2 1191.3 -11.0% PASS bench_card_infinite_scroll.canvaskit.preroll_frame.average 185.8 176.8 -4.9% PASS bench_card_infinite_scroll.canvaskit.totalUiFrame.average 1772.0 2244.0 26.6% FAIL draw_rect.canvaskit.drawFrameDuration.average 4915.7 6985.4 42.1% FAIL draw_rect.canvaskit.sceneBuildDuration.average 2.2 7.2 226.7% FAIL draw_rect.canvaskit.totalUiFrame.average 5218.0 6414.0 22.9% FAIL draw_rect.canvaskit.windowRenderDuration.average 1564.6 2595.2 65.9% FAIL FAILED: 1 or more metrics exceeded the regression threshold of 1.0%. ``` **4. Usage error** (exit code `2`): ``` $ dart bin/run_local_web_benchmark.dart --regression-threshold 5 --regression-threshold requires --baseline. ``` Part of flutter#191316 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
1 parent a5128c7 commit 7e4332b

3 files changed

Lines changed: 538 additions & 4 deletions

File tree

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
// Copyright 2014 The Flutter Authors. All rights reserved.
2+
// Use of this source code is governed by a BSD-style license that can be
3+
// found in the LICENSE file.
4+
5+
import 'dart:io' as io;
6+
7+
import 'package:flutter_devicelab/tasks/web_benchmarks.dart';
8+
9+
Future<void> main(List<String> args) async {
10+
io.exit(await runBenchmarkCli(args));
11+
}

0 commit comments

Comments
 (0)