Web Benchmarks Part 2: Benchmark Targeting & Fail-Fast Filtering by harryterkelsen · Pull Request #194400 · flutter/flutter

harryterkelsen · 2026-10-14T17:50:32Z

Adds benchmark targeting, fail-fast filtering, and incremental build support (--no-clean) to runWebBenchmark in dev/devicelab/lib/tasks/web_benchmarks.dart:

Retains the WebBenchmarkOptions record typedef and adds optional targetBenchmarks, filterPattern, and clean named parameters to runWebBenchmark with BENCHMARK_TARGETS, BENCHMARK_FILTER, and BENCHMARK_CLEAN environment variable fallbacks.
Adds runWebBenchmarkFromArgs so the dev/devicelab/bin/tasks/web_benchmarks_*.dart task entrypoints accept --benchmark (-b), --filter (-f), and --[no-]clean CLI flags.
Extracts BenchmarkFilter and processBenchmarkProfiles so unrecognized benchmark names (typos), zero-match regexes, or empty profile collections fail fast instead of returning a false-positive TaskResult.success({}).
Adds hermetic unit tests in dev/devicelab/test/tasks/web_benchmarks_test.dart.

Part of #191316

Pre-launch Checklist

…ests

gemini-code-assist

Code Review

This pull request introduces command-line argument parsing and filtering capabilities to the web benchmarks task in the Flutter devicelab, allowing benchmarks to be filtered by target names or regex patterns. Feedback on the changes suggests moving JSON decoding inside a try-catch block to prevent server crashes, using safe type checks instead of direct casts for profile names and score keys to avoid runtime type errors, and refactoring a test that incorrectly uses duplicate keys in a map literal.

Specifying explicit target benchmarks (--benchmark / BENCHMARK_TARGETS) together with a filter pattern (--filter / BENCHMARK_FILTER) is now an error instead of intersecting the two. This applies whether the values come from arguments, environment variables, or one of each, and fails before anything is built.

mdebbar · 2026-10-15T12:18:39Z

+    ..addMultiOption(
+      'benchmark',
+      help:
+          'Target benchmark names to run (comma-separated or multiple flags). '


How do you expect this to be invoked? Going through test_runner.dart test --task-args benchmark=draw_rect,bench_card, the runner splits --task-args on commas, so this task receives --benchmark=draw_rect --bench_card and fails on the unknown --bench_card option.

In a forthcoming PR I am adding a CLI entrypoint which let's you directly pass in --benchmark. So you could do dart dev/devicelab/bin/run_local_web_benchmark.dart --benchmark=draw_rect,bench_card.

But if running through task_runner.dart, you could still do --task-args benchmark=draw_rect,benchmark=bench_card.

If we think the "(comma-separated or multiple flags)" wording is misleading to users of task_runner.dart I can tweak the help text or we can make it so that we don't support comma-separated in this entrypoint and only allow it in my forthcoming local benchmark runner entrypoint

…ndling

harryterkelsen · 2026-10-15T17:21:13Z

Thanks, @mdebbar ! PTAL

…tter#194400) Adds benchmark targeting, fail-fast filtering, and incremental build support (`--no-clean`) to `runWebBenchmark` in `dev/devicelab/lib/tasks/web_benchmarks.dart`: - Retains the `WebBenchmarkOptions` record typedef and adds optional `targetBenchmarks`, `filterPattern`, and `clean` named parameters to `runWebBenchmark` with `BENCHMARK_TARGETS`, `BENCHMARK_FILTER`, and `BENCHMARK_CLEAN` environment variable fallbacks. - Adds `runWebBenchmarkFromArgs` so the `dev/devicelab/bin/tasks/web_benchmarks_*.dart` task entrypoints accept `--benchmark` (`-b`), `--filter` (`-f`), and `--[no-]clean` CLI flags. - Extracts `BenchmarkFilter` and `processBenchmarkProfiles` so unrecognized benchmark names (typos), zero-match regexes, or empty profile collections fail fast instead of returning a false-positive `TaskResult.success({})`. - Adds hermetic unit tests in `dev/devicelab/test/tasks/web_benchmarks_test.dart`. Part of flutter#191316 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing.  [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md

harryterkelsen added 13 commits October 7, 2026 11:38

test: add failing reproduction test for web benchmark targeting

bc166de

fix: implement web benchmark targeting and fail-fast filtering

0fc0a89

test: add hostile stress tests for web benchmarks

cc05f81

fix: address PR feedback on web benchmark filtering

731877f

style: apply dart format

249101a

fix: address PR feedback for web benchmarks

d6e5db0

fix: update BenchmarkFilter constructor and test usages

2c5145c

fix: address PR feedback for web benchmarks (part 2)

79f0e42

fix: move lint ignore to the correct place

ba03d11

fix: correctly parse BENCHMARK_TARGETS environment variable

105d416

fix: simplify BENCHMARK_CLEAN parsing

6142f98

fix: rename rawArgs to args in web benchmark entrypoints and update t…

2fcb0ff

…ests

fix: address remaining PR review comments

979a664

github-actions Bot added the team-web Owned by Web platform team label Oct 14, 2026

harryterkelsen requested a review from mdebbar October 14, 2026 18:15

harryterkelsen marked this pull request as ready for review October 14, 2026 19:18

gemini-code-assist Bot reviewed Oct 14, 2026

View reviewed changes

Comment thread dev/devicelab/lib/tasks/web_benchmarks.dart Resolved

Comment thread dev/devicelab/lib/tasks/web_benchmarks.dart Resolved

Comment thread dev/devicelab/lib/tasks/web_benchmarks.dart Resolved

Comment thread dev/devicelab/test/tasks/web_benchmarks_test.dart Resolved

fix: add missing license header and address PR review comments

6e33c59

harryterkelsen added the CICD Run CI/CD label Oct 14, 2026

mdebbar reviewed Oct 15, 2026

View reviewed changes

Address PR review comments: avoid dynamic and clarify unknown-flag ha…

1a78fb6

…ndling

harryterkelsen requested a review from mdebbar October 15, 2026 17:21

mdebbar approved these changes Oct 16, 2026

View reviewed changes

harryterkelsen added the autosubmit Merge PR when tree becomes green via auto submit App label Oct 16, 2026

auto-submit Bot added this pull request to the merge queue Oct 16, 2026

Merged via the queue into flutter:master with commit 449fb19 Oct 16, 2026
22 of 23 checks passed

flutter-dashboard Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Oct 16, 2026

This was referenced Oct 28, 2026

[Web Benchmarks] Headless single-command runner for local web benchmarks #191316

Closed

[Umbrella] Improve developer and agent feedback loops for Flutter Web #191313

Open

Navigation Menu

Repository navigation

Web Benchmarks Part 2: Benchmark Targeting & Fail-Fast Filtering - #194400

Web Benchmarks Part 2: Benchmark Targeting & Fail-Fast Filtering#194400
auto-submit[bot] merged 16 commits into
flutter:masterfrom
harryterkelsen:web-benchmarks-part-2-benchmark-targeting-fail

harryterkelsen commented Oct 14, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mdebbar Oct 15, 2026

Uh oh!

harryterkelsen Oct 15, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

harryterkelsen commented Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

harryterkelsen commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-launch Checklist

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mdebbar Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

harryterkelsen Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

harryterkelsen commented Oct 15, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

harryterkelsen commented Oct 14, 2026 •

edited

Loading