Repository navigation
[Web Benchmarks] Headless single-command runner for local web benchmarks #191316
Description
Part of #191313
Problem
Running web benchmarks locally is cumbersome. Engine micro-benchmarks require manual profiling builds, while macrobenchmarks (`dev/benchmarks/macrobenchmarks`) require either running the heavy devicelab harness or manually clicking in a browser.
Solution
- Provide a single-command CLI to execute web benchmarks headlessly in Chrome.
- Output structured JSON metrics (frame render times, draw times, memory, profiler counters) and a terminal summary table.
- Support comparing results against a baseline JSON or `main` branch to catch performance regressions locally.
Critical Edge Cases & Implementation Gotchas
- Local Web SDK / Uncommitted Engine Artifacts ([web] Add support for --local-web-sdk in benchmarks #123781): Crucial requirement for local engine development: the benchmark CLI must support running against local uncommitted engine builds (`--local-web-sdk`, `wasm_release`).
- Structured Metrics Alignment ([integration_profiling] Support programmatic performance integration tests on Flutter Web #180821): Standardize JSON metrics output format (frame render time, draw time, memory, timeline counters) to align with programmatic performance integration testing expectations.
Related Issues & Historical Context
- [web] Add support for --local-web-sdk in benchmarks #123781: [web] Add support for --local-web-sdk in benchmarks (Satisfied by local engine build support in the benchmark runner; close on landing)
- [integration_profiling] Support programmatic performance integration tests on Flutter Web #180821: [integration_profiling] Support programmatic performance integration tests on Flutter Web
Activity
[devicelab] Clean up Chrome profile directories and trap abort signal…
Revert: [devicelab] Clean up Chrome profile directories and trap abor…
RELAND: [devicelab] Clean up Chrome profile directories and trap abor…
[devicelab] Unify web benchmarks entrypoint using BENCHMARK_SERVER_PO…
Web Benchmarks Part 2: Benchmark Targeting & Fail-Fast Filtering (flu…
Add web benchmark baseline comparison, ANSI summary table, and struct…
Web Benchmarks Part 4: Standalone Local CLI Runner (flutter#194782)
Web Benchmarks Part 4: Standalone Local CLI Runner (flutter#194782)
[web] Add opt-in raw samples, measured sample count and warm-up phase…
harryterkelsen commentedon Oct 28, 2026
Completed and landed across the following PRs on
master:--benchmark,--filter,--[no-]clean, and fast-failure for unrecognized benchmark names.compareResults, zero-baseline handling,formatAnsiSummaryTable, and smoke-test definitions.dev/devicelab/bin/run_local_web_benchmark.dartsupporting--renderer,--benchmark,--smoke-test,--filter,--baseline,--results-file,--regression-threshold,--no-build, and runtime--local-web-sdk/--local-engine-src-pathoverrides, with a structured 4-state exit code taxonomy. (This also resolves [web] Add support for --local-web-sdk in benchmarks #123781.)Developers and agents can now run benchmarks locally and headlessly with:
dart dev/devicelab/bin/run_local_web_benchmark.dart --smoke-test --baseline /path/to/baseline.json --regression-threshold 5.0Closing as completed.
Web Benchmarks Part 2: Benchmark Targeting & Fail-Fast Filtering (flu…
Add web benchmark baseline comparison, ANSI summary table, and struct…
Web Benchmarks Part 4: Standalone Local CLI Runner (flutter#194782)
[web] Add opt-in raw samples, measured sample count and warm-up phase…
github-actions commentedon Nov 11, 2026
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.