Repository navigation
Add web benchmark baseline comparison, ANSI summary table, and structured JSON helpers - #194609
Merged
auto-submit[bot] merged 13 commits intoOct 19, 2026
Conversation
…ison.dart and move kSmokeTestBenchmarks
…nstead of asserting Callers no longer need to pre-sort metrics; the public constructor copies and sorts them, and exposes an unmodifiable list.
…ner needs compareResults now compares only `.average` metrics present in both the baseline and the current run. This removes the failOnMissingBaseline flag, missingMetrics, current-only "new metric" rows and the nullable baseline/current values, along with their N/A rendering. Also removes: - formatStructuredJson, since results are saved as the flattened TaskResult.data map. - BenchmarkComparison.regressionThreshold, which had no readers. - The sorting factory constructor; compareResults sorts the keys instead. Simplifies the delta and regression computation and shortens the docs. Tests: - Delta and threshold cases are now one table-driven matrix. It adds coverage for no threshold, a delta equal to the threshold, an infinite threshold and null values. - The hand-rolled table alignment checks are replaced by one exact expected-table string and an ANSI test.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Code Review
This pull request introduces a new utility for comparing web benchmark results. It adds kSmokeTestBenchmarks to web_benchmarks.dart and creates web_benchmarks_comparison.dart, which defines classes and functions (MetricComparison, BenchmarkComparison, compareResults, and formatAnsiSummaryTable) to compare baseline and current benchmark metrics, detect regressions based on a threshold, and format the results into an aligned ANSI-colored terminal table. Comprehensive unit tests for these comparison and formatting utilities are added in web_benchmarks_test.dart. There are no review comments, and I have no feedback to provide.
mdebbar
previously approved these changes
Oct 19, 2026
harryterkelsen
added
the
autosubmit
flutter-dashboard
Bot
removed
the
autosubmit
harryterkelsen
added
the
autosubmit
flutter-dashboard
Bot
removed
the
autosubmit
mdebbar
approved these changes
Oct 19, 2026
harryterkelsen
added
the
autosubmit
flutter-dashboard
Bot
removed
the
autosubmit
This was referenced Oct 19, 2026
This was referenced Oct 28, 2026
suryaKommana2662
pushed a commit
to suryaKommana2662/flutter
that referenced
this pull request
Oct 29, 2026
…ured JSON helpers (flutter#194609) Adds pure, unit-tested helpers in `dev/devicelab/lib/tasks/web_benchmarks_comparison.dart` that compare a web benchmark run against a saved baseline and print the result as a terminal table. A follow-up PR uses them in a local benchmark CLI. - `compareResults` compares the `.average` metrics that appear in both the current and the baseline `TaskResult.data` maps, sorted by key; all other keys are ignored. It computes each metric's percentage change and flags a regression when the change exceeds an optional `regressionThreshold`. Speedups are never regressions, and a metric that rises from zero has an infinite delta. Invalid thresholds throw `ArgumentError`, and compared values that are not finite, non-negative numbers throw `FormatException`. - `formatAnsiSummaryTable` prints a `Metric` / `Baseline` / `Current` / `Delta` / `Status` table with a header and separator row, right-aligned numbers, and an optionally ANSI-colored `PASS`/`FAIL` status. - Adds `kSmokeTestBenchmarks` to `web_benchmarks.dart`, the curated fast benchmark set (`draw_rect`, `bench_card_infinite_scroll`) for the CLI's `--smoke-test` flag. - Adds unit tests in `dev/devicelab/test/tasks/web_benchmarks_test.dart`: a table-driven matrix of delta and threshold cases (including zero baselines, no threshold and a delta equal to the threshold), key selection, input validation, and an exact expected-table test. Part of flutter#191316 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#ai-contribution-guidelines [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds pure, unit-tested helpers in
dev/devicelab/lib/tasks/web_benchmarks_comparison.dartthat compare a web benchmark run against a saved baseline and print the result as a terminal table. A follow-up PR uses them in a local benchmark CLI.compareResultscompares the.averagemetrics that appear in both the current and the baselineTaskResult.datamaps, sorted by key; all other keys are ignored. It computes each metric's percentage change and flags a regression when the change exceeds an optionalregressionThreshold. Speedups are never regressions, and a metric that rises from zero has an infinite delta. Invalid thresholds throwArgumentError, and compared values that are not finite, non-negative numbers throwFormatException.formatAnsiSummaryTableprints aMetric/Baseline/Current/Delta/Statustable with a header and separator row, right-aligned numbers, and an optionally ANSI-coloredPASS/FAILstatus.kSmokeTestBenchmarkstoweb_benchmarks.dart, the curated fast benchmark set (draw_rect,bench_card_infinite_scroll) for the CLI's--smoke-testflag.dev/devicelab/test/tasks/web_benchmarks_test.dart: a table-driven matrix of delta and threshold cases (including zero baselines, no threshold and a delta equal to the threshold), key selection, input validation, and an exact expected-table test.Part of #191316
Pre-launch Checklist
///).