Repository navigation
[web] Add opt-in raw samples, measured sample count and warm-up phase hook to the web benchmark recorder - #195118
Merged
auto-submit[bot] merged 10 commits intoOct 27, 2026
Conversation
…to web benchmark recorder
…d-trip test When sample reporting is enabled, each series now also reports its warm-up sample magnitudes under <series>.warmUpSamples. The legacy key set is unchanged when reporting is off. A new test checks that samples, statistics and scoreKeys survive jsonEncode/jsonDecode under both the JS and Wasm encoders.
Contributor
There was a problem hiding this comment.
Code Review
This pull request adds support for configuring measured sample counts, reporting raw measured and warm-up samples in the benchmark JSON output, and notifying the benchmark server when a benchmark switches from warm-up to measurement. It also introduces a comprehensive test suite for these features. Feedback is provided to wrap the asynchronous _requestXhr call in reportPhase with a try-catch block to prevent unhandled exceptions from propagating when the server returns a non-200 status code.
mdebbar
previously approved these changes
Oct 23, 2026
mdebbar
approved these changes
Oct 27, 2026
This was referenced Oct 27, 2026
auto-submit Bot
pushed a commit
to flutter/packages
that referenced
this pull request
Oct 28, 2026
flutter/flutter@d4c8577...0c2473c 2026-10-28 [email protected] Roll Packages from a8549ae to 13d38e0 (9 revisions) (flutter/flutter#195488) 2026-10-28 [email protected] Use int.oneBitCount in PersistentHashMap and scope 53-bit caps to JS (flutter/flutter#195005) 2026-10-28 [email protected] Roll pub packages (flutter/flutter#195483) 2026-10-28 [email protected] [iPhone Duo] Use the correct UIScreen to report the display size and update when fold/unfold (flutter/flutter#195199) 2026-10-28 [email protected] [fuchsia] Fix portable_ui_test.h includes for newer Fuchsia SDKs (flutter/flutter#195420) 2026-10-28 [email protected] Change default pipeline options to match production draw calls (flutter/flutter#194597) 2026-10-28 [email protected] Roll Fuchsia Test Scripts from 2T18KC7-LDQtmQiLP... to xl1LWBuppzu-BnOzV... (flutter/flutter#195450) 2026-10-27 [email protected] [flutter_tools] Inject dependencies into Darwin build targets, native assets, and Xcode build settings (flutter/flutter#194052) 2026-10-27 [email protected] Revert: Reland square corners fix (flutter/flutter#195436) 2026-10-27 [email protected] [flutter_tools] Add DeviceService app launch and flutter run support for tool extensions (flutter/flutter#194589) 2026-10-27 [email protected] [engine] Honor targetWidth/targetHeight for animated images (flutter/flutter#194615) 2026-10-27 [email protected] [Android][a11y] Set visibleToUser=false for hidden semantics nodes (flutter/flutter#194411) 2026-10-27 [email protected] Fix repo deletion test flake (flutter/flutter#195349) 2026-10-27 [email protected] Roll Packages from 23a05b6 to a8549ae (10 revisions) (flutter/flutter#195434) 2026-10-27 [email protected] [web] Capture pointers to the Flutter view so drags survive platform view iframes (flutter/flutter#195187) 2026-10-27 [email protected] [web] Add opt-in raw samples, measured sample count and warm-up phase hook to the web benchmark recorder (flutter/flutter#195118) 2026-10-27 [email protected] Link migration guide in deprecation messages (flutter/flutter#195361) 2026-10-27 [email protected] Pin basic_material_app_macos__compile to macOS 26 (flutter/flutter#195429) 2026-10-27 [email protected] Add benchmark for warmed pipeline usage (flutter/flutter#195001) 2026-10-27 [email protected] [flutter_tools] Add device log streaming support to DeviceService and ExtensionBackedDevice (flutter/flutter#194964) 2026-10-27 [email protected] Roll Dart SDK from 1d16c04dc346 to 65fc979759ca (4 revisions) (flutter/flutter#195383) If this roll has caused a breakage, revert this CL and set the roller to dry run mode using the controls here: https://autoroll.skia.org/r/flutter-packages Please CC [email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
This was referenced Oct 28, 2026
suryaKommana2662
pushed a commit
to suryaKommana2662/flutter
that referenced
this pull request
Oct 29, 2026
… hook to the web benchmark recorder (flutter#195118) <!-- Thanks for filing a pull request! Reviewers are typically assigned within a week of filing a request. To learn more about code review, see our documentation on Tree Hygiene: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md --> Adds three opt-in, compile-time `--dart-define`s to the web macrobenchmarks recorder (`dev/benchmarks/macrobenchmarks`). They let a local benchmark host compute frame-time percentiles and start a CPU profiler exactly when warm-up ends, without forking the app's reporting protocol: - `BENCHMARK_REPORT_SAMPLES=true`: `Profile.toJson` also reports two flat lists of raw sample magnitudes (in microseconds, in recording order) for each series, so `/profile-data` carries per-frame data: `<series>.samples` (the measured samples) and `<series>.warmUpSamples` (the warm-up samples). `scoreKeys` and the existing statistics are unchanged. - `BENCHMARK_MEASURED_SAMPLES=<n>`: overrides the number of measured samples, which defaults to 100. `kDefaultTotalSampleCount` follows. - `BENCHMARK_REPORT_PHASES=true`: installs a new static `Profile.phaseListener`, which `Profile.stopWarmingUp()` calls with `'measure'` at the warm-up → measure switch. `web_benchmarks.dart` uses it to send a fire-and-forget `POST benchmark-phase` to the benchmark server. The request is not awaited and is deferred with `Timer.run`, so the frame loop and the first measured frame are not affected. With none of the defines set, the output is the same as today, so CI and the dashboards are unaffected. To check this, I ran `dart dev/devicelab/bin/run.dart -t web_benchmarks_canvaskit` with `BENCHMARK_TARGETS=draw_rect` on this branch and on its `master` base. Both runs produced the same 13 `ResultData` keys and the same `BenchmarkScoreKeys`. The devicelab host is not changed in this PR. The new tests are in `dev/benchmarks/macrobenchmarks/test/web/recorder_test.dart`. They check: - the exact legacy key set when sample reporting is off; - that `<series>.samples` has only the measured samples and `<series>.warmUpSamples` only the warm-up samples, in order, with outliers kept, for reported and unreported series, and with each series split at its own warm-up boundary; - that `extraData` still merges last; - that `phaseListener` fires exactly once with `'measure'`, after the warm-up flag flips, on both automatic and manual warm-up; - that nothing is called when no listener is set; - that samples, warm-up samples, the scalar statistics and `scoreKeys` survive a `jsonEncode`/`jsonDecode` round trip (compared numerically, since the JS and Wasm encoders print doubles differently), and that the decoded key set is still the legacy one when reporting is off. `dev/bots` does not run the macrobenchmarks package tests, so I ran them manually from `dev/benchmarks/macrobenchmarks` with `flutter test --platform chrome test/web/recorder_test.dart` and `flutter test --platform chrome --wasm test/web/recorder_test.dart` (11 tests pass on both). Part of flutter#191316 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds three opt-in, compile-time
--dart-defines to the web macrobenchmarks recorder (dev/benchmarks/macrobenchmarks). They let a local benchmark host compute frame-time percentiles and start a CPU profiler exactly when warm-up ends, without forking the app's reporting protocol:BENCHMARK_REPORT_SAMPLES=true:Profile.toJsonalso reports two flat lists of raw sample magnitudes (in microseconds, in recording order) for each series, so/profile-datacarries per-frame data:<series>.samples(the measured samples) and<series>.warmUpSamples(the warm-up samples).scoreKeysand the existing statistics are unchanged.BENCHMARK_MEASURED_SAMPLES=<n>: overrides the number of measured samples, which defaults to 100.kDefaultTotalSampleCountfollows.BENCHMARK_REPORT_PHASES=true: installs a new staticProfile.phaseListener, whichProfile.stopWarmingUp()calls with'measure'at the warm-up → measure switch.web_benchmarks.dartuses it to send a fire-and-forgetPOST benchmark-phaseto the benchmark server. The request is not awaited and is deferred withTimer.run, so the frame loop and the first measured frame are not affected.With none of the defines set, the output is the same as today, so CI and the dashboards are unaffected. To check this, I ran
dart dev/devicelab/bin/run.dart -t web_benchmarks_canvaskitwithBENCHMARK_TARGETS=draw_recton this branch and on itsmasterbase. Both runs produced the same 13ResultDatakeys and the sameBenchmarkScoreKeys. The devicelab host is not changed in this PR.The new tests are in
dev/benchmarks/macrobenchmarks/test/web/recorder_test.dart. They check:<series>.sampleshas only the measured samples and<series>.warmUpSamplesonly the warm-up samples, in order, with outliers kept, for reported and unreported series, and with each series split at its own warm-up boundary;extraDatastill merges last;phaseListenerfires exactly once with'measure', after the warm-up flag flips, on both automatic and manual warm-up;scoreKeyssurvive ajsonEncode/jsonDecoderound trip (compared numerically, since the JS and Wasm encoders print doubles differently), and that the decoded key set is still the legacy one when reporting is off.dev/botsdoes not run the macrobenchmarks package tests, so I ran them manually fromdev/benchmarks/macrobenchmarkswithflutter test --platform chrome test/web/recorder_test.dartandflutter test --platform chrome --wasm test/web/recorder_test.dart(11 tests pass on both).Part of #191316
Pre-launch Checklist
///).If you need help, consider asking for advice on the #hackers-new channel on Discord.
If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance.
Note: The Flutter team is currently trialing the use of Gemini Code Assist for GitHub. Comments from the
gemini-code-assistbot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.