Skip to content

[web] Add opt-in raw samples, measured sample count and warm-up phase hook to the web benchmark recorder - #195118

Merged
auto-submit[bot] merged 10 commits into
flutter:masterfrom
harryterkelsen:web-benchmarks-phase-2-a-b-engine-variants
Oct 27, 2026
Merged

auto-submit[bot] merged 10 commits into
flutter:masterfrom
harryterkelsen:web-benchmarks-phase-2-a-b-engine-variants

Conversation

@harryterkelsen

Copy link
Copy Markdown
Contributor

Adds three opt-in, compile-time --dart-defines to the web macrobenchmarks recorder (dev/benchmarks/macrobenchmarks). They let a local benchmark host compute frame-time percentiles and start a CPU profiler exactly when warm-up ends, without forking the app's reporting protocol:

  • BENCHMARK_REPORT_SAMPLES=true: Profile.toJson also reports two flat lists of raw sample magnitudes (in microseconds, in recording order) for each series, so /profile-data carries per-frame data: <series>.samples (the measured samples) and <series>.warmUpSamples (the warm-up samples). scoreKeys and the existing statistics are unchanged.
  • BENCHMARK_MEASURED_SAMPLES=<n>: overrides the number of measured samples, which defaults to 100. kDefaultTotalSampleCount follows.
  • BENCHMARK_REPORT_PHASES=true: installs a new static Profile.phaseListener, which Profile.stopWarmingUp() calls with 'measure' at the warm-up → measure switch. web_benchmarks.dart uses it to send a fire-and-forget POST benchmark-phase to the benchmark server. The request is not awaited and is deferred with Timer.run, so the frame loop and the first measured frame are not affected.

With none of the defines set, the output is the same as today, so CI and the dashboards are unaffected. To check this, I ran dart dev/devicelab/bin/run.dart -t web_benchmarks_canvaskit with BENCHMARK_TARGETS=draw_rect on this branch and on its master base. Both runs produced the same 13 ResultData keys and the same BenchmarkScoreKeys. The devicelab host is not changed in this PR.

The new tests are in dev/benchmarks/macrobenchmarks/test/web/recorder_test.dart. They check:

  • the exact legacy key set when sample reporting is off;
  • that <series>.samples has only the measured samples and <series>.warmUpSamples only the warm-up samples, in order, with outliers kept, for reported and unreported series, and with each series split at its own warm-up boundary;
  • that extraData still merges last;
  • that phaseListener fires exactly once with 'measure', after the warm-up flag flips, on both automatic and manual warm-up;
  • that nothing is called when no listener is set;
  • that samples, warm-up samples, the scalar statistics and scoreKeys survive a jsonEncode/jsonDecode round trip (compared numerically, since the JS and Wasm encoders print doubles differently), and that the decoded key set is still the legacy one when reporting is off.

dev/bots does not run the macrobenchmarks package tests, so I ran them manually from dev/benchmarks/macrobenchmarks with flutter test --platform chrome test/web/recorder_test.dart and flutter test --platform chrome --wasm test/web/recorder_test.dart (11 tests pass on both).

Part of #191316

Pre-launch Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance.

Note: The Flutter team is currently trialing the use of Gemini Code Assist for GitHub. Comments from the gemini-code-assist bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.

…d-trip test

When sample reporting is enabled, each series now also reports its warm-up
sample magnitudes under <series>.warmUpSamples. The legacy key set is
unchanged when reporting is off. A new test checks that samples, statistics
and scoreKeys survive jsonEncode/jsonDecode under both the JS and Wasm
encoders.
@github-actions github-actions Bot added the team-web Owned by Web platform team label Oct 22, 2026
@harryterkelsen harryterkelsen added the CICD Run CI/CD label Oct 22, 2026
@harryterkelsen
harryterkelsen marked this pull request as ready for review October 23, 2026 00:00

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request adds support for configuring measured sample counts, reporting raw measured and warm-up samples in the benchmark JSON output, and notifying the benchmark server when a benchmark switches from warm-up to measurement. It also introduces a comprehensive test suite for these features. Feedback is provided to wrap the asynchronous _requestXhr call in reportPhase with a try-catch block to prevent unhandled exceptions from propagating when the server returns a non-200 status code.

mdebbar
mdebbar previously approved these changes Oct 23, 2026
@Piinks Piinks added the platform-web Web applications specifically label Oct 26, 2026
@harryterkelsen harryterkelsen added the autosubmit Merge PR when tree becomes green via auto submit App label Oct 27, 2026
@auto-submit
auto-submit Bot added this pull request to the merge queue Oct 27, 2026
Merged via the queue into flutter:master with commit b1e1fa6 Oct 27, 2026
27 of 28 checks passed
@flutter-dashboard flutter-dashboard Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Oct 27, 2026
auto-submit Bot pushed a commit to flutter/packages that referenced this pull request Oct 28, 2026
flutter/flutter@d4c8577...0c2473c

2026-10-28 [email protected] Roll Packages from a8549ae to 13d38e0 (9 revisions) (flutter/flutter#195488)
2026-10-28 [email protected] Use int.oneBitCount in PersistentHashMap and scope 53-bit caps to JS (flutter/flutter#195005)
2026-10-28 [email protected] Roll pub packages (flutter/flutter#195483)
2026-10-28 [email protected] [iPhone Duo] Use the correct UIScreen to report the display size and update when fold/unfold (flutter/flutter#195199)
2026-10-28 [email protected] [fuchsia] Fix portable_ui_test.h includes for newer Fuchsia SDKs (flutter/flutter#195420)
2026-10-28 [email protected] Change default pipeline options to match production draw calls (flutter/flutter#194597)
2026-10-28 [email protected] Roll Fuchsia Test Scripts from 2T18KC7-LDQtmQiLP... to xl1LWBuppzu-BnOzV... (flutter/flutter#195450)
2026-10-27 [email protected] [flutter_tools] Inject dependencies into Darwin build targets, native assets, and Xcode build settings (flutter/flutter#194052)
2026-10-27 [email protected] Revert: Reland square corners fix (flutter/flutter#195436)
2026-10-27 [email protected] [flutter_tools] Add DeviceService app launch and flutter run support for tool extensions (flutter/flutter#194589)
2026-10-27 [email protected] [engine] Honor targetWidth/targetHeight for animated images (flutter/flutter#194615)
2026-10-27 [email protected] [Android][a11y] Set visibleToUser=false for hidden semantics nodes (flutter/flutter#194411)
2026-10-27 [email protected] Fix repo deletion test flake (flutter/flutter#195349)
2026-10-27 [email protected] Roll Packages from 23a05b6 to a8549ae (10 revisions) (flutter/flutter#195434)
2026-10-27 [email protected] [web] Capture pointers to the Flutter view so drags survive platform view iframes (flutter/flutter#195187)
2026-10-27 [email protected] [web] Add opt-in raw samples, measured sample count and warm-up phase hook to the web benchmark recorder (flutter/flutter#195118)
2026-10-27 [email protected] Link migration guide in deprecation messages (flutter/flutter#195361)
2026-10-27 [email protected] Pin basic_material_app_macos__compile to macOS 26 (flutter/flutter#195429)
2026-10-27 [email protected] Add benchmark for warmed pipeline usage (flutter/flutter#195001)
2026-10-27 [email protected] [flutter_tools] Add device log streaming support to DeviceService and ExtensionBackedDevice (flutter/flutter#194964)
2026-10-27 [email protected] Roll Dart SDK from 1d16c04dc346 to 65fc979759ca (4 revisions) (flutter/flutter#195383)

If this roll has caused a breakage, revert this CL and set the roller
to dry run mode using the controls here:
https://autoroll.skia.org/r/flutter-packages
Please CC [email protected],[email protected] on the revert to ensure that a human
is aware of the problem.

To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose

To report a problem with the AutoRoller itself, please file a bug:
https://issues.skia.org/issues/new?component=1389291&template=1850622

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md
suryaKommana2662 pushed a commit to suryaKommana2662/flutter that referenced this pull request Oct 29, 2026
… hook to the web benchmark recorder (flutter#195118)

<!--
Thanks for filing a pull request!
Reviewers are typically assigned within a week of filing a request.
To learn more about code review, see our documentation on Tree Hygiene:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
-->

Adds three opt-in, compile-time `--dart-define`s to the web
macrobenchmarks recorder (`dev/benchmarks/macrobenchmarks`). They let a
local benchmark host compute frame-time percentiles and start a CPU
profiler exactly when warm-up ends, without forking the app's reporting
protocol:

- `BENCHMARK_REPORT_SAMPLES=true`: `Profile.toJson` also reports two
flat lists of raw sample magnitudes (in microseconds, in recording
order) for each series, so `/profile-data` carries per-frame data:
`<series>.samples` (the measured samples) and `<series>.warmUpSamples`
(the warm-up samples). `scoreKeys` and the existing statistics are
unchanged.
- `BENCHMARK_MEASURED_SAMPLES=<n>`: overrides the number of measured
samples, which defaults to 100. `kDefaultTotalSampleCount` follows.
- `BENCHMARK_REPORT_PHASES=true`: installs a new static
`Profile.phaseListener`, which `Profile.stopWarmingUp()` calls with
`'measure'` at the warm-up → measure switch. `web_benchmarks.dart` uses
it to send a fire-and-forget `POST benchmark-phase` to the benchmark
server. The request is not awaited and is deferred with `Timer.run`, so
the frame loop and the first measured frame are not affected.

With none of the defines set, the output is the same as today, so CI and
the dashboards are unaffected. To check this, I ran `dart
dev/devicelab/bin/run.dart -t web_benchmarks_canvaskit` with
`BENCHMARK_TARGETS=draw_rect` on this branch and on its `master` base.
Both runs produced the same 13 `ResultData` keys and the same
`BenchmarkScoreKeys`. The devicelab host is not changed in this PR.

The new tests are in
`dev/benchmarks/macrobenchmarks/test/web/recorder_test.dart`. They
check:
- the exact legacy key set when sample reporting is off;
- that `<series>.samples` has only the measured samples and
`<series>.warmUpSamples` only the warm-up samples, in order, with
outliers kept, for reported and unreported series, and with each series
split at its own warm-up boundary;
- that `extraData` still merges last;
- that `phaseListener` fires exactly once with `'measure'`, after the
warm-up flag flips, on both automatic and manual warm-up;
- that nothing is called when no listener is set;
- that samples, warm-up samples, the scalar statistics and `scoreKeys`
survive a `jsonEncode`/`jsonDecode` round trip (compared numerically,
since the JS and Wasm encoders print doubles differently), and that the
decoded key set is still the legacy one when reporting is off.

`dev/bots` does not run the macrobenchmarks package tests, so I ran them
manually from `dev/benchmarks/macrobenchmarks` with `flutter test
--platform chrome test/web/recorder_test.dart` and `flutter test
--platform chrome --wasm test/web/recorder_test.dart` (11 tests pass on
both).

Part of flutter#191316

## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [AI contribution guidelines] and understand my
responsibilities, or I am not using AI tools.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant in-code documentation (doc comments with
`///`).
- [x] If this PR introduces a new feature or capability, I created and
linked a website documentation issue or PR in [flutter/website] (or
verified none is needed).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel
on [Discord].

If this change needs to override an active code freeze, provide a
comment explaining why. The code freeze workflow can be overridden by
code reviewers. See pinned issues for any active code freezes with
guidance.

**Note**: The Flutter team is currently trialing the use of [Gemini Code
Assist for
GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code).
Comments from the `gemini-code-assist` bot should not be taken as
authoritative feedback from the Flutter team. If you find its comments
useful you can update your code accordingly, but if you are unsure or
disagree with the feedback, please feel free to wait for a Flutter team
member's review for guidance on which automated comments should be
addressed.

<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[AI contribution guidelines]:
https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[flutter/website]: https://github.com/flutter/website
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CICD Run CI/CD platform-web Web applications specifically team-web Owned by Web platform team

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants