Repository navigation
Commit b1e1fa6
authored
[web] Add opt-in raw samples, measured sample count and warm-up phase hook to the web benchmark recorder (flutter#195118)
<!--
Thanks for filing a pull request!
Reviewers are typically assigned within a week of filing a request.
To learn more about code review, see our documentation on Tree Hygiene:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
-->
Adds three opt-in, compile-time `--dart-define`s to the web
macrobenchmarks recorder (`dev/benchmarks/macrobenchmarks`). They let a
local benchmark host compute frame-time percentiles and start a CPU
profiler exactly when warm-up ends, without forking the app's reporting
protocol:
- `BENCHMARK_REPORT_SAMPLES=true`: `Profile.toJson` also reports two
flat lists of raw sample magnitudes (in microseconds, in recording
order) for each series, so `/profile-data` carries per-frame data:
`<series>.samples` (the measured samples) and `<series>.warmUpSamples`
(the warm-up samples). `scoreKeys` and the existing statistics are
unchanged.
- `BENCHMARK_MEASURED_SAMPLES=<n>`: overrides the number of measured
samples, which defaults to 100. `kDefaultTotalSampleCount` follows.
- `BENCHMARK_REPORT_PHASES=true`: installs a new static
`Profile.phaseListener`, which `Profile.stopWarmingUp()` calls with
`'measure'` at the warm-up → measure switch. `web_benchmarks.dart` uses
it to send a fire-and-forget `POST benchmark-phase` to the benchmark
server. The request is not awaited and is deferred with `Timer.run`, so
the frame loop and the first measured frame are not affected.
With none of the defines set, the output is the same as today, so CI and
the dashboards are unaffected. To check this, I ran `dart
dev/devicelab/bin/run.dart -t web_benchmarks_canvaskit` with
`BENCHMARK_TARGETS=draw_rect` on this branch and on its `master` base.
Both runs produced the same 13 `ResultData` keys and the same
`BenchmarkScoreKeys`. The devicelab host is not changed in this PR.
The new tests are in
`dev/benchmarks/macrobenchmarks/test/web/recorder_test.dart`. They
check:
- the exact legacy key set when sample reporting is off;
- that `<series>.samples` has only the measured samples and
`<series>.warmUpSamples` only the warm-up samples, in order, with
outliers kept, for reported and unreported series, and with each series
split at its own warm-up boundary;
- that `extraData` still merges last;
- that `phaseListener` fires exactly once with `'measure'`, after the
warm-up flag flips, on both automatic and manual warm-up;
- that nothing is called when no listener is set;
- that samples, warm-up samples, the scalar statistics and `scoreKeys`
survive a `jsonEncode`/`jsonDecode` round trip (compared numerically,
since the JS and Wasm encoders print doubles differently), and that the
decoded key set is still the legacy one when reporting is off.
`dev/bots` does not run the macrobenchmarks package tests, so I ran them
manually from `dev/benchmarks/macrobenchmarks` with `flutter test
--platform chrome test/web/recorder_test.dart` and `flutter test
--platform chrome --wasm test/web/recorder_test.dart` (11 tests pass on
both).
Part of flutter#191316
## Pre-launch Checklist
- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [AI contribution guidelines] and understand my
responsibilities, or I am not using AI tools.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant in-code documentation (doc comments with
`///`).
- [x] If this PR introduces a new feature or capability, I created and
linked a website documentation issue or PR in [flutter/website] (or
verified none is needed).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.
If you need help, consider asking for advice on the #hackers-new channel
on [Discord].
If this change needs to override an active code freeze, provide a
comment explaining why. The code freeze workflow can be overridden by
code reviewers. See pinned issues for any active code freezes with
guidance.
**Note**: The Flutter team is currently trialing the use of [Gemini Code
Assist for
GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code).
Comments from the `gemini-code-assist` bot should not be taken as
authoritative feedback from the Flutter team. If you find its comments
useful you can update your code accordingly, but if you are unsure or
disagree with the feedback, please feel free to wait for a Flutter team
member's review for guidance on which automated comments should be
addressed.
<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[AI contribution guidelines]:
https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[flutter/website]: https://github.com/flutter/website
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md1 parent 380cb8d commit b1e1fa6
3 files changed
Lines changed: 419 additions & 2 deletions
File tree
- dev/benchmarks/macrobenchmarks
- lib
- src/web
- test/web
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | | - | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
31 | 39 | | |
32 | 40 | | |
33 | 41 | | |
34 | 42 | | |
35 | 43 | | |
36 | 44 | | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
37 | 67 | | |
38 | 68 | | |
39 | 69 | | |
| |||
868 | 898 | | |
869 | 899 | | |
870 | 900 | | |
| 901 | + | |
| 902 | + | |
| 903 | + | |
| 904 | + | |
| 905 | + | |
| 906 | + | |
| 907 | + | |
871 | 908 | | |
872 | 909 | | |
873 | 910 | | |
| |||
894 | 931 | | |
895 | 932 | | |
896 | 933 | | |
| 934 | + | |
897 | 935 | | |
898 | 936 | | |
899 | 937 | | |
| |||
1024 | 1062 | | |
1025 | 1063 | | |
1026 | 1064 | | |
1027 | | - | |
| 1065 | + | |
| 1066 | + | |
| 1067 | + | |
| 1068 | + | |
| 1069 | + | |
| 1070 | + | |
| 1071 | + | |
| 1072 | + | |
| 1073 | + | |
| 1074 | + | |
1028 | 1075 | | |
1029 | 1076 | | |
1030 | 1077 | | |
| |||
1044 | 1091 | | |
1045 | 1092 | | |
1046 | 1093 | | |
| 1094 | + | |
| 1095 | + | |
| 1096 | + | |
| 1097 | + | |
| 1098 | + | |
| 1099 | + | |
| 1100 | + | |
| 1101 | + | |
| 1102 | + | |
| 1103 | + | |
1047 | 1104 | | |
1048 | 1105 | | |
1049 | 1106 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
91 | 91 | | |
92 | 92 | | |
93 | 93 | | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
94 | 102 | | |
95 | 103 | | |
96 | 104 | | |
| |||
117 | 125 | | |
118 | 126 | | |
119 | 127 | | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
120 | 135 | | |
121 | 136 | | |
122 | 137 | | |
| |||
408 | 423 | | |
409 | 424 | | |
410 | 425 | | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
411 | 450 | | |
412 | 451 | | |
413 | 452 | | |
| |||
0 commit comments