Skip to content

Commit b1e1fa6

Browse files
[web] Add opt-in raw samples, measured sample count and warm-up phase hook to the web benchmark recorder (flutter#195118)
<!-- Thanks for filing a pull request! Reviewers are typically assigned within a week of filing a request. To learn more about code review, see our documentation on Tree Hygiene: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md --> Adds three opt-in, compile-time `--dart-define`s to the web macrobenchmarks recorder (`dev/benchmarks/macrobenchmarks`). They let a local benchmark host compute frame-time percentiles and start a CPU profiler exactly when warm-up ends, without forking the app's reporting protocol: - `BENCHMARK_REPORT_SAMPLES=true`: `Profile.toJson` also reports two flat lists of raw sample magnitudes (in microseconds, in recording order) for each series, so `/profile-data` carries per-frame data: `<series>.samples` (the measured samples) and `<series>.warmUpSamples` (the warm-up samples). `scoreKeys` and the existing statistics are unchanged. - `BENCHMARK_MEASURED_SAMPLES=<n>`: overrides the number of measured samples, which defaults to 100. `kDefaultTotalSampleCount` follows. - `BENCHMARK_REPORT_PHASES=true`: installs a new static `Profile.phaseListener`, which `Profile.stopWarmingUp()` calls with `'measure'` at the warm-up → measure switch. `web_benchmarks.dart` uses it to send a fire-and-forget `POST benchmark-phase` to the benchmark server. The request is not awaited and is deferred with `Timer.run`, so the frame loop and the first measured frame are not affected. With none of the defines set, the output is the same as today, so CI and the dashboards are unaffected. To check this, I ran `dart dev/devicelab/bin/run.dart -t web_benchmarks_canvaskit` with `BENCHMARK_TARGETS=draw_rect` on this branch and on its `master` base. Both runs produced the same 13 `ResultData` keys and the same `BenchmarkScoreKeys`. The devicelab host is not changed in this PR. The new tests are in `dev/benchmarks/macrobenchmarks/test/web/recorder_test.dart`. They check: - the exact legacy key set when sample reporting is off; - that `<series>.samples` has only the measured samples and `<series>.warmUpSamples` only the warm-up samples, in order, with outliers kept, for reported and unreported series, and with each series split at its own warm-up boundary; - that `extraData` still merges last; - that `phaseListener` fires exactly once with `'measure'`, after the warm-up flag flips, on both automatic and manual warm-up; - that nothing is called when no listener is set; - that samples, warm-up samples, the scalar statistics and `scoreKeys` survive a `jsonEncode`/`jsonDecode` round trip (compared numerically, since the JS and Wasm encoders print doubles differently), and that the decoded key set is still the legacy one when reporting is off. `dev/bots` does not run the macrobenchmarks package tests, so I ran them manually from `dev/benchmarks/macrobenchmarks` with `flutter test --platform chrome test/web/recorder_test.dart` and `flutter test --platform chrome --wasm test/web/recorder_test.dart` (11 tests pass on both). Part of flutter#191316 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
1 parent 380cb8d commit b1e1fa6

3 files changed

Lines changed: 419 additions & 2 deletions

File tree

‎dev/benchmarks/macrobenchmarks/lib/src/web/recorder.dart‎

Lines changed: 59 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -27,13 +27,43 @@ const int _kDefaultWarmUpSampleCount = 200;
2727
/// The default number of samples collected to compute benchmark statistics.
2828
///
2929
/// This value is used when [Profile.useCustomWarmUp] is set to false.
30-
const int _kDefaultMeasuredSampleCount = 100;
30+
///
31+
/// Defaults to 100 and can be overridden at compile time with
32+
/// `--dart-define=BENCHMARK_MEASURED_SAMPLES=<n>`, where `<n>` must be a
33+
/// positive integer no greater than 800 (`dynamic_clip_on_static_picture`
34+
/// scrolls out of its picture beyond that).
35+
const int _kDefaultMeasuredSampleCount = int.fromEnvironment(
36+
'BENCHMARK_MEASURED_SAMPLES',
37+
defaultValue: 100,
38+
);
3139

3240
/// The default total number of samples collected by a benchmark.
3341
///
3442
/// This value is used when [Profile.useCustomWarmUp] is set to false.
3543
const int kDefaultTotalSampleCount = _kDefaultWarmUpSampleCount + _kDefaultMeasuredSampleCount;
3644

45+
/// Whether [Profile.toJson] includes the raw samples of each series.
46+
///
47+
/// When enabled, each series reports its measured samples under
48+
/// `<series>.samples` and its warm-up samples under `<series>.warmUpSamples`,
49+
/// both as sample magnitudes in microseconds.
50+
///
51+
/// Opt-in via `--dart-define=BENCHMARK_REPORT_SAMPLES=true`. When unset, the
52+
/// reported JSON is unchanged.
53+
const bool _kReportSamples = bool.fromEnvironment('BENCHMARK_REPORT_SAMPLES');
54+
55+
/// A phase of a benchmark run reported to a [BenchmarkPhaseListener].
56+
enum BenchmarkPhase {
57+
/// Warm-up has ended and measured samples start being recorded.
58+
measure,
59+
}
60+
61+
/// Called when a benchmark named [benchmarkName] enters a new [phase].
62+
///
63+
/// Currently the only reported phase is [BenchmarkPhase.measure], which is
64+
/// reported when warm-up ends and measured samples start being recorded.
65+
typedef BenchmarkPhaseListener = void Function(String benchmarkName, BenchmarkPhase phase);
66+
3767
/// A benchmark metric that includes frame-related computations prior to
3868
/// submitting layer and picture operations to the underlying renderer, such as
3969
/// CanvasKit. During this phase we compute transforms, clips, and other
@@ -868,6 +898,13 @@ class Profile {
868898
/// [kDefaultTotalSampleCount] samples and stops automatically.
869899
Profile({required this.name, this.useCustomWarmUp = false});
870900

901+
/// An optional listener notified when any profile changes phase.
902+
///
903+
/// Null by default, in which case no notifications are sent. It is invoked
904+
/// synchronously from [stopWarmingUp] with [BenchmarkPhase.measure], so it
905+
/// must not block.
906+
static BenchmarkPhaseListener? phaseListener;
907+
871908
/// The name of the benchmark that produced this profile.
872909
final String name;
873910

@@ -894,6 +931,7 @@ class Profile {
894931
throw StateError('Warm-up already stopped.');
895932
} else {
896933
_isWarmingUp = false;
934+
Profile.phaseListener?.call(name, BenchmarkPhase.measure);
897935
}
898936
}
899937

@@ -1024,7 +1062,16 @@ class Profile {
10241062

10251063
/// Returns a JSON representation of the profile that will be sent to the
10261064
/// server.
1027-
Map<String, dynamic> toJson() {
1065+
///
1066+
/// If [reportSamples] is true, each series also reports two flat lists of
1067+
/// raw sample magnitudes in microseconds, both in recording order:
1068+
///
1069+
/// * `<series>.samples`: the measured (non-warm-up) samples.
1070+
/// * `<series>.warmUpSamples`: the warm-up samples.
1071+
///
1072+
/// It defaults to the opt-in `--dart-define=BENCHMARK_REPORT_SAMPLES=true`
1073+
/// setting, which is off. When off, neither key is reported.
1074+
Map<String, dynamic> toJson({bool reportSamples = _kReportSamples}) {
10281075
final scoreKeys = <String>[];
10291076
final json = <String, dynamic>{'name': name, 'scoreKeys': scoreKeys};
10301077

@@ -1044,6 +1091,16 @@ class Profile {
10441091
json['$key.outlierAverage'] = stats.outlierAverage;
10451092
json['$key.outlierRatio'] = stats.outlierRatio;
10461093
json['$key.noise'] = stats.noise;
1094+
if (reportSamples) {
1095+
json['$key.samples'] = <double>[
1096+
for (final AnnotatedSample sample in stats.samples)
1097+
if (!sample.isWarmUpValue) sample.magnitude,
1098+
];
1099+
json['$key.warmUpSamples'] = <double>[
1100+
for (final AnnotatedSample sample in stats.samples)
1101+
if (sample.isWarmUpValue) sample.magnitude,
1102+
];
1103+
}
10471104
}
10481105

10491106
json.addAll(extraData);

‎dev/benchmarks/macrobenchmarks/lib/web_benchmarks.dart‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,14 @@ final Map<String, RecorderFactory> benchmarks = <String, RecorderFactory>{
9191

9292
late final LocalBenchmarkServerClient _client;
9393

94+
/// Whether to notify the benchmark server when a benchmark switches from
95+
/// warm-up to measurement.
96+
///
97+
/// Opt-in via `--dart-define=BENCHMARK_REPORT_PHASES=true`. When enabled, a
98+
/// fire-and-forget `POST benchmark-phase` request (resolved against the
99+
/// benchmark server origin) is sent at the switch.
100+
const bool _kReportPhases = bool.fromEnvironment('BENCHMARK_REPORT_PHASES');
101+
94102
Future<void> main(List<String> args) async {
95103
final parser = ArgParser()
96104
..addOption(
@@ -117,6 +125,13 @@ Future<void> main(List<String> args) async {
117125
return;
118126
}
119127

128+
if (_kReportPhases) {
129+
// The listener is called synchronously from `Profile.stopWarmingUp`, which
130+
// may run in the middle of a timed frame. Defer the request to a new task
131+
// so that its setup cost is not attributed to the first measured frame.
132+
Profile.phaseListener = (String name, BenchmarkPhase phase) =>
133+
Timer.run(() => unawaited(_client.reportPhase(name, phase)));
134+
}
120135
await _runBenchmark(nextBenchmark);
121136
web.window.location.reload();
122137
}
@@ -408,6 +423,30 @@ class LocalBenchmarkServerClient {
408423
);
409424
}
410425

426+
/// Tells the local benchmark server that the benchmark [name] entered
427+
/// [phase] (currently only [BenchmarkPhase.measure], sent when warm-up
428+
/// ends). The phase is sent as its `name`.
429+
///
430+
/// Only used when `--dart-define=BENCHMARK_REPORT_PHASES=true` is set. The
431+
/// response status is not checked, so a 404 from servers that do not handle
432+
/// this request is ignored, and network errors are caught and ignored too.
433+
/// Unlike the other requests this does not check for manual mode: the phase
434+
/// listener is only installed when a benchmark server is driving the run.
435+
Future<void> reportPhase(String name, BenchmarkPhase phase) async {
436+
try {
437+
await _requestXhr(
438+
serverOrigin.resolve('benchmark-phase'),
439+
requestHeaders: headers,
440+
method: 'POST',
441+
mimeType: 'application/json',
442+
sendData: json.encode(<String, String>{'name': name, 'phase': phase.name}),
443+
);
444+
} catch (_) {
445+
// `_requestXhr` completes normally on non-200 responses, so this only
446+
// catches network errors, which are ignored.
447+
}
448+
}
449+
411450
/// Sends the profile data collected by the benchmark to the local benchmark
412451
/// server.
413452
Future<void> sendProfileData(Profile profile) async {

0 commit comments

Comments
 (0)