Skip to content

Commit 9eab636

Browse files
Add web benchmark baseline comparison, ANSI summary table, and structured JSON helpers (flutter#194609)
Adds pure, unit-tested helpers in `dev/devicelab/lib/tasks/web_benchmarks_comparison.dart` that compare a web benchmark run against a saved baseline and print the result as a terminal table. A follow-up PR uses them in a local benchmark CLI. - `compareResults` compares the `.average` metrics that appear in both the current and the baseline `TaskResult.data` maps, sorted by key; all other keys are ignored. It computes each metric's percentage change and flags a regression when the change exceeds an optional `regressionThreshold`. Speedups are never regressions, and a metric that rises from zero has an infinite delta. Invalid thresholds throw `ArgumentError`, and compared values that are not finite, non-negative numbers throw `FormatException`. - `formatAnsiSummaryTable` prints a `Metric` / `Baseline` / `Current` / `Delta` / `Status` table with a header and separator row, right-aligned numbers, and an optionally ANSI-colored `PASS`/`FAIL` status. - Adds `kSmokeTestBenchmarks` to `web_benchmarks.dart`, the curated fast benchmark set (`draw_rect`, `bench_card_infinite_scroll`) for the CLI's `--smoke-test` flag. - Adds unit tests in `dev/devicelab/test/tasks/web_benchmarks_test.dart`: a table-driven matrix of delta and threshold cases (including zero baselines, no threshold and a delta equal to the threshold), key selection, input validation, and an exact expected-table test. Part of flutter#191316 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#ai-contribution-guidelines [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
1 parent b4262ef commit 9eab636

3 files changed

Lines changed: 329 additions & 1 deletion

File tree

‎dev/devicelab/lib/tasks/web_benchmarks.dart‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -510,6 +510,9 @@ Handler createBuildDirectoryHandler(String buildDirectoryPath) {
510510
};
511511
}
512512

513+
/// Standard curated fast smoke-test benchmarks.
514+
const List<String> kSmokeTestBenchmarks = <String>['draw_rect', 'bench_card_infinite_scroll'];
515+
513516
/// Filters benchmarks based on an exact list of targets or a regular expression pattern.
514517
///
515518
/// If [targetBenchmarks] is provided, only the exact benchmarks listed will be run. If
Lines changed: 167 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,167 @@
1+
// Copyright 2014 The Flutter Authors. All rights reserved.
2+
// Use of this source code is governed by a BSD-style license that can be
3+
// found in the LICENSE file.
4+
5+
import 'dart:math' as math;
6+
7+
/// The comparison of a single metric between a baseline run and the current run.
8+
class MetricComparison {
9+
/// Creates a comparison of [metricKey] between [baseline] and [current].
10+
const MetricComparison({
11+
required this.metricKey,
12+
required this.baseline,
13+
required this.current,
14+
required this.deltaPercent,
15+
required this.isRegression,
16+
});
17+
18+
/// The flattened metric key, such as `draw_rect.canvaskit.totalUiFrame.average`.
19+
final String metricKey;
20+
21+
/// The value of the metric in the baseline run.
22+
final double baseline;
23+
24+
/// The value of the metric in the current run.
25+
final double current;
26+
27+
/// The change from [baseline] to [current], as a percentage of [baseline].
28+
///
29+
/// Positive values are slowdowns. This is [double.infinity] when [baseline]
30+
/// is zero and [current] is not.
31+
final double deltaPercent;
32+
33+
/// Whether [deltaPercent] exceeds the regression threshold.
34+
final bool isRegression;
35+
}
36+
37+
/// The result of [compareResults].
38+
class BenchmarkComparison {
39+
/// Creates a comparison of [metrics].
40+
const BenchmarkComparison(this.metrics);
41+
42+
/// The compared metrics, which [compareResults] sorts by
43+
/// [MetricComparison.metricKey].
44+
final List<MetricComparison> metrics;
45+
46+
/// Whether any of the [metrics] is a regression.
47+
bool get hasRegressions => metrics.any((MetricComparison m) => m.isRegression);
48+
}
49+
50+
/// Compares the `.average` metrics in [current] against those in [baseline].
51+
///
52+
/// Both maps are flattened `TaskResult.data` maps, such as
53+
/// `{'draw_rect.canvaskit.totalUiFrame.average': 123.4}`. Only keys that end
54+
/// in `.average` and appear in both maps are compared; all other keys are
55+
/// ignored. The resulting metrics are sorted by key.
56+
///
57+
/// All metrics are durations or counts, so higher is worse. A metric is a
58+
/// regression if [regressionThreshold] is non-null and the metric's
59+
/// [MetricComparison.deltaPercent] is greater than it, so speedups are never
60+
/// regressions.
61+
///
62+
/// Throws an [ArgumentError] if [regressionThreshold] is negative or not
63+
/// finite, and a [FormatException] if a compared value is not a finite,
64+
/// non-negative number.
65+
BenchmarkComparison compareResults({
66+
required Map<String, Object?> current,
67+
required Map<String, Object?> baseline,
68+
double? regressionThreshold,
69+
}) {
70+
if (regressionThreshold != null && (regressionThreshold < 0 || !regressionThreshold.isFinite)) {
71+
throw ArgumentError.value(
72+
regressionThreshold,
73+
'regressionThreshold',
74+
'must be a finite, non-negative number',
75+
);
76+
}
77+
78+
double read(Map<String, Object?> values, String key) {
79+
final Object? value = values[key];
80+
if (value is! num || !value.isFinite || value < 0) {
81+
throw FormatException('Expected a finite, non-negative number for $key, but got $value.');
82+
}
83+
return value.toDouble();
84+
}
85+
86+
final List<String> keys =
87+
baseline.keys
88+
.where((String key) => key.endsWith('.average') && current.containsKey(key))
89+
.toList()
90+
..sort();
91+
return BenchmarkComparison(<MetricComparison>[
92+
for (final key in keys)
93+
_compareMetric(key, read(baseline, key), read(current, key), regressionThreshold),
94+
]);
95+
}
96+
97+
MetricComparison _compareMetric(
98+
String key,
99+
double baseline,
100+
double current,
101+
double? regressionThreshold,
102+
) {
103+
final double deltaPercent = baseline == 0
104+
? (current == 0 ? 0.0 : double.infinity)
105+
: (current - baseline) / baseline * 100;
106+
return MetricComparison(
107+
metricKey: key,
108+
baseline: baseline,
109+
current: current,
110+
deltaPercent: deltaPercent,
111+
isRegression: regressionThreshold != null && deltaPercent > regressionThreshold,
112+
);
113+
}
114+
115+
/// Formats [comparison] as a table for terminal output.
116+
///
117+
/// The table has a header row, a separator row, and one row per metric, with
118+
/// the columns `Metric`, `Baseline`, `Current`, `Delta` and `Status` two spaces
119+
/// apart and the numeric columns right-aligned. An infinite delta renders as
120+
/// `N/A`. The status is `FAIL` for regressions and `PASS` otherwise.
121+
///
122+
/// If [enableAnsi] is true, the status is colored red or green; pass false
123+
/// when the output is not a terminal.
124+
///
125+
/// Returns an empty string if [comparison] has no metrics.
126+
String formatAnsiSummaryTable(BenchmarkComparison comparison, {required bool enableAnsi}) {
127+
if (comparison.metrics.isEmpty) {
128+
return '';
129+
}
130+
final rows = <List<String>>[
131+
<String>['Metric', 'Baseline', 'Current', 'Delta', 'Status'],
132+
for (final MetricComparison m in comparison.metrics)
133+
<String>[
134+
m.metricKey,
135+
m.baseline.toStringAsFixed(1),
136+
m.current.toStringAsFixed(1),
137+
if (m.deltaPercent.isFinite) '${m.deltaPercent.toStringAsFixed(1)}%' else 'N/A',
138+
if (m.isRegression) 'FAIL' else 'PASS',
139+
],
140+
];
141+
final widths = <int>[
142+
for (var column = 0; column < rows.first.length; column++)
143+
rows.map((List<String> row) => row[column].length).reduce(math.max),
144+
];
145+
146+
final buffer = StringBuffer();
147+
for (var i = 0; i < rows.length; i++) {
148+
final List<String> row = rows[i];
149+
// The status column is last, so it is left unpadded to avoid trailing
150+
// whitespace, and colors do not affect the alignment.
151+
String status = row[4];
152+
if (enableAnsi && i > 0) {
153+
status = '${comparison.metrics[i - 1].isRegression ? '\x1B[31m' : '\x1B[32m'}$status\x1B[0m';
154+
}
155+
buffer.writeln(
156+
<String>[
157+
row[0].padRight(widths[0]),
158+
for (var column = 1; column < 4; column++) row[column].padLeft(widths[column]),
159+
status,
160+
].join(' '),
161+
);
162+
if (i == 0) {
163+
buffer.writeln(widths.map((int width) => '-' * width).join(' '));
164+
}
165+
}
166+
return buffer.toString();
167+
}

‎dev/devicelab/test/tasks/web_benchmarks_test.dart‎

Lines changed: 159 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,9 @@
44

55
import 'package:flutter_devicelab/framework/task_result.dart';
66
import 'package:flutter_devicelab/tasks/web_benchmarks.dart';
7-
import 'package:test/test.dart';
7+
import 'package:flutter_devicelab/tasks/web_benchmarks_comparison.dart';
8+
9+
import '../common.dart';
810

911
void main() {
1012
group('BenchmarkFilter', () {
@@ -325,6 +327,162 @@ void main() {
325327
expect(result.message, contains('BENCHMARK_FILTER'));
326328
});
327329
});
330+
331+
group('compareResults', () {
332+
for (final (
333+
String description,
334+
num baseline,
335+
num current,
336+
double? threshold,
337+
double delta,
338+
bool regression,
339+
)
340+
in <(String, num, num, double?, double, bool)>[
341+
('slowdown above the threshold is a regression', 100.0, 115.0, 10.0, 15.0, true),
342+
('slowdown within the threshold is not a regression', 100.0, 105.0, 10.0, 5.0, false),
343+
('slowdown equal to the threshold is not a regression', 100.0, 110.0, 10.0, 10.0, false),
344+
('speedup beyond the threshold is not a regression', 100.0, 75.0, 10.0, -25.0, false),
345+
('slowdown without a threshold is not a regression', 100.0, 150.0, null, 50.0, false),
346+
('any slowdown is a regression at threshold 0', 100.0, 101.0, 0.0, 1.0, true),
347+
('no change is not a regression at threshold 0', 100.0, 100.0, 0.0, 0.0, false),
348+
('rise from zero is an infinite regression', 0.0, 150.0, 10.0, double.infinity, true),
349+
(
350+
'rise from zero without a threshold is not a regression',
351+
0.0,
352+
150.0,
353+
null,
354+
double.infinity,
355+
false,
356+
),
357+
('zero to zero is unchanged', 0.0, 0.0, 10.0, 0.0, false),
358+
('integer values are accepted', 100, 150, 10.0, 50.0, true),
359+
]) {
360+
test(description, () {
361+
final BenchmarkComparison result = compareResults(
362+
baseline: <String, dynamic>{'a.average': baseline},
363+
current: <String, dynamic>{'a.average': current},
364+
regressionThreshold: threshold,
365+
);
366+
final MetricComparison metric = result.metrics.single;
367+
expect(metric.metricKey, 'a.average');
368+
expect(metric.baseline, baseline);
369+
expect(metric.current, current);
370+
expect(metric.deltaPercent, delta);
371+
expect(metric.isRegression, regression);
372+
expect(result.hasRegressions, regression);
373+
});
374+
}
375+
376+
test('compares only .average keys present in both maps, sorted by key', () {
377+
final BenchmarkComparison result = compareResults(
378+
baseline: <String, dynamic>{
379+
'c.average': 1,
380+
'a.average': 1,
381+
'a.median': double.nan,
382+
'baseline_only.average': -1,
383+
'name': 'draw_rect',
384+
},
385+
current: <String, dynamic>{
386+
'a.average': 1,
387+
'c.average': 1,
388+
'a.median': double.nan,
389+
'current_only.average': null,
390+
},
391+
);
392+
expect(result.metrics.map((MetricComparison m) => m.metricKey), <String>[
393+
'a.average',
394+
'c.average',
395+
]);
396+
expect(
397+
compareResults(
398+
baseline: <String, dynamic>{'a.average': 1},
399+
current: <String, dynamic>{'b.average': 1},
400+
).metrics,
401+
isEmpty,
402+
);
403+
});
404+
405+
test('rejects negative and non-finite thresholds', () {
406+
for (final threshold in <double>[-5.0, double.nan, double.infinity]) {
407+
expect(
408+
() => compareResults(
409+
baseline: <String, dynamic>{'a.average': 1},
410+
current: <String, dynamic>{'a.average': 1},
411+
regressionThreshold: threshold,
412+
),
413+
throwsArgumentError,
414+
reason: '$threshold',
415+
);
416+
}
417+
});
418+
419+
test('rejects compared values that are not finite, non-negative numbers', () {
420+
for (final value in <Object?>[-1.0, double.nan, double.infinity, 'fast', null]) {
421+
expect(
422+
() => compareResults(
423+
baseline: <String, dynamic>{'a.average': value},
424+
current: <String, dynamic>{'a.average': 1},
425+
),
426+
throwsFormatException,
427+
reason: 'baseline $value',
428+
);
429+
expect(
430+
() => compareResults(
431+
baseline: <String, dynamic>{'a.average': 1},
432+
current: <String, dynamic>{'a.average': value},
433+
),
434+
throwsFormatException,
435+
reason: 'current $value',
436+
);
437+
}
438+
});
439+
});
440+
441+
group('formatAnsiSummaryTable', () {
442+
final BenchmarkComparison comparison = compareResults(
443+
baseline: <String, dynamic>{
444+
'zero.average': 0,
445+
'a.average': 4915.7,
446+
'bench_card_infinite_scroll.canvaskit.apply_frame.average': 67.2,
447+
'huge.average': 123456789,
448+
},
449+
current: <String, dynamic>{
450+
'zero.average': 2.2,
451+
'a.average': 4596.2,
452+
'bench_card_infinite_scroll.canvaskit.apply_frame.average': 75.6,
453+
'huge.average': 123457,
454+
},
455+
regressionThreshold: 10.0,
456+
);
457+
458+
test('renders an aligned table with a header and separator', () {
459+
expect(formatAnsiSummaryTable(comparison, enableAnsi: false), '''
460+
Metric Baseline Current Delta Status
461+
-------------------------------------------------------- ----------- -------- ------ ------
462+
a.average 4915.7 4596.2 -6.5% PASS
463+
bench_card_infinite_scroll.canvaskit.apply_frame.average 67.2 75.6 12.5% FAIL
464+
huge.average 123456789.0 123457.0 -99.9% PASS
465+
zero.average 0.0 2.2 N/A FAIL
466+
''');
467+
});
468+
469+
test('colors only the status when ANSI is enabled', () {
470+
final String ansi = formatAnsiSummaryTable(comparison, enableAnsi: true);
471+
expect(ansi, contains('\x1B[31mFAIL\x1B[0m'));
472+
expect(ansi, contains('\x1B[32mPASS\x1B[0m'));
473+
expect(
474+
ansi.replaceAll(RegExp(r'\x1B\[\d+m'), ''),
475+
formatAnsiSummaryTable(comparison, enableAnsi: false),
476+
);
477+
});
478+
479+
test('returns an empty string when there are no metrics', () {
480+
expect(
481+
formatAnsiSummaryTable(const BenchmarkComparison(<MetricComparison>[]), enableAnsi: true),
482+
isEmpty,
483+
);
484+
});
485+
});
328486
}
329487

330488
/// Matches the exception thrown when target benchmarks and a filter pattern are

0 commit comments

Comments
 (0)