Skip to content

Commit 7a62b1f

Browse files
authored
Run 10 impeller iOS benchmarks as one devicelab task (flutter#195661)
draw_points, draw_vertices, draw_atlas, draw_arcs_all_fill_styles, draw_arcs_all_stroke_styles, rrect_blur, and *_perf_ios__timeline_summary tasks all build the same macrobenchmarks app with the same target and each draw for about 10 seconds. Each spends 1.4–2.6 minutes benchmarking inside a 5–10 minute bot run. This runs them in one task, `impeller_primitives_perf_suite_ios__timeline_summary`. Each benchmark still gets its own `flutter drive`, so the app is reinstalled and relaunched every time and nothing in-process carries over. Runs after the first reuse the already-built .app so the app is only built once. SkiaPerf traces are unchanged. TaskResult.merge prefixes each score key with the original task name and `metrics_center.parse` uploads it under that name. Tasks that don't use merge are unaffected. Per commit **this runs a 16 minutes benchmark on one Mac and iPhone instead of 76 minutes across 10 benchmarks**, saving ~100 device-hours a week. The suite becomes the longest Mac_ios task (~16 min), but these are postsubmit. Failure in any benchmark fails the task and drops that commit's metrics for all 10 benchmarks. The task runs every benchmark and reports all failures, not just the first. Delete the old builders (which also means release branch cp). Ideally we could run them next to each other for awhile, but unfortunately the way SkiaPerf works the metrics are the same, so they would stomp on each other and we wouldn't know which builder the data is coming from (they even upload in `bringup`). Let's see if any of the metrics move on this commit, if they do we can go back. [Test run build](https://ci.chromium.org/ui/p/flutter/builders/try/Mac_ios%20draw_arcs_all_fill_styles_perf_ios__timeline_summary/1/overview) - metrics at end of [stdout](https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8666540421791185473/+/u/run_impeller_primitives_perf_suite_ios__timeline_summary/stdout) - 14m total, 9.5m on the actual benchmark run step. Part of the work to reduce the number of Mac bots in CI. flutter#195654 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
1 parent 852534a commit 7a62b1f

19 files changed

Lines changed: 204 additions & 235 deletions

‎.ci.yaml‎

Lines changed: 3 additions & 83 deletions
Original file line numberDiff line numberDiff line change
@@ -3759,15 +3759,6 @@ targets:
37593759
["devicelab", "android", "linux", "pixel", "7pro"]
37603760
task_name: rrect_blur_perf__timeline_summary
37613761

3762-
- name: Mac_ios rrect_blur_perf_ios__timeline_summary
3763-
recipe: devicelab/devicelab_drone
3764-
presubmit: false
3765-
timeout: 60
3766-
properties:
3767-
tags: >
3768-
["devicelab", "ios", "mac"]
3769-
task_name: rrect_blur_perf_ios__timeline_summary
3770-
37713762
- name: Linux_pixel_7pro draw_arcs_all_fill_styles_perf__timeline_summary
37723763
recipe: devicelab/devicelab_drone
37733764
presubmit: false
@@ -3777,14 +3768,15 @@ targets:
37773768
["devicelab", "android", "linux", "pixel", "7pro"]
37783769
task_name: draw_arcs_all_fill_styles_perf__timeline_summary
37793770

3780-
- name: Mac_ios draw_arcs_all_fill_styles_perf_ios__timeline_summary
3771+
- name: Mac_ios impeller_primitives_perf_suite_ios__timeline_summary
37813772
recipe: devicelab/devicelab_drone
37823773
presubmit: false
3774+
bringup: true
37833775
timeout: 60
37843776
properties:
37853777
tags: >
37863778
["devicelab", "ios", "mac"]
3787-
task_name: draw_arcs_all_fill_styles_perf_ios__timeline_summary
3779+
task_name: impeller_primitives_perf_suite_ios__timeline_summary
37883780

37893781
- name: Linux_pixel_7pro draw_arcs_all_stroke_styles_perf__timeline_summary
37903782
recipe: devicelab/devicelab_drone
@@ -3795,15 +3787,6 @@ targets:
37953787
["devicelab", "android", "linux", "pixel", "7pro"]
37963788
task_name: draw_arcs_all_stroke_styles_perf__timeline_summary
37973789

3798-
- name: Mac_ios draw_arcs_all_stroke_styles_perf_ios__timeline_summary
3799-
recipe: devicelab/devicelab_drone
3800-
presubmit: false
3801-
timeout: 60
3802-
properties:
3803-
tags: >
3804-
["devicelab", "ios", "mac"]
3805-
task_name: draw_arcs_all_stroke_styles_perf_ios__timeline_summary
3806-
38073790
# Uses Impeller.
38083791
- name: Linux_pixel_7pro animated_blur_backdrop_filter_perf_opengles__timeline_summary
38093792
recipe: devicelab/devicelab_drone
@@ -3834,60 +3817,6 @@ targets:
38343817
["devicelab", "android", "linux", "pixel", "7pro"]
38353818
task_name: draw_vertices_perf__timeline_summary
38363819

3837-
- name: Mac_ios draw_vertices_perf_ios__timeline_summary
3838-
recipe: devicelab/devicelab_drone
3839-
presubmit: false
3840-
timeout: 60
3841-
properties:
3842-
tags: >
3843-
["devicelab", "ios", "mac"]
3844-
task_name: draw_vertices_perf_ios__timeline_summary
3845-
3846-
- name: Mac_ios draw_atlas_perf_ios__timeline_summary
3847-
recipe: devicelab/devicelab_drone
3848-
presubmit: false
3849-
timeout: 60
3850-
properties:
3851-
tags: >
3852-
["devicelab", "ios", "mac"]
3853-
task_name: draw_atlas_perf_ios__timeline_summary
3854-
3855-
- name: Mac_ios static_path_tessellation_perf_ios__timeline_summary
3856-
recipe: devicelab/devicelab_drone
3857-
presubmit: false
3858-
timeout: 60
3859-
properties:
3860-
tags: >
3861-
["devicelab", "ios", "mac"]
3862-
task_name: static_path_tessellation_perf_ios__timeline_summary
3863-
3864-
- name: Mac_ios dynamic_path_tessellation_perf_ios__timeline_summary
3865-
recipe: devicelab/devicelab_drone
3866-
presubmit: false
3867-
timeout: 60
3868-
properties:
3869-
tags: >
3870-
["devicelab", "ios", "mac"]
3871-
task_name: dynamic_path_tessellation_perf_ios__timeline_summary
3872-
3873-
- name: Mac_ios static_path_stroke_tessellation_perf_ios__timeline_summary
3874-
recipe: devicelab/devicelab_drone
3875-
presubmit: false
3876-
timeout: 60
3877-
properties:
3878-
tags: >
3879-
["devicelab", "ios", "mac"]
3880-
task_name: static_path_stroke_tessellation_perf_ios__timeline_summary
3881-
3882-
- name: Mac_ios dynamic_path_stroke_tessellation_perf_ios__timeline_summary
3883-
recipe: devicelab/devicelab_drone
3884-
presubmit: false
3885-
timeout: 60
3886-
properties:
3887-
tags: >
3888-
["devicelab", "ios", "mac"]
3889-
task_name: dynamic_path_stroke_tessellation_perf_ios__timeline_summary
3890-
38913820
- name: Mac_benchmark animated_complex_opacity_perf_macos__e2e_summary
38923821
presubmit: false
38933822
recipe: devicelab/devicelab_drone
@@ -6029,15 +5958,6 @@ targets:
60295958
["devicelab", "ios", "mac"]
60305959
task_name: animated_blur_backdrop_filter_perf_ios__timeline_summary
60315960

6032-
- name: Mac_ios draw_points_perf_ios__timeline_summary
6033-
recipe: devicelab/devicelab_drone
6034-
presubmit: false
6035-
timeout: 60
6036-
properties:
6037-
tags: >
6038-
["devicelab", "ios", "mac"]
6039-
task_name: draw_points_perf_ios__timeline_summary
6040-
60415961
- name: Mac_ios spell_check_test
60425962
recipe: devicelab/devicelab_drone
60435963
presubmit: false

‎TESTOWNERS‎

Lines changed: 1 addition & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -188,13 +188,7 @@
188188
/dev/devicelab/bin/tasks/complex_layout_scroll_perf_bad_ios__timeline_summary.dart @LongCatIsLooong @flutter/engine
189189
/dev/devicelab/bin/tasks/complex_layout_scroll_perf_ios__timeline_summary.dart @LongCatIsLooong @flutter/engine
190190
/dev/devicelab/bin/tasks/cubic_bezier_perf_ios_sksl_warmup__timeline_summary.dart @jtmcdole @flutter/engine
191-
/dev/devicelab/bin/tasks/draw_arcs_all_fill_styles_perf_ios__timeline_summary.dart @b-luk @flutter/engine
192-
/dev/devicelab/bin/tasks/draw_arcs_all_stroke_styles_perf_ios__timeline_summary.dart @b-luk @flutter/engine
193-
/dev/devicelab/bin/tasks/draw_atlas_perf_ios__timeline_summary.dart @cbracken @flutter/engine
194-
/dev/devicelab/bin/tasks/draw_points_perf_ios__timeline_summary.dart @cbracken @flutter/engine
195-
/dev/devicelab/bin/tasks/draw_vertices_perf_ios__timeline_summary.dart @cbracken @flutter/engine
196-
/dev/devicelab/bin/tasks/dynamic_path_stroke_tessellation_perf_ios__timeline_summary.dart @flar @flutter/engine
197-
/dev/devicelab/bin/tasks/dynamic_path_tessellation_perf_ios__timeline_summary.dart @cbracken @flutter/engine
191+
/dev/devicelab/bin/tasks/impeller_primitives_perf_suite_ios__timeline_summary.dart @gaaclarke @flutter/engine
198192
/dev/devicelab/bin/tasks/external_textures_integration_test_ios.dart @jtmcdole @flutter/engine
199193
/dev/devicelab/bin/tasks/flavors_test_ios.dart @vashworth @flutter/tool
200194
/dev/devicelab/bin/tasks/flutter_gallery__transition_perf_e2e_ios.dart @jtmcdole @flutter/engine
@@ -233,10 +227,7 @@
233227
/dev/devicelab/bin/tasks/platform_views_scroll_perf_non_intersecting_impeller_ios__timeline_summary.dart @hellohuanlin @flutter/engine
234228
/dev/devicelab/bin/tasks/post_backdrop_filter_perf_ios__timeline_summary.dart @hellohuanlin @flutter/engine
235229
/dev/devicelab/bin/tasks/route_test_ios.dart @okorohelijah @flutter/tool
236-
/dev/devicelab/bin/tasks/rrect_blur_perf_ios__timeline_summary.dart @gaaclarke @flutter/engine
237230
/dev/devicelab/bin/tasks/simple_animation_perf_ios.dart @LongCatIsLooong @flutter/engine
238-
/dev/devicelab/bin/tasks/static_path_stroke_tessellation_perf_ios__timeline_summary.dart @flar @flutter/engine
239-
/dev/devicelab/bin/tasks/static_path_tessellation_perf_ios__timeline_summary.dart @jonahwilliams @flutter/engine
240231
/dev/devicelab/bin/tasks/tiles_scroll_perf_ios__timeline_summary.dart @LongCatIsLooong @flutter/engine
241232
/dev/devicelab/bin/tasks/very_long_picture_scrolling_perf_ios__e2e_summary.dart @flar @flutter/engine
242233

‎dev/devicelab/bin/tasks/draw_arcs_all_fill_styles_perf_ios__timeline_summary.dart‎

Lines changed: 0 additions & 14 deletions
This file was deleted.

‎dev/devicelab/bin/tasks/draw_arcs_all_stroke_styles_perf_ios__timeline_summary.dart‎

Lines changed: 0 additions & 14 deletions
This file was deleted.

‎dev/devicelab/bin/tasks/draw_atlas_perf_ios__timeline_summary.dart‎

Lines changed: 0 additions & 14 deletions
This file was deleted.

‎dev/devicelab/bin/tasks/draw_points_perf_ios__timeline_summary.dart‎

Lines changed: 0 additions & 14 deletions
This file was deleted.

‎dev/devicelab/bin/tasks/draw_vertices_perf_ios__timeline_summary.dart‎

Lines changed: 0 additions & 14 deletions
This file was deleted.

‎dev/devicelab/bin/tasks/dynamic_path_stroke_tessellation_perf_ios__timeline_summary.dart‎

Lines changed: 0 additions & 14 deletions
This file was deleted.

‎dev/devicelab/bin/tasks/dynamic_path_tessellation_perf_ios__timeline_summary.dart‎

Lines changed: 0 additions & 14 deletions
This file was deleted.
Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
// Copyright 2014 The Flutter Authors. All rights reserved.
2+
// Use of this source code is governed by a BSD-style license that can be
3+
// found in the LICENSE file.
4+
5+
import 'package:flutter_devicelab/framework/devices.dart';
6+
import 'package:flutter_devicelab/framework/framework.dart';
7+
import 'package:flutter_devicelab/tasks/perf_tests.dart';
8+
9+
/// Keys are the former task names, which keeps their Skia Perf history.
10+
Future<void> main() async {
11+
deviceOperatingSystem = DeviceOperatingSystem.ios;
12+
await task(
13+
runBenchmarks(<String, TaskFunction>{
14+
'draw_points_perf_ios__timeline_summary': createDrawPointsPerfTest(),
15+
'draw_vertices_perf_ios__timeline_summary': createDrawVerticesPerfTest(),
16+
'draw_atlas_perf_ios__timeline_summary': createDrawAtlasPerfTest(),
17+
'draw_arcs_all_fill_styles_perf_ios__timeline_summary': createDrawArcsAllFillStylesPerfTest(),
18+
'draw_arcs_all_stroke_styles_perf_ios__timeline_summary':
19+
createDrawArcsAllStrokeStylesPerfTest(),
20+
'rrect_blur_perf_ios__timeline_summary': createRRectBlurPerfTest(enableImpeller: true),
21+
'static_path_tessellation_perf_ios__timeline_summary': createPathTessellationStaticPerfTest(),
22+
'dynamic_path_tessellation_perf_ios__timeline_summary':
23+
createPathTessellationDynamicPerfTest(),
24+
'static_path_stroke_tessellation_perf_ios__timeline_summary':
25+
createPathStrokeTessellationStaticPerfTest(),
26+
'dynamic_path_stroke_tessellation_perf_ios__timeline_summary':
27+
createPathStrokeTessellationDynamicPerfTest(),
28+
}),
29+
);
30+
}

0 commit comments

Comments
 (0)