Repository navigation
[CP-beta]Run 10 impeller iOS benchmarks as one devicelab task - #196009
Conversation
draw_points, draw_vertices, draw_atlas, draw_arcs_all_fill_styles, draw_arcs_all_stroke_styles, rrect_blur, and *_perf_ios__timeline_summary tasks all build the same macrobenchmarks app with the same target and each draw for about 10 seconds. Each spends 1.4–2.6 minutes benchmarking inside a 5–10 minute bot run. This runs them in one task, `impeller_primitives_perf_suite_ios__timeline_summary`. Each benchmark still gets its own `flutter drive`, so the app is reinstalled and relaunched every time and nothing in-process carries over. Runs after the first reuse the already-built .app so the app is only built once. SkiaPerf traces are unchanged. TaskResult.merge prefixes each score key with the original task name and `metrics_center.parse` uploads it under that name. Tasks that don't use merge are unaffected. Per commit **this runs a 16 minutes benchmark on one Mac and iPhone instead of 76 minutes across 10 benchmarks**, saving ~100 device-hours a week. The suite becomes the longest Mac_ios task (~16 min), but these are postsubmit. Failure in any benchmark fails the task and drops that commit's metrics for all 10 benchmarks. The task runs every benchmark and reports all failures, not just the first. Delete the old builders (which also means release branch cp). Ideally we could run them next to each other for awhile, but unfortunately the way SkiaPerf works the metrics are the same, so they would stomp on each other and we wouldn't know which builder the data is coming from (they even upload in `bringup`). Let's see if any of the metrics move on this commit, if they do we can go back. [Test run build](https://ci.chromium.org/ui/p/flutter/builders/try/Mac_ios%20draw_arcs_all_fill_styles_perf_ios__timeline_summary/1/overview) - metrics at end of [stdout](https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8666540421791185473/+/u/run_impeller_primitives_perf_suite_ios__timeline_summary/stdout) - 14m total, 9.5m on the actual benchmark run step. Part of the work to reduce the number of Mac bots in CI. flutter#195654 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
|
@jmagman please fill out the PR description above, afterwards the release team will review this request. |
|
This pull request was opened from and to a release candidate branch. This should only be done as part of the official Flutter release process. If you are attempting to make a regular contribution to the Flutter project, please close this PR and follow the instructions at Tree Hygiene for detailed instructions on contributing to Flutter. Reviewers: Use caution before merging pull requests to release branches. Ensure the proper procedure has been followed. |
There was a problem hiding this comment.
Code Review
This pull request consolidates multiple individual iOS performance benchmarks into a single unified suite, impeller_primitives_perf_suite_ios, and introduces a helper function runBenchmarks to execute them sequentially and merge their results. The CI configuration, test ownership, and metrics parsing logic are updated accordingly, accompanied by new unit tests. Feedback on the changes suggests increasing the CI task timeout to 120 minutes to prevent flaky timeouts from sequential execution, and improving error logging in the benchmark runner by printing failure messages and complete exception details immediately.
8e2a4aa
into
flutter:flutter-3.50-candidate.0
This pull request is created by automatic cherry pick workflow
Please fill in the form below, and a flutter domain expert will evaluate this cherry pick request.
Issue Link:
#195661 (PR, there was no open issue)
Impact Description:
Release builders should match the master branch configs. Without this change, the beta release branch will try to run the old builder names and fail because the infra config is missing.
Changelog Description:
N/A - CI test-only change
Workaround:
N/A
Risk:
What is the risk level of this cherry-pick?
Test Coverage:
Are you confident that your fix is well-tested by automated tests?
Validation Steps:
Release PRs should not complain about missing builders.