Skip to content

[CP-beta]Run 10 impeller iOS benchmarks as one devicelab task - #196009

Merged
auto-submit[bot] merged 1 commit into
flutter:flutter-3.50-candidate.0from
flutteractionsbot:cp-beta-7a62b1f707a1b54918efe2e4f2a2a159b538a4cd
Nov 5, 2026
Merged

auto-submit[bot] merged 1 commit into
flutter:flutter-3.50-candidate.0from
flutteractionsbot:cp-beta-7a62b1f707a1b54918efe2e4f2a2a159b538a4cd

Conversation

@flutteractionsbot

@flutteractionsbot flutteractionsbot commented Nov 4, 2026 •

Copy link
Copy Markdown
Contributor

This pull request is created by automatic cherry pick workflow
Please fill in the form below, and a flutter domain expert will evaluate this cherry pick request.

Issue Link:

#195661 (PR, there was no open issue)

Impact Description:

Release builders should match the master branch configs. Without this change, the beta release branch will try to run the old builder names and fail because the infra config is missing.

Changelog Description:

N/A - CI test-only change

Workaround:

N/A

Risk:

What is the risk level of this cherry-pick?

  • Low
  • Medium
  • High

Test Coverage:

Are you confident that your fix is well-tested by automated tests?

  • Yes
  • No

Validation Steps:

Release PRs should not complain about missing builders.

draw_points, draw_vertices, draw_atlas, draw_arcs_all_fill_styles,
draw_arcs_all_stroke_styles, rrect_blur, and
*_perf_ios__timeline_summary tasks all build the same macrobenchmarks
app with the same target and each draw for about 10 seconds. Each spends
1.4–2.6 minutes benchmarking inside a 5–10 minute bot run. This runs
them in one task,
`impeller_primitives_perf_suite_ios__timeline_summary`.

Each benchmark still gets its own `flutter drive`, so the app is
reinstalled and relaunched every time and nothing in-process carries
over. Runs after the first reuse the already-built .app so the app is
only built once.

SkiaPerf traces are unchanged. TaskResult.merge prefixes each score key
with the original task name and `metrics_center.parse` uploads it under
that name. Tasks that don't use merge are unaffected.

Per commit **this runs a 16 minutes benchmark on one Mac and iPhone
instead of 76 minutes across 10 benchmarks**, saving ~100 device-hours a
week. The suite becomes the longest Mac_ios task (~16 min), but these
are postsubmit. Failure in any benchmark fails the task and drops that
commit's metrics for all 10 benchmarks. The task runs every benchmark
and reports all failures, not just the first.

Delete the old builders (which also means release branch cp). Ideally we
could run them next to each other for awhile, but unfortunately the way
SkiaPerf works the metrics are the same, so they would stomp on each
other and we wouldn't know which builder the data is coming from (they
even upload in `bringup`). Let's see if any of the metrics move on this
commit, if they do we can go back.

[Test run
build](https://ci.chromium.org/ui/p/flutter/builders/try/Mac_ios%20draw_arcs_all_fill_styles_perf_ios__timeline_summary/1/overview)
- metrics at end of
[stdout](https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8666540421791185473/+/u/run_impeller_primitives_perf_suite_ios__timeline_summary/stdout)
- 14m total, 9.5m on the actual benchmark run step.

Part of the work to reduce the number of Mac bots in CI. 

flutter#195654

## Pre-launch Checklist

- [x] I read the [Contributor Guide] and followed the process outlined
there for submitting PRs.
- [x] I read the [AI contribution guidelines] and understand my
responsibilities, or I am not using AI tools.
- [x] I read the [Tree Hygiene] wiki page, which explains my
responsibilities.
- [x] I read and followed the [Flutter Style Guide], including [Features
we expect every widget to implement].
- [x] I signed the [CLA].
- [x] I listed at least one issue that this PR fixes in the description
above.
- [x] I updated/added relevant in-code documentation (doc comments with
`///`).
- [x] If this PR introduces a new feature or capability, I created and
linked a website documentation issue or PR in [flutter/website] (or
verified none is needed).
- [x] I added new tests to check the change I am making, or this PR is
[test-exempt].
- [x] I followed the [breaking change policy] and added [Data Driven
Fixes] where supported.
- [x] All existing and new tests are passing.

If you need help, consider asking for advice on the #hackers-new channel
on [Discord].

If this change needs to override an active code freeze, provide a
comment explaining why. The code freeze workflow can be overridden by
code reviewers. See pinned issues for any active code freezes with
guidance.

**Note**: The Flutter team is currently trialing the use of [Gemini Code
Assist for
GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code).
Comments from the `gemini-code-assist` bot should not be taken as
authoritative feedback from the Flutter team. If you find its comments
useful you can update your code accordingly, but if you are unsure or
disagree with the feedback, please feel free to wait for a Flutter team
member's review for guidance on which automated comments should be
addressed.

<!-- Links -->
[Contributor Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview
[AI contribution guidelines]:
https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md
[Tree Hygiene]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md
[test-exempt]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests
[Flutter Style Guide]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md
[Features we expect every widget to implement]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement
[CLA]: https://cla.developers.google.com/
[flutter/tests]: https://github.com/flutter/tests
[flutter/website]: https://github.com/flutter/website
[breaking change policy]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes
[Discord]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md
[Data Driven Fixes]:
https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
@flutteractionsbot flutteractionsbot added the cp: review Cherry-picks in the review queue label Nov 4, 2026
@flutteractionsbot

Copy link
Copy Markdown
Contributor Author

@jmagman please fill out the PR description above, afterwards the release team will review this request.

@flutter-dashboard flutter-dashboard Bot added the CICD Run CI/CD label Nov 4, 2026
@flutter-dashboard

Copy link
Copy Markdown

This pull request was opened from and to a release candidate branch. This should only be done as part of the official Flutter release process. If you are attempting to make a regular contribution to the Flutter project, please close this PR and follow the instructions at Tree Hygiene for detailed instructions on contributing to Flutter.

Reviewers: Use caution before merging pull requests to release branches. Ensure the proper procedure has been followed.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request consolidates multiple individual iOS performance benchmarks into a single unified suite, impeller_primitives_perf_suite_ios, and introduces a helper function runBenchmarks to execute them sequentially and merge their results. The CI configuration, test ownership, and metrics parsing logic are updated accordingly, accompanied by new unit tests. Feedback on the changes suggests increasing the CI task timeout to 120 minutes to prevent flaky timeouts from sequential execution, and improving error logging in the benchmark runner by printing failure messages and complete exception details immediately.

Comment thread .ci.yaml Resolved
@reidbaker reidbaker added the autosubmit Merge PR when tree becomes green via auto submit App label Nov 5, 2026
@auto-submit
auto-submit Bot merged commit 8e2a4aa into flutter:flutter-3.50-candidate.0 Nov 5, 2026
11 checks passed
@flutteractionsbot
flutteractionsbot deleted the cp-beta-7a62b1f707a1b54918efe2e4f2a2a159b538a4cd branch November 5, 2026 17:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

autosubmit Merge PR when tree becomes green via auto submit App CICD Run CI/CD cp: review Cherry-picks in the review queue

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants