Repository navigation
Run 10 impeller iOS benchmarks as one devicelab task - #195661
auto-submit[bot] merged 3 commits into
Conversation
There was a problem hiding this comment.
Code Review
This pull request consolidates multiple individual iOS performance benchmarks into a single suite, impeller_primitives_perf_suite_ios__timeline_summary, and introduces a runBenchmarks helper to execute and merge results from multiple benchmarks. Feedback suggests wrapping benchmark execution in a try-catch block to prevent unhandled exceptions in one benchmark from stopping the entire suite, and adding a corresponding unit test to verify this behavior.
gaaclarke
left a comment
There was a problem hiding this comment.
This moves the benchmarks to a new track. Can we not spoof the test name so that they are recorded to the same track they were previously on? I guess maybe that would be confusing though. What was your thinking about that?
I'm not sure what "track" means here, but
Unless there's a bug here or I'm misunderstanding, the "spoofing" and breaking the name back apart is what the new code is doing: |
Track means a continuous flow of performance benchmarks. Will this constitute a breakage in the measurement? From reading the code I thought |
I designed it so there wouldn't be breakage in measurement (again unless I have a bug in there). |
Oh also the builder name isn't used anywhere, it looks like this on collection: And that's what's split back apart on metrics_center upload (which is what is being tested from the test in my last comment) This log was run on this PR: |
|
okay, thanks. I'll give it another look |
gaaclarke
left a comment
There was a problem hiding this comment.
okay, looks good to me. results are muxed in task_result.dart, but demuxed before reported to serve in metrics_center.dart.
draw_points, draw_vertices, draw_atlas, draw_arcs_all_fill_styles, draw_arcs_all_stroke_styles, rrect_blur, and *_perf_ios__timeline_summary tasks all build the same macrobenchmarks app with the same target and each draw for about 10 seconds. Each spends 1.4–2.6 minutes benchmarking inside a 5–10 minute bot run. This runs them in one task, `impeller_primitives_perf_suite_ios__timeline_summary`. Each benchmark still gets its own `flutter drive`, so the app is reinstalled and relaunched every time and nothing in-process carries over. Runs after the first reuse the already-built .app so the app is only built once. SkiaPerf traces are unchanged. TaskResult.merge prefixes each score key with the original task name and `metrics_center.parse` uploads it under that name. Tasks that don't use merge are unaffected. Per commit **this runs a 16 minutes benchmark on one Mac and iPhone instead of 76 minutes across 10 benchmarks**, saving ~100 device-hours a week. The suite becomes the longest Mac_ios task (~16 min), but these are postsubmit. Failure in any benchmark fails the task and drops that commit's metrics for all 10 benchmarks. The task runs every benchmark and reports all failures, not just the first. Delete the old builders (which also means release branch cp). Ideally we could run them next to each other for awhile, but unfortunately the way SkiaPerf works the metrics are the same, so they would stomp on each other and we wouldn't know which builder the data is coming from (they even upload in `bringup`). Let's see if any of the metrics move on this commit, if they do we can go back. [Test run build](https://ci.chromium.org/ui/p/flutter/builders/try/Mac_ios%20draw_arcs_all_fill_styles_perf_ios__timeline_summary/1/overview) - metrics at end of [stdout](https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8666540421791185473/+/u/run_impeller_primitives_perf_suite_ios__timeline_summary/stdout) - 14m total, 9.5m on the actual benchmark run step. Part of the work to reduce the number of Mac bots in CI. flutter#195654 ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [x] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
Roll Flutter from fcfb4f203eae to 934720261be0 (38 revisions) flutter/flutter@fcfb4f2...9347202 2026-11-03 [email protected] Roll Skia from 5bceaa033999 to c733ad3e37cb (5 revisions) (flutter/flutter#195872) 2026-11-03 [email protected] Roll Skia from 34d2bf479385 to 5bceaa033999 (9 revisions) (flutter/flutter#195864) 2026-11-03 [email protected] fix: prefer pointer events over semantics tap for mouse clicks on web (flutter/flutter#191529) 2026-11-03 [email protected] [VPAT] Expose a semantics text field value while it is not being edited (flutter/flutter#195445) 2026-11-03 [email protected] [Impeller] Make a GL context current before constructing an AiksContext without a surface (flutter/flutter#195629) 2026-11-03 [email protected] [flutter_tools] Eliminate globals.dart from project.dart (flutter/flutter#194066) 2026-11-03 [email protected] [tool] Complete context runner elimination and migrate AppContext usages to ToolContext (flutter/flutter#190797) 2026-11-03 [email protected] [web] Unify font collections between CanvasKit and Skwasm (flutter/flutter#195539) 2026-11-03 [email protected] [Android] Replay predictive back state for retained engines (flutter/flutter#192286) 2026-11-03 [email protected] [vulkan]: Adds depth/stencil synchronization for subpasses (flutter/flutter#195689) 2026-11-03 [email protected] [web] Honor requested upscaling in instantiateImageCodecWithSize and ImageDescriptor.instantiateCodec (flutter/flutter#195836) 2026-11-03 [email protected] Roll Skia from 866b61ca92c9 to 34d2bf479385 (9 revisions) (flutter/flutter#195848) 2026-11-03 [email protected] Roll Fuchsia Test Scripts from xl1LWBuppzu-BnOzV... to B6zarykZp6PKWRueg... (flutter/flutter#195847) 2026-11-03 [email protected] Remove unused ffi (flutter/flutter#195710) 2026-11-02 [email protected] Revert: [Android] Let the raster thread use every core of a chip without little cores (flutter/flutter#195838) 2026-11-02 [email protected] Run 10 impeller iOS benchmarks as one devicelab task (flutter/flutter#195661) 2026-11-02 [email protected] Use named uniforms instead of setFloat in StretchEffect (flutter/flutter#195372) 2026-11-02 [email protected] [flutter_tools] Report when the browser disconnects during a web test run (flutter/flutter#192198) 2026-11-02 [email protected] fix: Resolve memory leak in image_list example (flutter/flutter#189526) 2026-11-02 [email protected] Only run the common local engine configurations in presubmit (flutter/flutter#195688) 2026-11-02 [email protected] Update gcloud_version to version [email protected] (flutter/flutter#195769) 2026-11-02 [email protected] [Impeller] Order the Vulkan copy-texture-to-texture layout transition after the copy (flutter/flutter#195814) 2026-11-02 [email protected] [Impeller] Order the Vulkan ResizeTexture layout transition after its blit (flutter/flutter#195721) 2026-11-02 [email protected] [web] Fix CanvasKit fragment shader uniforms being freed or mutated after recording (flutter/flutter#195718) 2026-11-02 [email protected] Roll Fuchsia Linux SDK from MqypI6RNyVZ_FLUE-... to T_-6-6X1lDMghhAvX... (flutter/flutter#195809) 2026-11-02 [email protected] [Android] Log why Impeller falls back to OpenGLES (flutter/flutter#194764) 2026-11-02 [email protected] Roll Skia from 3f35fb6e591b to 866b61ca92c9 (11 revisions) (flutter/flutter#195802) 2026-11-02 [email protected] Roll Skia from 3f35fb6e591b to 866b61ca92c9 (11 revisions) (flutter/flutter#195802) 2026-11-02 [email protected] Only run the common local engine configurations in presubmit (flutter/flutter#195688) 2026-11-02 [email protected] Balance tool_integration_tests subshards by expected duration (flutter/flutter#195628) 2026-11-02 [email protected] [web_paragraph] Map code units without a cluster to a cluster (flutter/flutter#195384) 2026-11-02 [email protected] [web] Hide synthetic autofill inputs from assistive technology (flutter/flutter#195618) 2026-11-02 [email protected] Roll Packages from 3742a4e to 3f34a6f (8 revisions) (flutter/flutter#195795) 2026-11-02 [email protected] [web] Unskip web tests that pass on CanvasKit and Skwasm: platform_view_test, route_notification_messages_test (flutter/flutter#195605) 2026-11-02 [email protected] Roll Skia from e5ad3868b8db to 3f35fb6e591b (1 revision) (flutter/flutter#195789) 2026-11-02 [email protected] Roll Skia from f5215e74aa16 to e5ad3868b8db (3 revisions) (flutter/flutter#195779) 2026-11-02 [email protected] Roll Skia from e43478a2f9f8 to f5215e74aa16 (1 revision) (flutter/flutter#195776) 2026-11-01 [email protected] Roll Fuchsia Linux SDK from kW0A0xxNkqf38ljm4... to MqypI6RNyVZ_FLUE-... (flutter/flutter#195768) If this roll has caused a breakage, revert this CL and set the roller to dry run mode using the controls here: https://autoroll.skia.org/r/flutter-packages Please CC [email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose ...
|
Failed to create CP due to merge conflicts. |
…gup (flutter#196007) Passing in postsubmit flutter#195661, take out of bringup https://ci.chromium.org/ui/p/flutter/builders/prod/Mac_ios%20impeller_primitives_perf_suite_ios__timeline_summary/50/overview ## Pre-launch Checklist - [x] I read the [Contributor Guide] and followed the process outlined there for submitting PRs. - [x] I read the [AI contribution guidelines] and understand my responsibilities, or I am not using AI tools. - [x] I read the [Tree Hygiene] wiki page, which explains my responsibilities. - [x] I read and followed the [Flutter Style Guide], including [Features we expect every widget to implement]. - [x] I signed the [CLA]. - [ ] I listed at least one issue that this PR fixes in the description above. - [x] I updated/added relevant in-code documentation (doc comments with `///`). - [x] If this PR introduces a new feature or capability, I created and linked a website documentation issue or PR in [flutter/website] (or verified none is needed). - [x] I added new tests to check the change I am making, or this PR is [test-exempt]. - [x] I followed the [breaking change policy] and added [Data Driven Fixes] where supported. - [x] All existing and new tests are passing. If you need help, consider asking for advice on the #hackers-new channel on [Discord]. If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance. **Note**: The Flutter team is currently trialing the use of [Gemini Code Assist for GitHub](https://developers.google.com/gemini-code-assist/docs/review-github-code). Comments from the `gemini-code-assist` bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed. <!-- Links --> [Contributor Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#overview [AI contribution guidelines]: https://github.com/flutter/flutter/blob/main/CONTRIBUTING.ai.md [Tree Hygiene]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md [test-exempt]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#tests [Flutter Style Guide]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md [Features we expect every widget to implement]: https://github.com/flutter/flutter/blob/main/docs/contributing/Style-guide-for-Flutter-repo.md#features-we-expect-every-widget-to-implement [CLA]: https://cla.developers.google.com/ [flutter/tests]: https://github.com/flutter/tests [flutter/website]: https://github.com/flutter/website [breaking change policy]: https://github.com/flutter/flutter/blob/main/docs/contributing/Tree-hygiene.md#handling-breaking-changes [Discord]: https://github.com/flutter/flutter/blob/main/docs/contributing/Chat.md [Data Driven Fixes]: https://github.com/flutter/flutter/blob/main/docs/contributing/Data-driven-Fixes.md
This pull request is created by [automatic cherry pick workflow](https://github.com/flutter/flutter/blob/main/docs/releases/Flutter-Cherrypick-Process.md#automatically-creates-a-cherry-pick-request) Please fill in the form below, and a flutter domain expert will evaluate this cherry pick request. ### Issue Link: #195661 (PR, there was no open issue) ### Impact Description: Release builders should match the master branch configs. Without this change, the beta release branch will try to run the old builder names and fail because the infra config is missing. ### Changelog Description: N/A - CI test-only change ### Workaround: N/A ### Risk: What is the risk level of this cherry-pick? ### Test Coverage: Are you confident that your fix is well-tested by automated tests? ### Validation Steps: Release PRs should not complain about missing builders.
draw_points, draw_vertices, draw_atlas, draw_arcs_all_fill_styles, draw_arcs_all_stroke_styles, rrect_blur, and *_perf_ios__timeline_summary tasks all build the same macrobenchmarks app with the same target and each draw for about 10 seconds. Each spends 1.4–2.6 minutes benchmarking inside a 5–10 minute bot run. This runs them in one task,
impeller_primitives_perf_suite_ios__timeline_summary.Each benchmark still gets its own
flutter drive, so the app is reinstalled and relaunched every time and nothing in-process carries over. Runs after the first reuse the already-built .app so the app is only built once.SkiaPerf traces are unchanged. TaskResult.merge prefixes each score key with the original task name and
metrics_center.parseuploads it under that name. Tasks that don't use merge are unaffected.Per commit this runs a 16 minutes benchmark on one Mac and iPhone instead of 76 minutes across 10 benchmarks, saving ~100 device-hours a week. The suite becomes the longest Mac_ios task (~16 min), but these are postsubmit. Failure in any benchmark fails the task and drops that commit's metrics for all 10 benchmarks. The task runs every benchmark and reports all failures, not just the first.
Delete the old builders (which also means release branch cp). Ideally we could run them next to each other for awhile, but unfortunately the way SkiaPerf works the metrics are the same, so they would stomp on each other and we wouldn't know which builder the data is coming from (they even upload in
bringup). Let's see if any of the metrics move on this commit, if they do we can go back.Test run build - metrics at end of stdout - 14m total, 9.5m on the actual benchmark run step.
Part of the work to reduce the number of Mac bots in CI.
#195654
Pre-launch Checklist
///).If you need help, consider asking for advice on the #hackers-new channel on Discord.
If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance.
Note: The Flutter team is currently trialing the use of Gemini Code Assist for GitHub. Comments from the
gemini-code-assistbot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.