Split mac_host_{debug,profile,release}_arm64_tests into one step per type by jmagman · Pull Request #196031 · flutter/flutter

jmagman · 2026-11-05T08:27:51Z

Mac mac_host_{debug,profile,release}_arm64_tests ran all its tests in one 25–27 minute run_tests.py step, so any flake reruns the whole thing. Split it by --type so a retry now repeats only the step that failed.

Funny enough I hit a flake in this very PR and you can see it re-ran a 30 second test suite but didn't run the 4+ minute Engine unit tests for host_profile or the 3 minute Dart host tests for host_profile:

https://ci.chromium.org/ui/p/flutter/builders/try/Mac%20mac_host_profile_arm64_tests/2/overview

Part of improving #185908

Pre-launch Checklist

If you need help, consider asking for advice on the #hackers-new channel on Discord.

If this change needs to override an active code freeze, provide a comment explaining why. The code freeze workflow can be overridden by code reviewers. See pinned issues for any active code freezes with guidance.

Note: The Flutter team is currently trialing the use of Gemini Code Assist for GitHub. Comments from the gemini-code-assist bot should not be taken as authoritative feedback from the Flutter team. If you find its comments useful you can update your code accordingly, but if you are unsure or disagree with the feedback, please feel free to wait for a Flutter team member's review for guidance on which automated comments should be addressed.

…type The single run_tests.py step took 25-27 minutes and reran in full whenever one test flaked. Run engine, dart, dart-host, and impeller-golden as separate steps so a retry only repeats the failing one.

jmagman · 2026-11-05T19:52:56Z

/gemini review

gemini-code-assist

Code Review

This pull request splits combined host test suites into individual test steps for each test type across several macOS host builder configuration files. The feedback highlights that the --engine-capture-core-dump parameter was omitted from the newly created non-engine test steps (such as dart, dart-host, font-subset, and impeller-golden) and recommends retaining this parameter across all split steps to ensure core dumps are captured during crashes.

gemini-code-assist

Code Review

This pull request splits combined host test definitions into separate test configurations for engine, dart, dart-host, font-subset, and impeller-golden types across several macOS ARM64 builder configuration files. Feedback indicates that splitting these tests removed the --engine-capture-core-dump parameter from non-engine test types (such as Dart, Dart host, Font subset, and Impeller golden tests), which prevents capturing core dumps if those executables crash during CI runs.

gemini-code-assist · 2026-11-05T20:55:20Z

+        {
+            "language": "python3",
+            "name": "Dart tests for host_debug",
+            "script": "flutter/testing/run_tests.py",
+            "parameters": [
+                "--variant",
+                "ci/host_debug_arm64_tests",
+                "--type",
+                "dart"
+            ]
+        },
+        {
+            "language": "python3",
+            "name": "Dart host tests for host_debug",
+            "script": "flutter/testing/run_tests.py",
+            "parameters": [
+                "--variant",
+                "ci/host_debug_arm64_tests",
+                "--type",
+                "dart-host"
+            ]
+        },
+        {
+            "language": "python3",
+            "name": "Font subset tests for host_debug",
+            "script": "flutter/testing/run_tests.py",
+            "parameters": [
+                "--variant",
+                "ci/host_debug_arm64_tests",
+                "--type",
+                "font-subset"
+            ]
        }


By splitting the tests, the --engine-capture-core-dump parameter was only kept for the engine unit tests. However, Dart tests, Dart host tests, and Font subset tests also run executables (like the Dart VM or font-subset) that can crash. Retaining --engine-capture-core-dump for all test types ensures we still capture core dumps if any of these tests crash, which is critical for debugging CI failures.

{ "language": "python3", "name": "Dart tests for host_debug", "script": "flutter/testing/run_tests.py", "parameters": [ "--variant", "ci/host_debug_arm64_tests", "--type", "dart", "--engine-capture-core-dump" ] }, { "language": "python3", "name": "Dart host tests for host_debug", "script": "flutter/testing/run_tests.py", "parameters": [ "--variant", "ci/host_debug_arm64_tests", "--type", "dart-host", "--engine-capture-core-dump" ] }, { "language": "python3", "name": "Font subset tests for host_debug", "script": "flutter/testing/run_tests.py", "parameters": [ "--variant", "ci/host_debug_arm64_tests", "--type", "font-subset", "--engine-capture-core-dump" ] }

Yea, i think it's worth keeping this the same.

The bot keeps repeating comments, I had already resolved this one above:
--engine-capture-core-dump is only consumed by --type engine tests. The other types don't read it at all.

Only read here:

flutter/engine/src/flutter/testing/run_tests.py

Lines 1497 to 1498 in 9d41d2b

if 'engine' in types:

run_cc_tests(build_dir, engine_filter, args.coverage, args.engine_capture_core_dump)

gemini-code-assist · 2026-11-05T20:55:20Z

+        {
+            "language": "python3",
+            "name": "Dart tests for host_profile",
+            "script": "flutter/testing/run_tests.py",
+            "parameters": [
+                "--variant",
+                "ci/host_profile_arm64_tests",
+                "--type",
+                "dart"
+            ]
+        },
+        {
+            "language": "python3",
+            "name": "Dart host tests for host_profile",
+            "script": "flutter/testing/run_tests.py",
+            "parameters": [
+                "--variant",
+                "ci/host_profile_arm64_tests",
+                "--type",
+                "dart-host"
+            ]
        }


By splitting the tests, the --engine-capture-core-dump parameter was only kept for the engine unit tests. However, Dart tests and Dart host tests also run executables (like the Dart VM) that can crash. Retaining --engine-capture-core-dump for all test types ensures we still capture core dumps if any of these tests crash, which is critical for debugging CI failures.

{ "language": "python3", "name": "Dart tests for host_profile", "script": "flutter/testing/run_tests.py", "parameters": [ "--variant", "ci/host_profile_arm64_tests", "--type", "dart", "--engine-capture-core-dump" ] }, { "language": "python3", "name": "Dart host tests for host_profile", "script": "flutter/testing/run_tests.py", "parameters": [ "--variant", "ci/host_profile_arm64_tests", "--type", "dart-host", "--engine-capture-core-dump" ] }

--engine-capture-core-dump is only consumed by --type engine tests. The other types don't read it at all.

gaaclarke

concept lgtm, just one request to keep the "capture core dump". I'm not sure I've ever used that, sounds useful and worth keeping old behavior.

gaaclarke · 2026-11-05T20:57:38Z

+        {
+            "language": "python3",
+            "name": "Dart tests for host_debug",
+            "script": "flutter/testing/run_tests.py",
+            "parameters": [
+                "--variant",
+                "ci/host_debug_arm64_tests",
+                "--type",
+                "dart"
+            ]
+        },
+        {
+            "language": "python3",
+            "name": "Dart host tests for host_debug",
+            "script": "flutter/testing/run_tests.py",
+            "parameters": [
+                "--variant",
+                "ci/host_debug_arm64_tests",
+                "--type",
+                "dart-host"
+            ]
+        },
+        {
+            "language": "python3",
+            "name": "Font subset tests for host_debug",
+            "script": "flutter/testing/run_tests.py",
+            "parameters": [
+                "--variant",
+                "ci/host_debug_arm64_tests",
+                "--type",
+                "font-subset"
+            ]
        }


Yea, i think it's worth keeping this the same.

jmagman · 2026-11-05T22:13:48Z

concept lgtm, just one request to keep the "capture core dump". I'm not sure I've ever used that, sounds useful and worth keeping old behavior.

--engine-capture-core-dump is only consumed by --type engine tests. The other types don't read it at all.

flutter/engine/src/flutter/testing/run_tests.py

Lines 1497 to 1498 in 9d41d2b

    
           if 'engine' in types: 
        
             run_cc_tests(build_dir, engine_filter, args.coverage, args.engine_capture_core_dump)

Actually even on --type engine it's a no-op on macOS it's only a thing on Linux, but that's kind of a detail, it should stay for --type engine but there's no reason to keep it for the others.

gaaclarke

--engine-capture-core-dump is only consumed by --type engine tests. The other types don't read it at all.

Hmm, that begs the question if the others should be using it as well. Seems like it. Oh well, we can tweak it if we need to in the future. This looks good because it keeps the same behavior.

The 29 builder files that upstream's "Goma is dead, long live RBE! (Take 2)" (flutter#196212) rewrote conflicted on their gn arrays with the `--bazel`/`--no-rbe` lines: resolved as ours without `--no-goma`, which tools/gn no longer accepts (the Bazel-only builds in linux_unopt.json and mac_clang_tidy.json the same). The split of the Mac standalone test builders into one step per test type (flutter#196031) merged by itself. Regenerated after the Dart SDK roll (flutter#196217), the pub and packages rolls: deps.lock.bzl, pub.lock.bzl (`dart pub get` at the root and in lib/web_ui first), and the converter's builds.json, ci/builders/ BUILD.bazel and configs.bazelrc (127 builds). Checked: gn_to_bazel --check, check_build_configs.sh, bazel_ci_test.py (23), engine_build_configs (84), bazel_tools (46), buildifier; Bazel analysis of ci_host_debug and of //:tool_tests //:dart_tests.

flutter/flutter@d463821...fffd8c2 2026-11-10 [email protected] Roll Fuchsia Test Scripts from 8QcsP_SmbsT09VgUs... to PXc5D7S_q6sIYpfFJ... (flutter/flutter#196273) 2026-11-10 [email protected] [web] Accept the capped letter spacing line shift in kitchensink (flutter/flutter#196148) 2026-11-10 [email protected] Fix Vulkan MSAA diagonal seam in FramebufferBlend (flutter/flutter#196197) 2026-11-10 [email protected] Read back images on the raster thread on OpenGLES (flutter/flutter#195927) 2026-11-10 [email protected] [web] Unify ui.Paint across CanvasKit and Skwasm (flutter/flutter#196071) 2026-11-10 [email protected] Roll Fuchsia Test Scripts from p3N3B-Kchh1bthB6n... to 8QcsP_SmbsT09VgUs... (flutter/flutter#196249) 2026-11-09 [email protected] [Impeller] Reland clipped BackdropFilter square corners fix (flutter/flutter#195716) 2026-11-09 [email protected] [web] Recreate canvas and context on CkOffscreenSurface resize to fix WebGL clipping (flutter/flutter#192769) 2026-11-09 [email protected] Roll Dart SDK from ee0f66fa0ad2 to a2baae273bb2 (1 revision) (flutter/flutter#196245) 2026-11-09 [email protected] Add Mac web_tool_tests_1_2 and _2_2 in bringup (flutter/flutter#196241) 2026-11-09 [email protected] Roll pub packages (flutter/flutter#196246) 2026-11-09 [email protected] Increase number of 'Windows tool_integration_tests' subshards (flutter/flutter#196243) 2026-11-09 [email protected] Roll Fuchsia Test Scripts from Jo-ZgQSNDuPsjlZ4s... to p3N3B-Kchh1bthB6n... (flutter/flutter#196238) 2026-11-09 [email protected] Skip UIScene pre-migration test cases on Xcode 27 (flutter/flutter#194755) 2026-11-09 [email protected] [flutter_tools] Share Analytics.isBot and make isRunningOnBot sync (flutter/flutter#194744) 2026-11-09 [email protected] [flutter_tools] Retry chromedriver startup when its port is taken (flutter/flutter#196150) 2026-11-09 [email protected] Remove unused vulkan_provider.cc (flutter/flutter#195513) 2026-11-09 [email protected] Roll Dart SDK from 3b174b7217fe to ee0f66fa0ad2 (13 revisions) (flutter/flutter#196217) 2026-11-09 [email protected] Goma is dead, long live RBE! (Take 2) (flutter/flutter#196212) 2026-11-09 [email protected] Split DevTools customer tests into its own customer_testing_devtools shard (flutter/flutter#195871) 2026-11-09 [email protected] Split mac_host_{debug,profile,release}_arm64_tests into one step per type (flutter/flutter#196031) If this roll has caused a breakage, revert this CL and set the roller to dry run mode using the controls here: https://autoroll.skia.org/r/flutter-packages Please CC [email protected],[email protected] on the revert to ensure that a human is aware of the problem. To file a bug in Packages: https://github.com/flutter/flutter/issues/new/choose To report a problem with the AutoRoller itself, please file a bug: https://issues.skia.org/issues/new?component=1389291&template=1850622 Documentation for the AutoRoller is here: https://skia.googlesource.com/buildbot/+doc/main/autoroll/README.md

jmagman added the CICD Run CI/CD label Nov 5, 2026

github-actions Bot added engine flutter/engine related. See also e: labels. team-infra Owned by Infrastructure team d: docs/ flutter/flutter/docs, for contributors labels Nov 5, 2026

Split mac_host_{debug,profile,release}_arm64_tests into one step per …

315d0c7

…type The single run_tests.py step took 25-27 minutes and reran in full whenever one test flaked. Run engine, dart, dart-host, and impeller-golden as separate steps so a retry only repeats the failing one.

jmagman force-pushed the split-mac-host-debug-unopt-tests branch from 2d66df1 to 315d0c7 Compare November 5, 2026 19:51

gemini-code-assist Bot reviewed Nov 5, 2026

View reviewed changes

Comment thread engine/src/flutter/ci/builders/standalone/mac_host_debug_arm64_tests.json Resolved

Comment thread engine/src/flutter/ci/builders/standalone/mac_host_debug_unopt_arm64.json Resolved

Comment thread engine/src/flutter/ci/builders/standalone/mac_host_profile_arm64_tests.json Resolved

github-actions Bot removed team-infra Owned by Infrastructure team d: docs/ flutter/flutter/docs, for contributors labels Nov 5, 2026

jmagman marked this pull request as ready for review November 5, 2026 20:54

jmagman requested a review from gaaclarke November 5, 2026 20:54

gemini-code-assist Bot reviewed Nov 5, 2026

View reviewed changes

gaaclarke requested changes Nov 5, 2026

View reviewed changes

jmagman requested a review from gaaclarke November 5, 2026 22:36

gaaclarke approved these changes Nov 5, 2026

View reviewed changes

jmagman added the autosubmit Merge PR when tree becomes green via auto submit App label Nov 9, 2026

auto-submit Bot added this pull request to the merge queue Nov 9, 2026

Merged via the queue into flutter:master with commit 3cc7361 Nov 9, 2026
23 checks passed

flutter-dashboard Bot removed the autosubmit Merge PR when tree becomes green via auto submit App label Nov 9, 2026

engine-flutter-autoroll mentioned this pull request Nov 10, 2026

Roll Flutter from d46382112aa6 to fffd8c22ae64 (21 revisions) flutter/packages#13614

Merged

jmagman deleted the split-mac-host-debug-unopt-tests branch November 11, 2026 00:03

	if 'engine' in types:
	run_cc_tests(build_dir, engine_filter, args.coverage, args.engine_capture_core_dump)

Conversation

jmagman commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Pre-launch Checklist

Uh oh!

jmagman commented Nov 5, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

gemini-code-assist Bot Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

gaaclarke Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

jmagman Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

jmagman Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

gemini-code-assist Bot Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

jmagman Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

gaaclarke left a comment

Choose a reason for hiding this comment

Uh oh!

gaaclarke Nov 5, 2026

Choose a reason for hiding this comment

Uh oh!

jmagman commented Nov 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gaaclarke left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

jmagman commented Nov 5, 2026 •

edited

Loading

jmagman commented Nov 5, 2026 •

edited

Loading