Skip to content

Proposal: Remove engine top-level orchestrator builders that only kick off and collect sub-builds (no generators), and hoist sub-build to "standalone" #155041

Open

Description

@jmagman

Some "top-level" orchestrator builds only kick off sub-builds and then do nothing but wait for the results. Mac mac_unopt is an example of this:

Screenshot 2024-09-11 at 12 43 27 PM

Note there are no "Global generators" or tests.

GitHub currently only allows reruns on these top-level builds when there's a failure in the subbuild. In the case of mac_unopt that means all the sub-builds are re-run, even though most of them succeeded (example). Rerunning all of them ties up CI resources and makes the rerun longer, if the failing test isn't the long-pole of the top-level build total time. It also means the top-level orchestrator build idles bot while it waits on the builds to finish.

mac_ios_engine is an example of a build that can't be trivially replaced by it subviews (like ci/ios_debug_sim) because it has generators that operate on the archives:
https://github.com/flutter/engine/blob/38e37ce9879b2efe4890a8d09f4207dc5686a1bd/ci/builders/mac_ios_engine.json#L488C6-L488C16

This proposes removing all top-level builds that only kick off subbuilds, and hoisting the sub-builds into the .ci.yaml.

cc @zanderso @jtmcdole

Activity

  1. added
    engineflutter/engine related. See also e: labels.
    c: proposalA detailed proposal for a change to Flutter
    on Sep 11, 2024
  2. jtmcdole commented on Sep 12, 2024

    @jtmcdole
    Member

    A couple of questions for my own edification:

    1. How many builders can we elevate? Are most of the builders able to do this and it's just the mac_ios_engine that's special?
    2. Are the "waiting" machines actually consuming machines that could be used for other builds and tests?

    I like the idea on the surface.

  3. zanderso commented on Sep 12, 2024

    @zanderso
    Member

    This is how the builds with json files under ci/builders/standalone work. One example is windows_unopt:

    https://github.com/flutter/engine/blob/main/ci/builders/standalone/windows_unopt.json

    https://github.com/flutter/engine/blob/main/.ci.yaml#L555 <- note that the recipe is engine_v2/builder rather than engine_v2/engine_v2.

    https://ci.chromium.org/ui/p/flutter/builders/prod/Windows%20windows_unopt/8783/overview <- example build.

    The engine_v2 recipe runs all the sub-builds and then runs the global tests and generators on the orchestrator.

    The builder recipe just runs the single build definition in the indicated json file.

  4. jmagman commented on Sep 12, 2024

    @jmagman
    MemberAuthor

    This is how the builds with json files under ci/builders/standalone work.

    @zanderso do you see a downside to moving the build-only (no tests, generators) builds to standalone (plus recipe change)? Was that always the plan with standalone? Is this proposal already half-implemented?

    • How many builders can we elevate? Are most of the builders able to do this and it's just the mac_ios_engine that's special?

    @jtmcdole I started making a list at the bottom of the issue starting with "Linux clangd" but I ran out of steam about halfway through, all the json files in https://github.com/flutter/engine/tree/38e37ce9879b2efe4890a8d09f4207dc5686a1bd/ci/builders should be audited as part of this change.

    Are the "waiting" machines actually consuming machines that could be used for other builds and tests?

    @jtmcdole yes, the waiting "orchestrator" builds are consuming a machine. We've tried to make as many of them as possible "cheap" Linux VMs which we have many of, but the mac_unopt example is holding a more precious Mac.

  5. changed the title [-]Proposal: Remove engine top-level builder that only kick off and collect sub-builds (no generators), and hoist sub-build to top[/-] [+]Proposal: Remove engine top-level orchestrator builders that only kick off and collect sub-builds (no generators), and hoist sub-build to top[/+] on Sep 12, 2024
  6. changed the title [-]Proposal: Remove engine top-level orchestrator builders that only kick off and collect sub-builds (no generators), and hoist sub-build to top[/-] [+]Proposal: Remove engine top-level orchestrator builders that only kick off and collect sub-builds (no generators), and hoist sub-build to "standalone"[/+] on Sep 12, 2024
  7. jtmcdole commented on Sep 12, 2024

    @jtmcdole
    Member

    Lets chat about this in a merged repo world

  8. zanderso commented on Sep 13, 2024

    @zanderso
    Member

    This is how the builds with json files under ci/builders/standalone work.

    @zanderso do you see a downside to moving the build-only (no tests, generators) builds to standalone (plus recipe change)? Was that always the plan with standalone? Is this proposal already half-implemented?

    Yes, this is already half-implemented. I don't think any recipe change is needed for this, but I might be missing something. The only possible downside I could imagine is if someone wants to expand a file like windows_unopt.json with some additional builds, it will have to jump back out of standalone, and that would be some additional churn. But afaict that's how the engine_v2 recipes are designed to work, so it's not really the end of the world.

  9. added
    P1High-priority issues at the top of the work list
    on Sep 23, 2024
  10. jmagman commented on Oct 22, 2024

    @jmagman
    MemberAuthor

    @reidbaker if linux_android_emulator_34 and linux_android_emulator were moved to standalone then I think you could deduplicate them into one json file per build (android_emulator_debug_x64 and android_emulator_debug_x86), and set the android_virtual_device and avd_cipd_version dependencies from the .ci.yaml.

        properties:
          config_name: linux_android_emulator
          dependencies: >-
            [
              {"dependency": "android_virtual_device", "version": "android_28_google_apis_x86.textpb"}
              {"dependency": "avd_cipd_version", "version": "build_id:8740267484269553649"}
            ]
    

    The android_emulator_debug_x86 one would only need to swap avd_cipd_version per build and not android_virtual_device since both are targeting SDK 28 Android Unit Tests (API 28) (if I'm reading flutter-team-archive/engine#54186 right).

    You would then only need to create a new builder in the .ci.yaml for the next SDK, and not copy the entire json config.

  11. self-assigned this
    on Jan 9, 2025
  12. removed their assignment
    on Jan 29, 2025
  13. flutter-triage-bot commented on Jun 4, 2025

    @flutter-triage-bot

    This issue is marked P1 but has had no recent status updates.

    The P1 label indicates high-priority issues that are at the top of the work list. This is the highest priority level a bug can have if it isn't affecting a top-tier customer or breaking the build. Bugs marked P1 are generally actively being worked on unless the assignee is dealing with a P0 bug (or another P1 bug). Issues at this level should be resolved in a matter of months and should have monthly updates on GitHub.

    Please consider where this bug really falls in our current priorities, and label it or assign it accordingly. This allows people to have a clearer picture of what work is actually planned. Thanks!

  14. flutter-triage-bot commented on Oct 12, 2025

    @flutter-triage-bot

    This issue is marked P1 but has had no recent status updates.

    The P1 label indicates high-priority issues that are at the top of the work list. This is the highest priority level a bug can have if it isn't affecting a top-tier customer or breaking the build. Bugs marked P1 are generally actively being worked on unless the assignee is dealing with a P0 bug (or another P1 bug). Issues at this level should be resolved in a matter of months and should have monthly updates on GitHub.

    Please consider where this bug really falls in our current priorities, and label it or assign it accordingly. This allows people to have a clearer picture of what work is actually planned. Thanks!

  15. added
    P2Important issues not at the top of the work list
    and removed
    P1High-priority issues at the top of the work list
    on Oct 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Important issues not at the top of the work listc: proposalA detailed proposal for a change to Flutterengineflutter/engine related. See also e: labels.team-engineOwned by Engine teamtriaged-engineTriaged by Engine team

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions