Skip to content

Chromium Mac Bots become unresponsive and Xcode commands hang #157636

Closed
0 of 1 issue completed
Closed

Description

@zanderso

From https://chromium-swarm.appspot.com/bot?id=build806-m9

Screenshot 2024-10-25 at 13 25 01

XCode installs are failing due to data corruption as in:

https://logs.chromium.org/logs/flutter/buildbucket/cr-buildbucket/8733087603218910657/+/u/install_xcode/install_xcode_from_cipd/stdout

There are other failure modes as well. I suspect a bad disk or bad memory. The automatic reboots on task failure don't seem to be helping.

I put up this CL to remove it from the prod pool, but I'm wondering if this is the right thing to do? https://chrome-internal-review.googlesource.com/c/infradata/config/+/7775927

cc @christopherfujino I'm wondering if bad machines like this are the underlying cause of #156614 ?

Activity

  1. added
    P0Critical issues such as a build break or regression
    triaged-infraTriaged by Infrastructure team
    on Oct 25, 2024
  2. changed the title [-]Bot build806-m9 appears to be in bad shape[/-] [+]A few intel macs appear to be in bad shape[/+] on Oct 28, 2024
  3. zanderso commented on Oct 28, 2024

    @zanderso
    MemberAuthor

    808 and 812 also have an abnormally high job failure rate:

    builder red purple total rate
    build806-m9 26 10 90 40.0%
    build808-m9 13 4 90 18.9%
    build812-m9 13 1 90 15.6%

    In the try pool, the outliers I'm seeing are:

    buidler red purple total rate
    build857-m9 43 0 90 47.8%
    build862-m9 41 0 90 45.6%
    build858-m9 27 0 90 30.0%
  4. christopherfujino commented on Oct 28, 2024

    @christopherfujino
    Contributor
  5. zanderso commented on Oct 29, 2024

    @zanderso
    MemberAuthor

    Based on the fact that many of the task failures are due to a failure to install Xcode, my suspicion is that we should be following the flow to delete the caches on the machine before requesting that the machines be re-imaged, or suspecting that the hardware is bad: go/flutter-luci-playbook under "Delete corrupted Mac caches".

  6. zanderso commented on Nov 15, 2024

    @zanderso
    MemberAuthor

    @jmagman after chatting with the LUCI team, it sounds like caches can be corrupted if we have bots switching back and forth between Xcode versions. It looks like we are still on Xcode 15, but some machines have Xcode 16 on them. Do you know where the Xcode 16 installs are coming from?

  7. jmagman commented on Nov 15, 2024

    @jmagman
    Member

    If it's the try pool then likely from the PRs where I was testing Xcode 16+
    flutter-team-archive/engine#55053
    flutter/packages#7484
    #155980

    I thought the caches handled Xcode versions, and it even cleaned up any detected corrupted versions. Maybe that mechanism broke?
    #138238 (comment)
    #139152 (comment)

    cc @cbracken who maybe worked with @vashworth on this while I was out.

  8. christopherfujino commented on Nov 15, 2024

    @christopherfujino
    Contributor

    In the last 30 days, I count 199 builds using version: 16b40

  9. 23 remaining items

  10. vashworth commented on Jan 24, 2025

    @vashworth
    Contributor
  11. changed the title [-]A few intel macs appear to be in bad shape[/-] [+]Chromium Mac Bots become unresponsive and Xcode commands hang[/+] on Jan 24, 2025
  12. vashworth commented on Jan 24, 2025

    @vashworth
    Contributor
  13. christopherfujino commented on Jan 24, 2025

    @christopherfujino
    Contributor

    I filed #162183 to investigate disabling spotlight on these vms.

  14. harri35 commented on Jan 27, 2025

    @harri35
    Contributor

    Mentioning this here as it may be useful.

    An engineer from Codemagic (a Flutter CI) pointed me to this old issue #35988.
    They told me they used to see a lot of xcoderun hangs with Xcode 11.x and 12.x, mainly with xcoderun -showBuildSettings.

    They checked their current telemetry - they have not logged any instances of it happening in the last month.

    Then it was happening:

    • They never understood why this was happening: the builder was not out of memory, the CPU was not overloaded, etc.
    • As a fix, they just added a heuristic where they used logs to detect these situations. When the situation was detected, they killed the stuck process and restarted it. The command worked fine after the process restart

    I checked Apple's release notes from that time and found no indication of any bug fixes from Xcode 11.x-12.x or later. So either at that time it was not Apple's bug or they fixed it accidentally.

    The only ones I found were some very old ones:

    Xcode 8.3: https://developer.apple.com/library/archive/releasenotes/DeveloperTools/RN-Xcode/Chapters/Introduction.html#//apple_ref/doc/uid/TP40001051-CH1-SW876

    • xcodebuild no longer hangs when invoked with a project or workspace containing no schemes. (20450296)
    • Passing -showBuildSettings to xcodebuild for a project containing a Core Data model should no longer result in xcodebuild hanging. (26749142)

    Xcode 9.3: https://developer.apple.com/library/archive/releasenotes/DeveloperTools/RN-Xcode/Chapters/Introduction.html#//apple_ref/doc/uid/TP40001051-CH1-SW876

    • Fixed a bug which would sometimes cause the -showBuildSettings option to xcodebuild to hang. (33881306)
  15. vashworth commented on Jan 28, 2025

    @vashworth
    Contributor

    Tentative fix is out: https://flutter-review.googlesource.com/c/recipes/+/62584

    Will probably take a few days to propagate across all the bots. Feel free to post timing out tests here if we continue to see issues

  16. vashworth commented on Feb 4, 2025

    @vashworth
    Contributor

    I've been watching the bots and so far so good. I'm considering this fixed. Closing.

  17. zanderso commented on Feb 4, 2025

    @zanderso
    MemberAuthor

    I've been watching the bots and so far so good. I'm considering this fixed. Closing.

    Woo hoo!

  18. jtmcdole commented on Feb 4, 2025

    @jtmcdole
    Member

    Thank you, @vashworth!

  19. github-actions commented on Feb 18, 2025

    @github-actions

    This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.

  20. locked as resolved and limited conversation to collaborators on Feb 18, 2025
  21. jtmcdole commented on Feb 28, 2025

    @jtmcdole
    Member

    We should probably revisit ci.yaml changes that have "#157636" listed as a reason for increasing retries.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

P0Critical issues such as a build break or regressionteam-infraOwned by Infrastructure teamtriaged-infraTriaged by Infrastructure team

Type

No type

Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions