You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Chromium Mac Bots become unresponsive and Xcode commands hang #157636
Based on the fact that many of the task failures are due to a failure to install Xcode, my suspicion is that we should be following the flow to delete the caches on the machine before requesting that the machines be re-imaged, or suspecting that the hardware is bad: go/flutter-luci-playbook under "Delete corrupted Mac caches".
@jmagman after chatting with the LUCI team, it sounds like caches can be corrupted if we have bots switching back and forth between Xcode versions. It looks like we are still on Xcode 15, but some machines have Xcode 16 on them. Do you know where the Xcode 16 installs are coming from?
I thought the caches handled Xcode versions, and it even cleaned up any detected corrupted versions. Maybe that mechanism broke? #138238 (comment) #139152 (comment)
An engineer from Codemagic (a Flutter CI) pointed me to this old issue #35988.
They told me they used to see a lot of xcoderun hangs with Xcode 11.x and 12.x, mainly with xcoderun -showBuildSettings.
They checked their current telemetry - they have not logged any instances of it happening in the last month.
Then it was happening:
They never understood why this was happening: the builder was not out of memory, the CPU was not overloaded, etc.
As a fix, they just added a heuristic where they used logs to detect these situations. When the situation was detected, they killed the stuck process and restarted it. The command worked fine after the process restart
I checked Apple's release notes from that time and found no indication of any bug fixes from Xcode 11.x-12.x or later. So either at that time it was not Apple's bug or they fixed it accidentally.
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of flutter doctor -v and a minimal reproduction of the issue.
Activity
[-]Bot build806-m9 appears to be in bad shape[/-][+]A few intel macs appear to be in bad shape[/+]zanderso commentedon Oct 28, 2024
808 and 812 also have an abnormally high job failure rate:
In the try pool, the outliers I'm seeing are:
christopherfujino commentedon Oct 28, 2024
Filed internal https://b/376081455 to investigate the bots.
zanderso commentedon Oct 29, 2024
Based on the fact that many of the task failures are due to a failure to install Xcode, my suspicion is that we should be following the flow to delete the caches on the machine before requesting that the machines be re-imaged, or suspecting that the hardware is bad: go/flutter-luci-playbook under "Delete corrupted Mac caches".
Retry mac_unopt one time in presubmit
Retry mac_unopt one time in presubmit (#56319)
zanderso commentedon Nov 15, 2024
@jmagman after chatting with the LUCI team, it sounds like caches can be corrupted if we have bots switching back and forth between Xcode versions. It looks like we are still on Xcode 15, but some machines have Xcode 16 on them. Do you know where the Xcode 16 installs are coming from?
jmagman commentedon Nov 15, 2024
If it's the try pool then likely from the PRs where I was testing Xcode 16+
flutter-team-archive/engine#55053
flutter/packages#7484
#155980
I thought the caches handled Xcode versions, and it even cleaned up any detected corrupted versions. Maybe that mechanism broke?
#138238 (comment)
#139152 (comment)
cc @cbracken who maybe worked with @vashworth on this while I was out.
christopherfujino commentedon Nov 15, 2024
In the last 30 days, I count 199 builds using version: 16b40
23 remaining items
vashworth commentedon Jan 24, 2025
@jmagman Here's some examples of other bot types (other than macOS 14 x86):
arm64 macOS 14 -> https://luci-milo.appspot.com/ui/p/flutter/builders/prod/Mac_arm64%20build_tests_1_4/5110/overview
macos 13 (although this could be a different issue) -> https://ci.chromium.org/ui/p/flutter/builders/try/Mac_arm64%20ios_platform_tests_shard_2%20master/15600/overview
Also I remembered I filed this forever ago and could be related: #138224
[-]A few intel macs appear to be in bad shape[/-][+]Chromium Mac Bots become unresponsive and Xcode commands hang[/+]vashworth commentedon Jan 24, 2025
Other notes:
I did the
lsregistercommand @cbracken recommended in #157636 (comment) on bot https://chromium-swarm.appspot.com/bot?id=build874-m9 sometime on 01/22 in the evening.And I just did it for https://chromium-swarm.appspot.com/bot?id=build866-m9.
Update: Also reset build850-m9 and build855-m9
christopherfujino commentedon Jan 24, 2025
I filed #162183 to investigate disabling spotlight on these vms.
harri35 commentedon Jan 27, 2025
Mentioning this here as it may be useful.
An engineer from Codemagic (a Flutter CI) pointed me to this old issue #35988.
They told me they used to see a lot of
xcoderunhangs with Xcode 11.x and 12.x, mainly withxcoderun -showBuildSettings.They checked their current telemetry - they have not logged any instances of it happening in the last month.
Then it was happening:
I checked Apple's release notes from that time and found no indication of any bug fixes from Xcode 11.x-12.x or later. So either at that time it was not Apple's bug or they fixed it accidentally.
The only ones I found were some very old ones:
Xcode 8.3: https://developer.apple.com/library/archive/releasenotes/DeveloperTools/RN-Xcode/Chapters/Introduction.html#//apple_ref/doc/uid/TP40001051-CH1-SW876
Xcode 9.3: https://developer.apple.com/library/archive/releasenotes/DeveloperTools/RN-Xcode/Chapters/Introduction.html#//apple_ref/doc/uid/TP40001051-CH1-SW876
vashworth commentedon Jan 28, 2025
Tentative fix is out: https://flutter-review.googlesource.com/c/recipes/+/62584
Will probably take a few days to propagate across all the bots. Feel free to post timing out tests here if we continue to see issues
vashworth commentedon Feb 4, 2025
I've been watching the bots and so far so good. I'm considering this fixed. Closing.
zanderso commentedon Feb 4, 2025
Woo hoo!
jtmcdole commentedon Feb 4, 2025
Thank you, @vashworth!
github-actions commentedon Feb 18, 2025
This thread has been automatically locked since there has not been any recent activity after it was closed. If you are still experiencing a similar issue, please open a new bug, including the output of
flutter doctor -vand a minimal reproduction of the issue.jtmcdole commentedon Feb 28, 2025
We should probably revisit ci.yaml changes that have "#157636" listed as a reason for increasing retries.