This is a bit confusing because this buildroot roll passed the presubmit checks, but it looks like it caused build failures on the prod builder. @godofredoc are you aware of any configurations differences between the machines?

This is a bit confusing because this buildroot roll passed the presubmit checks, but it looks like it caused build failures on the prod builder. @godofredoc are you aware of any configurations differences between the machines?

Is it possible that the build failing on prod was filtered out from the PR? all the mac bots have the same specs and provisioned with the same configs.

Here: https://github.com/flutter/engine/pull/23986/checks

"Mac Android AOT Engine" passed the presubmit check here https://ci.chromium.org/ui/p/flutter/builders/try/Mac%20Android%20AOT%20Engine/13901/overview

but failed in postsubmit here: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20Android%20AOT%20Engine/8662/overview

Here: https://github.com/flutter/engine/pull/23986/checks

"Mac Android AOT Engine" passed the presubmit check here https://ci.chromium.org/ui/p/flutter/builders/try/Mac%20Android%20AOT%20Engine/13901/overview

but failed in postsubmit here: https://ci.chromium.org/ui/p/flutter/builders/prod/Mac%20Android%20AOT%20Engine/8662/overview

Took a look at the two builds the configurations are exactly the same. Not sure if having having some intermediate objects in cache can cause this behavior. Prod builds usually have a higher cache hit rate than presubmit builds.

In the interests of getting the tree open again, I'm going to land this revert, but we will need to ask @godofredoc for help if the potential caching issues happen again on the reland, and we can dig in a bit more then during the day.

The linked patch does make it so that the executable will be more diligent about not leaking libc++ symbols. Maybe we were just getting away with it earlier? We should try to reproduce LTO with bitcode builds on Mac with that commit patched it. I suspect it may not be an issue with cached contents.

Was it just Android that failed, or all platforms?

Looks like mac building for both Android and iOS, but check the red cells on the dashboard on the roll commit for the definitive answer.

I believe try builders do not use LTO.

To reproduce this locally, you'll want to build with LTO and with bitcode enabled, which unfortunately means no goma.