Repository navigation
OSX 10.15 images deprecated in azure, to be removed end of August #1798
Description
Message started appearing in OSX jobs:
The macOS-10.15 environment is deprecated, consider switching to macos-11(macos-latest),
macos-12 instead. For more details see https://github.com/actions/virtual-environments/issues/5583
Quoting from the upstream issue:
Deprecation will begin on 5/31/2022 and the image will be fully unsupported by 8/30/2022
To raise awareness of the upcoming removal, we will temporarily fail jobs using macOS 10.15. Builds that are scheduled to run during the brownout periods will fail. The brownouts are scheduled for the following dates and times:
GitHub Actions:
- July 21, 12:00 UTC - July 22, 18:00 UTC
- July 27, 00:00 UTC - July 28, 00:00 UTC
- August 3, 00:00 UTC - August 4, 00:00 UTC
- August 15, 00:00 UTC - August 16, 00:00 UTC
- August 26, 00:00 UTC - August 27, 00:00 UTC
- August 29, 00:00 UTC - August 30, 00:00 UTC
Azure DevOps
- August 1, 12:00 UTC - August 1, 14:00 UTC
- August 9, 16:00 UTC - August 9, 18:00 UTC
- August 17, 14:00 UTC - August 17, 18:00 UTC
- August 25, 14:00 UTC - August 25, 20:00 UTC
- September 2, 12:00 UTC - September 2, 22:00 UTC
- September 8, 00:00 UTC - September 9, 00:00 UTC
- September 14, 00:00 UTC - September 15, 00:00 UTC
- September 20, 00:00 UTC - September 21, 00:00 UTC
- September 26, 00:00 UTC - September 27, 00:00 UTC
Those brownouts get pretty extensive - 24h for all but the first on GHA, and the same a bit later on Azure - so it sounds we need to move until end of July with this.
Reacted by ngam
Activity
jakirkham commentedon Jul 16, 2022
cc @conda-forge/core
beckermr commentedon Jul 16, 2022
Is this for azure or GitHub actions or both?
h-vetinari commentedon Jul 16, 2022
The brownouts mentioned in actions/runner-images#5583 and the OP are for GHAs, and doesn't give a target date for azure, but the warning that caused me to open this issue was on azure
ngam commentedon Jul 16, 2022
It's the same for azure. Some builds will get this warnings, e.g. this one,
h-vetinari commentedon Jul 21, 2022
I just checked, the oldest SDK that macos-11 comes with is 10.15, plus xcode 11.7. As far as I can tell, that version still supports deployment target 10.9 (actually 10.6-10.15), so it doesn't sound like the image change forces us to change anything substantial about MacOS support 1
Footnotes
It's worth noting though that 10.9 is EOL for more than 5 years already, but moving on from that is a separate discussion. ↩
beckermr commentedon Jul 21, 2022
Thank you!
@chrisburr had some nice stats about where we go next on osx pulled from pypi usage. I forget but moving to somewhere around 10.12 would be pretty non-destructive IIRC.
jakirkham commentedon Jul 21, 2022
Would suggest we separate discussing the CI image change from updating the SDK used to build (if possible). Given CI is the primary driver of discussion here, let's focus on that. If we would like to discuss SDK changes, let's open a new issue (unless these are somehow intertwined).
In terms of CI image used (the primary concern here), moving macOS 11 is the next oldest thing we can use (as 10.15 is the last in the 10.x line).
h-vetinari commentedon Jul 21, 2022
AFAIU, the SDK is not really the issue, the
MACOS_DEPLOYMENT_TARGETspecifies the compatibility. That said, I had assumed that we were using one of the SDK's in the current 10.15 image, but that's not the case, because we're still using the 10.9 SDK, but I couldn't easily find (e.g. in smithy) where that is injected.So with that misunderstanding out of the way - fine by me to leave the SDK untouched (assuming the current SDK injection mechanism works in the macos-11 image).
jakirkham commentedon Jul 21, 2022
All good. Just trying to help dial things in a bit so we can make decisions 🙂
jaimergp commentedon Jul 22, 2022
So we have roughly a month to:
macos-11as soon as possibleDoes that sound correct?
beckermr commentedon Jul 22, 2022
We have not historically worried about rerendering in these cases. People should rerender almost always when making new PRs.
beckermr commentedon Jul 22, 2022
So we only need a PR to smithy.
jaimergp commentedon Jul 22, 2022
And we don't have a reason to suspect that the current way we are installing SDKs on the image would happen to fail on MacOS 11, right? I guess we can only try... I'll open a PR on conda-smithy to see what happens.
jaimergp commentedon Jul 22, 2022
Let's see: conda-forge/dav1d-feedstock#2
jakirkham commentedon Jul 22, 2022
Don't think so, but agree a test is a better way to check
6 remaining items
jakirkham commentedon Jul 22, 2022
Ah sorry. Was thinking beginning of August for some reason 🤦♂️
Yeah brownouts are an issue too though
Thanks Jaime! 🙏
jaimergp commentedon Jul 25, 2022
Just realized we also need to do this for staged-recipes. Submitted PR here: conda-forge/staged-recipes#19800
After that, I think we can consider this solved and closed.
ngam commentedon Jul 25, 2022
No, as you figured (we also have used variations of macos-11 and macos-12 in different feedstocks, one could set this in conda-forge.yml, e.g. see this PR from a while back: #1638 and it works fines).
However, it may be worth considering trying to better understand the SDKs situation. The website/repo we get stuff from is no longer maintained, and it doesn't have many of the newer SDKs. Fortunately, for now, most of these newer SDKs come with the images anyway so it's not a big deal. However, it would likely be best to think this through again and decide how best to proceed (i.e. use another repo as shown here: conda-forge/conda-forge-ci-setup-feedstock#201 or rely entirely on what the vmImages offer for example this is what macos-11 has: https://github.com/actions/virtual-environments/blob/main/images/macos/macos-11-Readme.md#installed-sdks)
ngam commentedon Jul 25, 2022
They're somewhat intertwined because the images come preconfigured with certain SDKs and that's exactly what we relied on in the pytorch feedstock to support MPS (metal). But yes, this should be more thoroughly discussed elsewhere. Just wanted to point out this was on the radar before and highlight how the SDKs and vmImages relate to a degree.
Yes, from my previous testing, this is solved and closed after the two PRs
jaimergp commentedon Jul 25, 2022
Thanks @ngam!
ngam commentedon Jul 25, 2022
Thank YOU for the PRs 😃
ngam commentedon Jul 25, 2022
FYI @beckermr the change may take a bit to come online with conda-smithy (hence rerendering may not effect changes yet) because the CDN cloning has a "major outage"... https://conda-forge.org/status/
ngam commentedon Jul 25, 2022
Also, before this is closed, we need to push a release of conda-smithy, I believe. Otherwise, the changes won't take place.
h-vetinari commentedon Aug 1, 2022
We're now in the first brownout on azure (dates of which have been added to the upstream issue, as well as the OP).
Could we have a new smithy release?
ngam commentedon Aug 1, 2022
@conda-forge/core @conda-forge/conda-smithy
h-vetinari commentedon Aug 1, 2022
Felipe commented in conda-forge/conda-smithy#1645
To take the heat off a bit - it's not a crisis. The brownout is only two hours this time. But since that smithy PR has been merged (and I assume the repo is in a releasable state) it would still be good to get a new smithy release soon (i.e. before the next brownout in a week). It'll take a while anyway until feedstocks have been rerendered in large numbers.
beckermr commentedon Aug 1, 2022
release is done - should be live in an hour or so
h-vetinari commentedon Aug 1, 2022
With smithy & staged-recipes now done, I think we can close this. Thanks all! :)
jakirkham commentedon Aug 12, 2022
JFYI we recently saw an (admittedly somewhat niche) issue with Python 3.7 on macOS 11 ( conda-forge/python-feedstock#575 ). Mentioning here merely for awareness.