You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
CI wheel building broken due to numpy 1.26 (exotic archs) #15333
I think the builds have started failing just now, because numpy 1.26 is out. It requires GCC >= 8.4, which the CI setup does not provide.
This failure maybe should make us revisit the idea to compile against a new numpy on an old python... It seems that we have become more sensitive recently to updates to dependencies, which then block every PR.
Thanks for looking into this! Could it be that they don't provide a numpy 1.26 wheel for python 3.9? I agree we probably will need some type of pinning in the interim.
More generally, perhaps we need to fix more (all?) of the build dependencies in the various CI runs... I feel we're just chasing geese here (with you getting most of the work...).
perhaps we need to fix more (all?) of the build dependencies in the various CI runs.
I think it is quite unreasonable to ask us to build with unstable numpy. We already test against dev numpy at runtime and none of these were caught. Ideas are welcome though.
I don't think numpy 1.26 is itself unstable, it just requires some relatively new environment to be built in...
My suggestion: we fix the build environment (i.e., all versions fixed) any time we have a new release.
But right now your suggestion of a numpy pin seems the easiest! It could be for the wheel build one only
p.s. You probably already checked, but nominally numpy 1.26 supports python 3.9, so indeed I'd think there be wheels for it. It may well be that the problem is pip/pypi here, going for a source build when we'd happily take numpy 1.25. No idea whether one can turn that off...
To clarify, I was referring to future-proofing our own builds against unreleased numpy, which I think is an unwinnable battle, especially since they might not even push dev wheels for all the archs we test against.
@pllim - OK, I hadn't realized this is not even in our control... I don't think I understand the wheel building environment well enough to comment any further!
For those who read this issue later, while this does make a mess of the CI, astropy at runtime for typical users should not be affected. We already tested against unreleased numpy and 1.26rc1, so I have high confidence the code would work. This should only affect the builds (which would theoretically block our releases if not fixed quickly enough).
changed the title [-]CI cp39-manylinux_x86_64 wheel building broken due to numpy 1.26 [/-][+]CI wheel building broken due to numpy 1.26 (exotic archs)[/+]on Sep 18, 2023
Update: The normal wheels are fixed at #15339 (thanks, William)!
There is still the cron job with "exotic" archs that is failing with 3/4 archs. We do not use cibuildwheel for those, so they have to be investigated separately.
tl;dr -- PRs with "Extra CI" label are still blocked.
#15339 resolved the main problem raised in this issue. The exotic archs CI is really a separate issue from this discussion, as such I am closing this issue in favor of #15351 which is specifically focused on that problem.
Activity
pllim commentedon Sep 18, 2023
That is weird, why would it need GCC if we can grab the wheel?
pllim commentedon Sep 18, 2023
This is coming from
astropy/pyproject.toml
Line 5 in b0065e8
"Exotic archs" are also failing: https://github.com/astropy/astropy/actions/runs/6218804937/job/16875804544
pllim commentedon Sep 18, 2023
I asked NumPy people. But in the meantime, we might have to do a
<1.26pin onpyproject.toml.mhvk commentedon Sep 18, 2023
Thanks for looking into this! Could it be that they don't provide a numpy 1.26 wheel for python 3.9? I agree we probably will need some type of pinning in the interim.
More generally, perhaps we need to fix more (all?) of the build dependencies in the various CI runs... I feel we're just chasing geese here (with you getting most of the work...).
pllim commentedon Sep 18, 2023
They do at https://pypi.org/project/numpy/#files but I don't know how to translate the filenames to why some builds are failing. Maybe it takes them a while to get all the archs up, or they dropped support for a bunch of stuff? 🤷
I think it is quite unreasonable to ask us to build with unstable numpy. We already test against dev numpy at runtime and none of these were caught. Ideas are welcome though.
mhvk commentedon Sep 18, 2023
I don't think
numpy 1.26is itself unstable, it just requires some relatively new environment to be built in...My suggestion: we fix the build environment (i.e., all versions fixed) any time we have a new release.
But right now your suggestion of a numpy pin seems the easiest! It could be for the wheel build one only
p.s. You probably already checked, but nominally numpy 1.26 supports python 3.9, so indeed I'd think there be wheels for it. It may well be that the problem is
pip/pypihere, going for a source build when we'd happily take numpy 1.25. No idea whether one can turn that off...pllim commentedon Sep 18, 2023
I don't understand what you mean here. The build environments seem fine to me until numpy 1.26 came out. What needs fixing?
pllim commentedon Sep 18, 2023
To clarify, I was referring to future-proofing our own builds against unreleased numpy, which I think is an unwinnable battle, especially since they might not even push dev wheels for all the archs we test against.
pllim commentedon Sep 18, 2023
Numpy also said something about we have to "update our environments" and pointed to https://docs.scipy.org/doc/scipy/dev/toolchain.html but I don't have time to look at this. So if someone is passionate about builds, please propose a better fix than #15336 . Thanks!
pllim commentedon Sep 18, 2023
p.s. Since we also grab workflows from OpenAstronomy, I don't even know if it is something we can fix here or someone needs to patch https://github.com/OpenAstronomy/github-actions-workflows/blob/main/.github/workflows/publish.yml .
@astrofrog @ConorMacBride ?
ngoldbaum commentedon Sep 18, 2023
In particular, the wheel building environment needs to be updated to the manylinux2 build images for wheels that depend on numpy 1.26 or newer.
pllim commentedon Sep 18, 2023
Yes, I agree. #15336 is a no-go.
pllim commentedon Sep 18, 2023
In the meantime, feel free to ping me to override branch protection for uncontroversial PRs to be merged. Thanks!
mhvk commentedon Sep 18, 2023
@pllim - OK, I hadn't realized this is not even in our control... I don't think I understand the wheel building environment well enough to comment any further!
pllim commentedon Sep 18, 2023
For those who read this issue later, while this does make a mess of the CI, astropy at runtime for typical users should not be affected. We already tested against unreleased numpy and 1.26rc1, so I have high confidence the code would work. This should only affect the builds (which would theoretically block our releases if not fixed quickly enough).
pllim commentedon Sep 18, 2023
I misunderstood the problem. William pointed out that we might indeed be able to fix within this repo. Thanks, William!
astrofrog commentedon Sep 18, 2023
I'm pretty sure we can configure this via pyproject.toml - will look shortly
[-]CI cp39-manylinux_x86_64 wheel building broken due to numpy 1.26 [/-][+]CI wheel building broken due to numpy 1.26 (exotic archs)[/+]pllim commentedon Sep 18, 2023
Update: The normal wheels are fixed at #15339 (thanks, William)!
There is still the cron job with "exotic" archs that is failing with 3/4 archs. We do not use cibuildwheel for those, so they have to be investigated separately.
tl;dr -- PRs with "Extra CI" label are still blocked.
WilliamJamieson commentedon Sep 19, 2023
#15339 resolved the main problem raised in this issue. The exotic archs CI is really a separate issue from this discussion, as such I am closing this issue in favor of #15351 which is specifically focused on that problem.