Skip to content

CI wheel building broken due to numpy 1.26 (exotic archs) #15333

Closed

Description

@mhvk

Description

I think the builds have started failing just now, because numpy 1.26 is out. It requires GCC >= 8.4, which the CI setup does not provide.

This failure maybe should make us revisit the idea to compile against a new numpy on an old python... It seems that we have become more sensitive recently to updates to dependencies, which then block every PR.

Sample failing build: https://github.com/astropy/astropy/actions/runs/6212653493/job/16863021788?pr=15304

Blocked by

Activity

  1. pllim commented on Sep 18, 2023

    @pllim
    Member

    That is weird, why would it need GCC if we can grab the wheel?

  2. pllim commented on Sep 18, 2023

    @pllim
    Member
  3. pllim commented on Sep 18, 2023

    @pllim
    Member

    I asked NumPy people. But in the meantime, we might have to do a <1.26 pin on pyproject.toml.

  4. mhvk commented on Sep 18, 2023

    @mhvk
    ContributorAuthor

    Thanks for looking into this! Could it be that they don't provide a numpy 1.26 wheel for python 3.9? I agree we probably will need some type of pinning in the interim.

    More generally, perhaps we need to fix more (all?) of the build dependencies in the various CI runs... I feel we're just chasing geese here (with you getting most of the work...).

  5. pllim commented on Sep 18, 2023

    @pllim
    Member

    they don't provide a numpy 1.26 wheel for python 3.9?

    They do at https://pypi.org/project/numpy/#files but I don't know how to translate the filenames to why some builds are failing. Maybe it takes them a while to get all the archs up, or they dropped support for a bunch of stuff? 🤷

    perhaps we need to fix more (all?) of the build dependencies in the various CI runs.

    I think it is quite unreasonable to ask us to build with unstable numpy. We already test against dev numpy at runtime and none of these were caught. Ideas are welcome though.

  6. mhvk commented on Sep 18, 2023

    @mhvk
    ContributorAuthor

    I don't think numpy 1.26 is itself unstable, it just requires some relatively new environment to be built in...

    My suggestion: we fix the build environment (i.e., all versions fixed) any time we have a new release.

    But right now your suggestion of a numpy pin seems the easiest! It could be for the wheel build one only

    p.s. You probably already checked, but nominally numpy 1.26 supports python 3.9, so indeed I'd think there be wheels for it. It may well be that the problem is pip/pypi here, going for a source build when we'd happily take numpy 1.25. No idea whether one can turn that off...

  7. pllim commented on Sep 18, 2023

    @pllim
    Member

    fix the build environment

    I don't understand what you mean here. The build environments seem fine to me until numpy 1.26 came out. What needs fixing?

  8. pllim commented on Sep 18, 2023

    @pllim
    Member

    I don't think numpy 1.26 is itself unstable

    To clarify, I was referring to future-proofing our own builds against unreleased numpy, which I think is an unwinnable battle, especially since they might not even push dev wheels for all the archs we test against.

  9. pllim commented on Sep 18, 2023

    @pllim
    Member

    Numpy also said something about we have to "update our environments" and pointed to https://docs.scipy.org/doc/scipy/dev/toolchain.html but I don't have time to look at this. So if someone is passionate about builds, please propose a better fix than #15336 . Thanks!

  10. pllim commented on Sep 18, 2023

    @pllim
    Member

    p.s. Since we also grab workflows from OpenAstronomy, I don't even know if it is something we can fix here or someone needs to patch https://github.com/OpenAstronomy/github-actions-workflows/blob/main/.github/workflows/publish.yml .

    @astrofrog @ConorMacBride ?

  11. ngoldbaum commented on Sep 18, 2023

    @ngoldbaum
    Contributor

    In particular, the wheel building environment needs to be updated to the manylinux2 build images for wheels that depend on numpy 1.26 or newer.

  12. pllim commented on Sep 18, 2023

    @pllim
    Member

    Yes, I agree. #15336 is a no-go.

  13. pllim commented on Sep 18, 2023

    @pllim
    Member

    In the meantime, feel free to ping me to override branch protection for uncontroversial PRs to be merged. Thanks!

  14. mhvk commented on Sep 18, 2023

    @mhvk
    ContributorAuthor

    @pllim - OK, I hadn't realized this is not even in our control... I don't think I understand the wheel building environment well enough to comment any further!

  15. pllim commented on Sep 18, 2023

    @pllim
    Member

    For those who read this issue later, while this does make a mess of the CI, astropy at runtime for typical users should not be affected. We already tested against unreleased numpy and 1.26rc1, so I have high confidence the code would work. This should only affect the builds (which would theoretically block our releases if not fixed quickly enough).

  16. pllim commented on Sep 18, 2023

    @pllim
    Member

    I misunderstood the problem. William pointed out that we might indeed be able to fix within this repo. Thanks, William!

  17. astrofrog commented on Sep 18, 2023

    @astrofrog
    Member

    In particular, the wheel building environment needs to be updated to the manylinux2 build images for wheels that depend on numpy 1.26 or newer.

    I'm pretty sure we can configure this via pyproject.toml - will look shortly

  18. changed the title [-]CI cp39-manylinux_x86_64 wheel building broken due to numpy 1.26 [/-] [+]CI wheel building broken due to numpy 1.26 (exotic archs)[/+] on Sep 18, 2023
  19. pllim commented on Sep 18, 2023

    @pllim
    Member

    Update: The normal wheels are fixed at #15339 (thanks, William)!

    There is still the cron job with "exotic" archs that is failing with 3/4 archs. We do not use cibuildwheel for those, so they have to be investigated separately.

    tl;dr -- PRs with "Extra CI" label are still blocked.

  20. WilliamJamieson commented on Sep 19, 2023

    @WilliamJamieson
    Contributor

    #15339 resolved the main problem raised in this issue. The exotic archs CI is really a separate issue from this discussion, as such I am closing this issue in favor of #15351 which is specifically focused on that problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions