Cannot build astropy in CI for exotic archs #15351

This issue is related to and referenced by #15333, but it is a separate issue from the general wheel building which was fixed by #15339.

Last week (2023-09-11), all of the weekly cron CI's "exotic" arch tests succeeded: https://github.com/astropy/astropy/actions/runs/6142559584. However this week (2023-09-18) we got failures in the:

• s390x
• ppc64Ie
• armv7 (32bit)

tests: https://github.com/astropy/astropy/actions/runs/6218804937.

The failures are due to failure to even create the build environment for building the astropy wheel to test rather than failures in astropy itself. It appears the issue stems from a combination of the changes from #14949 and the release of numpy 1.26.

PR #14949 made the minimum numpy needed to build (not run) the astropy wheel 1.25. This means that numpy 1.25+ is needed in order to build the astropy wheel in the build environment setup by pip. This is posing a problem because prebuilt versions of numpy are not available for the archs in question. In particular, we were use the Ubuntu python3-numpy package installed via apt-get install rather than a wheel provided by pip. Currently, the latest version of numpy provided by Ubuntu (for the lunar version we use in CI) is numpy 1.24.2. The reason why #14949 did not cause exotic arch failures when it required numpy>=1.25 for building astropy is because it just so happens that the environment we build in these systems was able to allow pip to build numpy 1.25 from source as part of creating the build environment for building the astropy wheel.

Now that numpy 1.26 has been released, pip will grab numpy 1.26 for the astropy build environment (because its the latest version which satisfies our requirements), which also has no pre-builds available for these archs. This is the root of the issue because while our CI happened to have an environment which supported building numpy 1.25 it does not support building numpy 1.26.

As we have discussed in previous in #15018, it is preferable for astropy to not concern itself with how to build numpy in its CI, especially in the exotic archs CI. I believe that this is an entirely reasonable position, especially since numpy seems to be shifting its build system around in ways that make it so we have to play constant catch-up in order to continue our exotic arch testing CI. This leads to several possible solutions to this issue:

1. Cease the exotic arch testing on archs which numpy does not directly provide a wheel for (so anything that isn't x86_64, aarch64, or arm64).

   • This is likely an untenable option, as there are community members who really need us to do this.

2. Ask numpy to provide prebuilt wheels for the arches we need.

   • numpy is unlikely to do this just for astropy.

3. Ask Ubuntu or Debian to provide packages we can install through their package managers.

   • This might be possible as it has been done for us in the past by @olebole for other packages (on Debian at least).

4. Add some complex logic to

   astropy/pyproject.toml

   Line 5 in b9b552a

   "numpy>=1.25,<2",

   in order to select numpy < 1.25 (or 1.26) in the archs we are experiencing difficulties with.
   • This is likely possible, but misleading as astropy prefers the 1.25 build requirement, see Drop oldest-supported-numpy from build requirements when numpy 1.25 is released #14915. Moreover, I do not think it is appropriate for astropy to do something this drastic just to make CI pass on archs that it does not want to maintain numpy building itself on.

5. Revert the Build: Replace oldest-supported-numpy #14949 changing the numpy requirement for the build environment from oldest_supported_numpy to 1.25.

   • The discussions around Drop oldest-supported-numpy from build requirements when numpy 1.25 is released #14915 indicate that we would prefer to keep the 1.25 requirement.

6. Pin the numpy requirement for the build environment <1.26.

   • I don't think this is a good option as one of our goals for the 1.25 lower pin was that so versions of astropy built with this pin should be at least forwards compatible with the numpy 2.0 C API.

7. Muddle through and attempt to somehow make our exotic arch environment compatible with building numpy from source under these circumstances.

   • As already alluded to, astropy does not really want to do this just to support the exotic archs builds as it is a time sink. However, doing this as an interim patch until some other fix for the issue is found.

This solution list is likely not exhaustive so alternate solutions to discuss are welcome.

In any case, I personally don't have much more time to devote to resolving the exotic archs CI failures so I think this issue needs to be opened to the general community to find a tenable (hopefully more long-term) solution.