Repository navigation
Should astropy move 32bit CI to weekly. #15007
Description
numpy stopped providing wheels for 32-bit systems starting with numpy 1.22. Since NEP 29 recently dropped numpy 1.21 support, per APE 18 astropy needs to drop support for it as well.
In #15006, we start the process of dropping support for numpy 1.21 however, it appears that since numpy does not provide 32 bit wheels in 1.22+ the 32-bit and parallel CI no longer functions properly without serious work to get numpy to build in our 32bit environment.
numpy stopped providing wheels for 32-bit systems starting with numpy 1.22. However, NEP 29 indicates that numpy recently dropped numpy 1.21 support, per APE 18 astropy needs to drop support for numpy 1.21 as well.
This leads to an issue with our CI. Namely, PR #15006 starts the process of dropping numpy 1.21 support, however due to the lack of 32-bit wheels on PyPi for numpy 1.22+ the 32-bit and parallel CI no longer functions properly because we currently do not have an environment setup to build these wheels.
The current discussion in this Issue indicates that we should not drop 32-bit testing if we can avoid it. This leaves two possibilities:
- Get
numpy 1.22to build in the current 32-bit and parallel CI job. This appears to be fairly complex and will require extensive modifications to the existing workflows to get functional. - Move the 32-bit CI to the Weekly Cron CI. This is much simpler as we already have CI set up to run the astropy tests on non-standard architectures, and we can add a 32-bit architecture to the list of architectures this job is run on. In this job, we pull a version of
numpybuilt and maintained by Debian from the Debian package manager; meaning, we do not have to support figuring out how to buildnumpyon these architectures.
In my opinion option 2 will be better for us to use as 32-bit systems are becoming increasingly "non-standard", and it alleviates our need to figure out how to build numpy in this circumstance. The main downside is that errors normally caught by the 32-bit job (though fairly rare) will not be caught at the PR level, but later when the Weekly Cron is run. In the cases where we suspect 32-bit or other architectures may cause issues we have the Extra CI label which will run the Daily and Weekly Cron jobs on the PR. One other downside of moving the 32-bit job to the Weekly CI is that it is integrated into our parallel and "double" testing job. In this case the 32-bit portion can be removed from the job and the parallel portion can be moved to one of the other jobs.
Activity
[-]Drop support for 32-bit systems[/-][+]Should astropy drop support for 32-bit systems?[/+]saimn commentedon Jun 30, 2023
Ref #14513 and #14517: we recently stopped building i686 wheels, but there was a will from Debian to keep i686 / 32-bit support, and there are other 32-bit architectures.
WilliamJamieson commentedon Jun 30, 2023
The issue is that as far as I can tell
numpydoes not offer pre-built 1.22 wheels.WilliamJamieson commentedon Jun 30, 2023
If we really do want to continue to support 32-bit I suggest pushing it into the weekly "other architectures" matrix.
saimn commentedon Jun 30, 2023
Yes that's an option.
pllim commentedon Jun 30, 2023
cc @olebole and @jehturner
pllim commentedon Jun 30, 2023
Re: other architecture option -- How do we pull in numpy then?
WilliamJamieson commentedon Jun 30, 2023
Well I guess it would work the same way as the s390x and ppc641e CI jobs. Numpy does not provide wheels for those architectures, so they must be building numpy as part of the job.
olebole commentedon Jul 1, 2023
As long as it is not too hard, I would vote for continuing some 32-bit support. Probably not mainly for desktop systems, but there may be use cases for embedded systems f.e. on an older Raspberry.
However, also in Debian there are discussions about how to continue support for i386, and the use of these systems is limited nowadays.
IMO one other reason here is that a supporting 32-bit ensures clean coding (remember: one major reason IRAF was declining was that the world made a 32-64 bit transition, and IRAF, coming from the 80s, had a lot of implicit assumptions about data lengths). Maybe not tomorrow, but there are 128-bit processors knocking on the door (the RISC_V128 architecture is already defined); and looking 30 years forward, supporting 64 bit only may be one reason why people start to hate AstroPy :-)
pllim commentedon Jul 1, 2023
I added this to the 2023-07-07 dev telecon agenda. FYI.
mhvk commentedon Jul 1, 2023
Just so we don't confuse ourselves: for
s390,aarch64andppclewe don't build from source but rely on the Debian builds of numpy (via ubuntu-latest). That seems a good precedent to follow: we should not be building 32-bit numpy ourselves, but as long as it exists in Debian, we might as well support it, and the cron jobs are a good way to ensure we do. If/when Debian stops providing 32-bit numpy, we can stop support too.As far as I know, numpy has only stopped building the wheels, presumably because they do not get used much directly, but not with the intent of no longer supporting bug-fixes for 32-bit.
pllim commentedon Jul 2, 2023
Yes, definitely better if we can just pull the 32-bit version of numpy from Debian itself. What is the incantation for that, @olebole ? Or does it automatically do it with regular apt-get ?
mhvk commentedon Jul 2, 2023
I guess at the worst, can just add the
i386architecture totest_more_architecturesatastropy/.github/workflows/ci_cron_weekly.yml
Lines 111 to 114 in 7ede931
olebole commentedon Jul 2, 2023
I think that i386 is no longer supported as distribution on Ubuntu (and on Github actions); the individual packages are however still supported. That means, one can install them on a x86_64 system.
On esheldon/fitsio, there was a problem with 32-bit last week, which was solved by @esheldon by this (temporary) workflow: .github/workflows/tests-i386.yaml. This may be a starting point for a regular test.
esheldon commentedon Jul 2, 2023
I never got that workflow to run. I ended up using a 32-bit virtual machine on my local system to run tests outside of a workflow.
pllim commentedon Jul 3, 2023
So, what do we do now if we don't want to build 32-bit numpy from source in the CI?
olebole commentedon Jul 3, 2023
I played around a bit with creating a working environment based on @esheldon's work here and this file
seems to work and passes a trivial test about 32/64 bit.Äähm, except that numpy is not available for 32 bit-ubuntu. Hmm.pllim commentedon Jul 3, 2023
p.s. checkout is v3 now, so you should use that instead of v2
WilliamJamieson commentedon Jul 3, 2023
The weekly_cron uses this action: https://github.com/uraimo/run-on-arch-action to select its architectures. I believe arm7 is the architecture used by the 32 bit (native) raspberry pi boards. The data sheet for arm7 indicates it is 32 bit. So I think we can just add that to:
astropy/.github/workflows/ci_cron_weekly.yml
Lines 112 to 114 in 645e587
EDIT: see #15018
mhvk commentedon Jul 3, 2023
Nice using
arm7! Probably covers more users thani386![-]Should astropy drop support for 32-bit systems?[/-][+]Should astropy move 32bit CI to weekly.[/+]