Repository navigation
Add aarch64, ppc64le, and s390x builds to GitHub actions - #11045
Conversation
|
@astrofrog - this looks nice. On the system libraries, I think it is important that we test them somewhere, and ideally in a run that also does other stuff. I thought the other archs were useful for this, since those essentially are used only through Debian/Ubuntu anyway, so we may as well use their packages. But more importantly, you note in the PR that the runs are so slow because they are run on an emulator. I am not sure that in that case there is much point - my sense would be to just keep travis cron jobs instead (this despite rather disliking having multiple systems). |
@mhvk - sadly it's not really an option any more, travis stopped working 2 days ago for all astropy repos, thus the recent hard work of moving everything to other platforms. |
|
As of backport: yes, I think it would be useful to have the test run on 4.0.x, but it cannot be a backport but a separate PR to that branch. Also, I think it's worth to have these tests run on 4.2.x. For the latter, it can be a backport, but there I could bring it to the normal matrix rather than leave it in cron given the branch is low traffic and it's best to see all passes when backports are done, but no need for a cronjob there. |
While it's not ideal, I think that's OK for this infrequent cron. It would be nice to have the cron jobs report to slack though. |
|
Ouch, were we somehow kicked off travis? Anyway, in that case 👍 to this effort. I guess this will catch most bugs (not that we've had many) and in the end we can still count on getting reports from Debian using real hardware. |
|
@mhvk - long story short (more details will come to astropy-dev in an email, soonish): their policy toward open source changed at the beginning of the month, effective immediately, and we run out of the buffer credit they gave us very quickly (all repos in the org use the same resource): https://blog.travis-ci.com/2020-11-02-travis-ci-new-billing |
|
@mhvk - I've now tried to replicate as close as possible to what you had originally - hopefully this will also speed up the builds |
|
Thanks, @bsipocz, given that notice there indeed doesn't seem much future for astropy on travis. Of course, someone has to pay the bills for those CI computers, so I suspect others will eventually follow, and we may end up with some shared infrastructure via NumFocus or so. But hopefully a few years down the line. |
|
p.s. FWIW, numpy runs just the other platforms on travis still. |
| # os: ubuntu-latest | ||
| # python: 3.10-dev | ||
| # toxenv: pydev-test | ||
| shell: /bin/sh |
There was a problem hiding this comment.
@mhvk thanks for investigating! I probably need to change this then :) Will try shortly.
There was a problem hiding this comment.
/bin/bash should exist, since it is a Debian system
There was a problem hiding this comment.
For future reference, there is no need to delete .circleci, ci_cron.yml, or Azure YAML. CircleCI won't run if you do [ci skip] but it does not affect Actions. ci_cron.yml and Azure won't run anyway.
As for the actual implementation of exotic archs, I'll have to defer to @mhvk . Failure now is error trying to exec 'cc1plus': execvp: No such file or directory.
runs just the other platforms on travis still
@mhvk , I think the credits apply to the entire orgs and we run a risk of other packages in the org eating up the credits unexpectedly. If there is somehow a way to tell Travis CI that we can only use the credits for certain repos, then maybe, maybe...
Not sure if we need to backport this or not - what do others think?
@astrofrog , not sure. If we make new release on, say, LTS, is it critical for these archs to run? Actually, maybe discuss at #11050 ?
4476ee8 to
2a48661
Compare
|
I could create a backport (linked to the Buster version of cfitsio) and put it to buster-backports. You would need to add the line to |
|
@olebole - yes please, that would be great. I'm very confused as to how this ever worked before on Travis though 😕 |
|
@astrofrog, @olebole - on travis, I had moved to Ubuntu focal, which likely had newer packages then buster. I don't know how the choice of distribution is made in Actions, but is there perhaps an option to run Debian testing? |
|
Ubuntu Focal (20.04 LTS) has wcslib-7.1, which should be compatible. |
|
@olebole - just to check, could you put a comment here once WCSLIB 7 should be in buster-backports? |
|
Sure. It is still in the ftpmaster queue, since it has to go through a review for the first time. They are quite quick in the moment, so I expect it within a few days. I'll put a note here. |
|
@astrofrog The package was accepted today and should pop up in buster-backports. It may take some hours until it is propagated to all mirrors, and built for all archs. |
2a48661 to
00164e1
Compare
|
@olebole - thanks! |
|
@olebole - I think I'm doing something silly but I can't get apt to find WCSLIB 7: What am I doing wrong? |
|
The package is called |
be2fa5e to
4335d09
Compare
|
Note for other maintainers: still waiting on s390x then will remove the commit disabling the cron |
|
Now, the s390x, ppl64el, arm64 build should be there, says the log. |
|
Thanks! |
|
Great, aarch64 and s390x passed! PPC failed but those are known issues: https://github.com/astropy/astropy/issues?q=is%3Aopen+is%3Aissue+label%3Aplatform-ppc Here's a screenshot of the builds:
I'll now remove the temporary commit. |
…is CI configuration, and rename daily cron file
…s directly with pytest
4335d09 to
0ff2f4f
Compare
|
This is ready for review - note that we don't need to wait for the CI to pass since this just runs in cron. Note that this also removes the Travis configuration 🔥 |
pllim
left a comment
pllim
left a comment
There was a problem hiding this comment.
LGTM, thanks!
As for the known failures, if we are not able to fix them in the next few weeks, maybe it is better to mark them as xfail for PPC platform?
|
@pllim - I don't think there is any harm in keeping them not xfail - to me xfail is where tests go to get ignored forever 😆 - at least it's not like it will fail all CI, just that job. |
Right, but we will get weekly email about failures we have no plans to fix... |
mhvk
left a comment
mhvk
left a comment
There was a problem hiding this comment.
Thanks so much for getting this working, @astrofrog and @olebole !
|
Thanks, all! |
|
🔥 🔥 🔥 |
This makes use of the run-on-arch action to run the test suite with minimal dependencies (for now) on aarch64, ppc64le, and s390x.
The builds are very slow, usually between 1 and 2 hours each but given that this only done once per week, I don't think it's actually a problem.
The builds on Travis used the system libraries bit I think it might be better to use the bundled ones to make sure we compile them right on those systems?
There are some real PPC failures that have been identified by the ppc64le build (#11025, #11026, and #11027). We didn't have a PPC build before, just aarch64 and s390x.
Not sure if we need to backport this or not - what do others think?