TST: aarch64 job times out on main and v6.1.x backport branches

See #16397 (comment)

Apparently the timeout has been consistent but hard to notice when cron jobs do not run on backport branches according to schedule, just when we ask for it.

Update: Actually main is affected too and it has been such for 2 weeks. https://github.com/astropy/astropy/actions/workflows/ci_cron_weekly.yml?query=is%3Asuccess

Interestingly, it seems to almost complete the test suite (timeouts consistently happen on 99% completion), which seems to indicate that something is halting pytest near completion, or that we're extremely unlucky and the whole job would actually take 6hrs + epsilon (which would also be a problem; it's not supposed to take more than an hour or two).

We could try and run the tests just for the sub-package where it is failing to see if it hangs too.

The last time it did not timeout was two weeks ago: https://github.com/astropy/astropy/actions/runs/8838322552/job/24269182128

We could try and run the tests just for the sub-package where it is failing to see if it hangs too.

I'm going to run this experiment on my fork: https://github.com/neutrinoceros/astropy/actions/workflows/ci_cron_weekly.yml

Ok I know that it's not the last tests (docs) that are causing the hang, so the problem could be anywhere. I'll start searching for it by dichotomy, and will update the following list (checked items are known-good)

Maybe see which PRs were merged between last success and first failure, and go for those subpackages first? 🤞

Actually I get no timeouts if I split the test suite in any way, but it does seem that running the whole thing in one go requires about 1 hour to build and 5 more to run every tests, so I'm starting to think we may actually be in my "extremely unlucky" scenario. Something must have changed in the environment for it to get this slow. Next step would be to try and bump the timeout limit a notch and try to hunt down what external changes happened between the latest successful run and the first timeout.

For what it's worth I'm now running an experiment on my fork to with a bumped timeout limit. I'll check back in the morning.

It went through https://github.com/neutrinoceros/astropy/actions/runs/8991193525/job/24698228435
(for reference, I raised the timeout limit to 7 hours, yet it completed in 5h56)

Now we should try to look for differences in the environment. I'll give it a try this morning.

Here's a diff between the outputs of pip list from the last success (> in green) and the first failure (< in red)

<   astropy                  6.1.0rc2.dev25+g3b169347e9 /home/runner/work/astropy/astropy
---
>   astropy                  6.1.0rc2.dev29+gc8016b407c /home/runner/work/astropy/astropy
5,6c5,6
<   attrs                    23.1.0
<   coverage                 7.2.7
---
>   attrs                    23.2.0
>   coverage                 7.4.4
8d7
<   exceptiongroup           1.1.2
10,11c9,10
<   extension-helpers        1.1.0
<   hypothesis               6.82.4
---
>   extension-helpers        1.1.1
>   hypothesis               6.98.15
14,19c13,17
<   MarkupSafe               2.1.3
<   numpy                    1.24.2
<   packaging                23.1
<   pip                      23.2
<   pluggy                   1.2.0
<   psutil                   5.9.4
---
>   MarkupSafe               2.1.5
>   numpy                    1.26.4
>   packaging                24.0
>   pip                      24.0
>   pluggy                   1.4.0
21,23c19,21
<   pytest                   7.4.0
<   pytest-arraydiff         0.5.0
<   pytest-astropy           0.10.0
---
>   pytest                   7.4.4
>   pytest-arraydiff         0.6.1
>   pytest-astropy           0.11.0
26,30c24,27
<   pytest-doctestplus       1.0.0
<   pytest-filter-subpackage 0.1.2
<   pytest-mock              3.11.1
<   pytest-openfiles         0.5.0
<   pytest-remotedata        0.4.0
---
>   pytest-doctestplus       1.2.0
>   pytest-filter-subpackage 0.2.0
>   pytest-mock              3.12.0
>   pytest-remotedata        0.4.1
35c32
<   setuptools-scm           7.1.0
---
>   setuptools-scm           8.0.4
39,41c36,37
<   tomli                    2.0.1
<   typing_extensions        4.7.1
<   wheel                    0.41.0
---
>   typing_extensions        4.10.0
>   wheel                    0.42.0

A lot has changed, so it's still not obvious what makes the job twice slower, if it's even due to a python dependency (as opposed to something more fundamental in the runner image). My primary suspects would be test-related and performance-critical dependencies

package	last success	first failure	notes
coverage	7.2.7	7.4.4	not actually used here
hypothesis	6.82.4	6.98.15
numpy	1.24.2	1.26.4
pytest	7.4.0	7.4.4
pytest-arraydiff	0.5.0	0.6.1	probably not used in enough tests to matter
pytest-astropy	0.10.0	0.11.0	meta-package, no effect at runtime
pytest-doctestplus	1.0.0	1.2.0	probably not used in enough tests to matter
pytest-filter-subpackage	0.1.2	0.2.0
pytest-mock	3.11.1	3.12.0
pytest-remotedata	0.4.0	0.4.1

hypothesis and numpy seem like the two most likely candidates. I'll look up hypothesis' tracker for signs of regressions, but any regression in numpy would more likely be a platform-specific packaging issue.

Hypothesis seems to be subject to memory leaks HypothesisWorks/hypothesis#3975, which could degrade performances if the leak gets too big, but that's a long shot. I'm now running a new experiment on my fork: excluding hypothesis tests and see how much shorter the run gets. Stay tuned for more in a couple hours 🤞🏻

Yes we could in principle skip the hypothesis tests like we do for wheel building.

Actually, do we just want to skip hypothesis for all the exotic archs? Probably easier that way and also might speed up the other archs run time too.

The job has been running for over 5 hours now and isn't finished yet, so it still might save some time but I think we can now rule out hypothesis being responsible for timeouts.

Could be something shipped with the distro too.

ASTROPY_USE_SYSTEM_ALL=1 pip3 install -v --no-build-isolation -e .[test]

p.s. Why are we testing editable install instead of normal?

Do we have anyone with a physical aarch64 machine to see if this is purely a CI problem or something more fundamental?

Why are we testing editable install instead of normal?

I think that's a workaround the fact that we don't have an src-layout : since we're running pytest from the top level of the repo, and the current dir is at the start of PYTHON_PATH, import astropy will look up code locally instead of wherever else pip might install it, so extensions need to be built in-place, otherwise we'd get an ImportError. Editable installs achieve this.

https://github.com/neutrinoceros/astropy/actions/runs/8999144914 completed in 5h33 by skipping hypothesis

I'm hesitant to just do that to "fix" this issue, but it might be better than doing nothing and risk getting used to failures on that job.

@astrofrog

Green with less features is better than timing out. I think @astrofrog already agrees based on #16405 (comment) , so let's do that. Thanks!

#16417 👀

Navigation Menu

Repository navigation

TST: aarch64 job times out on main and v6.1.x backport branches #16405

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

TST: aarch64 job times out on main and v6.1.x backport branches #16405

Description

Activity

neutrinoceros commented on May 7, 2024

astrofrog commented on May 7, 2024

neutrinoceros commented on May 7, 2024

neutrinoceros commented on May 7, 2024

neutrinoceros commented on May 7, 2024

pllim commented on May 7, 2024

neutrinoceros commented on May 7, 2024

neutrinoceros commented on May 7, 2024

neutrinoceros commented on May 8, 2024

neutrinoceros commented on May 8, 2024

neutrinoceros commented on May 8, 2024

neutrinoceros commented on May 8, 2024

astrofrog commented on May 8, 2024

pllim commented on May 8, 2024

neutrinoceros commented on May 8, 2024

pllim commented on May 8, 2024

pllim commented on May 8, 2024

pllim commented on May 8, 2024

neutrinoceros commented on May 8, 2024

neutrinoceros commented on May 8, 2024

pllim commented on May 8, 2024

neutrinoceros commented on May 8, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions