Skip to content

Install openblas to compile recent Numpy versions - #15589

Merged
pllim merged 3 commits into
astropy:mainfrom
saimn:exotic-archs
Nov 8, 2023
Merged

pllim merged 3 commits into
astropy:mainfrom
saimn:exotic-archs

Conversation

@saimn

@saimn saimn commented Nov 6, 2023 •

Copy link
Copy Markdown
Contributor

Ref #15351

Fix #15351

Description

This pull request is to address ...

Fixes #

  • By checking this box, the PR author has requested that maintainers do NOT use the "Squash and Merge" button. Maintainers should respect this when possible; however, the final decision is at the discretion of the maintainer that merges the PR.

@saimn saimn added Extra CI Run cron CI as part of PR backport-v6.0.x labels Nov 6, 2023
@saimn saimn added this to the v6.0.1 milestone Nov 6, 2023
@github-actions

github-actions Bot commented Nov 6, 2023

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Astropy! 🌌 This checklist is meant to remind the package maintainers who will review this pull request of some common things to look for.

  • Do the proposed changes actually accomplish desired goals?
  • Do the proposed changes follow the Astropy coding guidelines?
  • Are tests added/updated as required? If so, do they follow the Astropy testing guidelines?
  • Are docs added/updated as required? If so, do they follow the Astropy documentation guidelines?
  • Is rebase and/or squash necessary? If so, please provide the author with appropriate instructions. Also see instructions for rebase and squash.
  • Did the CI pass? If no, are the failures related? If you need to run daily and weekly cron jobs as part of the PR, please apply the "Extra CI" label. Codestyle issues can be fixed by the bot.
  • Is a change log needed? If yes, did the change log check pass? If no, add the "no-changelog-entry-needed" label. If this is a manual backport, use the "skip-changelog-checks" label unless special changelog handling is necessary.
  • Is this a big PR that makes a "What's new?" entry worthwhile and if so, is (1) a "what's new" entry included in this PR and (2) the "whatsnew-needed" label applied?
  • Is a milestone set? Milestone must be set but we cannot check for it on Actions; do not let the green checkmark fool you.
  • At the time of adding the milestone, if the milestone set requires a backport to release branch(es), apply the appropriate "backport-X.Y.x" label(s) before merge.

@github-actions

github-actions Bot commented Nov 6, 2023

Copy link
Copy Markdown
Contributor

👋 Thank you for your draft pull request! Do you know that you can use [ci skip] or [skip ci] in your commit messages to skip running continuous integration tests until you are ready?

@saimn

saimn commented Nov 7, 2023

Copy link
Copy Markdown
Contributor Author

It works, the ppc job passed, and I just pushed a fix for the s390x job.
But armv7 job fails while compiling Numpy (building ninja, which itself uses scibit-build and cmake...). Maybe we just skip this one for now ?

@saimn
saimn marked this pull request as ready for review November 7, 2023 11:30
@pllim

pllim commented Nov 7, 2023

Copy link
Copy Markdown
Member

armv7 is the 32-bit replacement added by @WilliamJamieson . Maybe he has ideas? Otherwise, maybe @olebole can advise?

@WilliamJamieson

Copy link
Copy Markdown
Contributor

armv7 is the 32-bit replacement added by @WilliamJamieson.

This is correct, the original intent was to install numpy via the OS's package manager:

However, both Ubuntu and Debian do not currently do not provide numpy>=1.25 for any of the OS versions, the most they provide is numpy==1.24.2. Since numpy>=1.25 is required by the build system:
"numpy>=1.25,<2",
the test requires us to have the later version to be installed in order to just install astropy in order to run the test.

This means if we want to require numpy>=1.25 to build the astropy wheel, we are stuck until Ubuntu or Debian get around to publishing numpy==1.25 packages or we work out what additional packages we need to install via the OS package manager in order to build numpy from source.

Comment thread .github/workflows/ci_cron_weekly.yml Outdated
wcslib-dev \
liberfa1
liberfa1 \
libopenblas-dev

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
libopenblas-dev
libopenblas-dev \
ninja-build

This might fix the build error originating from building the ninja wheel.

@olebole

olebole commented Nov 7, 2023

Copy link
Copy Markdown
Member

@WilliamJamieson

This means if we want to require numpy>=1.25 to build the astropy wheel, we are stuck until Ubuntu or Debian get around to publishing numpy==1.25 packages or we work out what additional packages we need to install via the OS package manager in order to build numpy from source.

Unfortunately, this seems to be delayed somehow. I opened Debian#1052320 6 weeks ago, and didn't get a response; no idea why.
The additional packages required to build are listed in Debian's source package for version 1.24, however some featured may be patched out (or fixed), so it may be useful to check the patches for 1.24 as well.

@saimn

saimn commented Nov 7, 2023

Copy link
Copy Markdown
Contributor Author

Right, then I think we can build directly with the system packages in non-isolated mode. Let's see...

@WilliamJamieson

Copy link
Copy Markdown
Contributor

An alternate to having to use the OS package manager to install the various packages needed to build astropy might be to move the exotic arch builds to follow what numpy is doing in its CI as suggested by #15351 (comment). This will require a complete redo of this part of the job.

@saimn

saimn commented Nov 7, 2023

Copy link
Copy Markdown
Contributor Author

Seems it worked, but all 4 jobs now fail on the same 2 tests:

  FAILED astropy/coordinates/tests/test_representation.py::TestCartesianRepresentation::test_transform_non_contiguous_matrix - assert False
   +  where False = representation_equal(<CartesianRepresentation (x, y, z) in kpc\n    (1., 8., 12.)>, <CartesianRepresentation (x, y, z) in kpc\n    (1., 2., 3.)>)
   +    where <CartesianRepresentation (x, y, z) in kpc\n    (1., 8., 12.)> = <bound method CartesianRepresentation.transform of <CartesianRepresentation (x, y, z) in kpc\n    (1., 2., 3.)>>(array([[1., 0., 0.],\n       [0., 1., 0.],\n       [0., 0., 1.]]))
   +      where <bound method CartesianRepresentation.transform of <CartesianRepresentation (x, y, z) in kpc\n    (1., 2., 3.)>> = <CartesianRepresentation (x, y, z) in kpc\n    (1., 2., 3.)>.transform
  FAILED astropy/time/tests/test_basic.py::test_to_datetime_pytz - pytz.exceptions.UnknownTimeZoneError: 'US/Hawaii'

@pllim

pllim commented Nov 7, 2023

Copy link
Copy Markdown
Member

pytz.exceptions.UnknownTimeZoneError: 'US/Hawaii'

Wut? 🤯

Do we also have to install tzdata? https://stackoverflow.com/questions/21717411/timezone-information-missing-in-pytz

@olebole

olebole commented Nov 7, 2023

Copy link
Copy Markdown
Member

@pllim on new Debian system (not sure whether before of after Debian 12 "Bookworm" release), one also needs the tzdata-legacy package. This is no longer a mandatory package.

@saimn

saimn commented Nov 8, 2023

Copy link
Copy Markdown
Contributor Author

Finally all builds are green !
Though 5h42m for aarch64 seems a bit excessive ... do we really need to run such a slow thing ?
image

@pllim pllim left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Amazing. You are a hero!

@pllim
pllim merged commit 1ae26ae into astropy:main Nov 8, 2023
meeseeksmachine pushed a commit to meeseeksmachine/astropy that referenced this pull request Nov 8, 2023
@saimn
saimn deleted the exotic-archs branch November 8, 2023 17:13
@saimn saimn removed this from the v6.0.1 milestone Nov 8, 2023
@saimn saimn added this to the v6.0 milestone Nov 8, 2023
pllim added a commit that referenced this pull request Nov 8, 2023
…589-on-v6.0.x

Backport PR #15589 on branch v6.0.x (Install openblas to compile recent Numpy versions)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cannot build astropy in CI for exotic archs

4 participants