ENH: upload musllinux wheels to PyPI

Proposed new feature or change:

CI request: build PEP 656-compliant "musllinux" wheels. This will allow easy installation in Alpine Docker images. Mailing list message

I guess we could use a manylinux image for this. If we do this, we should start with x86_64 and add other architectures on demand. I would like to support aarch64, but as it is the travis-ci.com support for ARM is flaky.

Mailing list thread on this topic here.

Any updates on this?

Migration to cibuildwheel is in progress for the next release. We will not do anything about musllinux before that migration is complete. Then we will assess - if it's easy to do, we will likely add musllinux wheels. If not, then not.

@rgommers

@rgommers it seems like with #20102, #21128 and others being closed, it seems like cibuildwheel is pretty well baked into numpy at this point - is that accurate? If so, this might be ready to go sometime soon.

I hope it's as simple as just adding musl to the cibuildwheel config but even if it's not super straightforward, I'd advocate for putting some effort into getting them working (I'd be willing to help out if I can / if needed). Minimal docker containers are about the perfect use case for wheels, every webdev numpy user would love shaving 10+ minutes off of their image builds with no effort.

Just to get an idea of how much tweaking might be needed, I tried building a wheel on Alpine and running the tests. Results:

============== 19 failed, 17416 passed, 1407 skipped, 1300 deselected, 21 xfailed, 5 xpassed, 15 errors in 321.71s (0:05:21) ===============

The failures are in test_umath (divide by 0 issues) and the errors are all in test_cython (issues with subprocess calls to run setup.py) and test_mem_policy (compilation errors). I couldn't run with -m full for some reason, test_format gets marked as "killed" and stopped the tests. I'm not sure if these issues are just due to my environment setup, and didn't look too much into what might have caused them.

Anyway, if anyone wants to quickly recreate:

# On host
docker run -it --rm python:3.10-alpine sh

# In container
apk add git build-base
pip install build virtualenv
git clone https://github.com/numpy/numpy.git && cd numpy
git submodule update --init
python -m build
python -m virtualenv /venv
source /venv/bin/activate
pip install -r test_requirements.txt
pip install dist/numpy*.whl
apk del git build-base    # make sure we aren't reliant on any headers
python runtests.py -v -n

The first step in this would be for someone to submit a PR to add the musl build. We use the pyproject.toml, which currently skips the *-musllinux* builds.

@rgommers

The first step in this would be for someone to submit a PR to add the musl build.

Or start trying to fix the 19 failures.

@rgommers it seems like with #20102, #21128 and others being closed, it seems like cibuildwheel is pretty well baked into numpy at this point - is that accurate? If so, this might be ready to go sometime soon.

It's indeed getting close. We should probably do a full test run of that machinery. Or maybe you have already done that @charris?

Or maybe you have already done that ...?

cibuildwheel is running on a cron job. Here is the latest run, the wheels generated have the numpy-1.23.0.dev0+867.gfa0881fe1 prefix. These were uploaded to https://anaconda.org/scipy-wheels-nightly/numpy/files. PR #21095 will add the missing sdist which is the last part before we can start using this in earnest.

Alright, I did the basic changes on #21200. No guarantee that the same issues will show up so we'll see what happens when the pipeline goes through.

I have a couple questions but I'll follow up with them on the PR.

Do not attempt to build musllinux on aarch64

Add support for building on aarch64 on Linux (#771)

Just a note here: calling people who want this to work to take a look at the failing tests in #21200. There are apparently some glitches to get NumPy working with musllinux. Since this is only a "nice to have if it doesn't cause a maintenance burden" feature for most core NumPy devs, others who feel this is important will have to help out to get the tests to pass.

Let's go for it. Closing.

@mattip

Let's go for it. Closing.

Agreed, thanks for finalizing this @mattip. And sorry for not having written the platform support NEP yet (too many NEPs to write!).

Thanks for this.
Just wanted to clarify if aarch64/arm64 wheels would be part of the release.

Just wanted to clarify if aarch64/arm64 wheels would be part of the release.

It looks like we have x86-64 wheels only right now, so that should be it for 1.25.0. The demand for aarch64 musllinux support seems smaller, so we'd need to separately assess later. Can you elaborate on how/why you'd use aarch64 wheels?

It looks like we have x86-64 wheels only right now, so that should be it for 1.25.0. The demand for aarch64 musllinux support seems smaller, so we'd need to separately assess later. Can you elaborate on how/why you'd use aarch64 wheels?

Docker/Kubernetes in the Apple m-series processor mac's & Amazon Graviton backed EKS.

As well as Raspberry Pi clusters.

@rgommers

@rgommers I don't see musllinux wheels on https://pypi.org/project/numpy/#files, but this PR was merged. Where to look?

Raspberry Pi wheels are available from piwheels.org. It's not simple to compile them on CI infrastructure, coz it's not a native platform, and there are several cpu architectures, e.g. armv6, armv7, armv8. It's always a tradeoff between demand and the work required to support a target.

I don't see musllinux wheels

https://pypi.org/project/numpy/1.25.0rc1/, it is a pre-release.

@andyfaff

@andyfaff I see. But I was asking about musllinux wheels in general.

@charris

@charris But then again, this PR was merged. The issue tracker's state suggest the work is finished, but it isn't, is it?

The consequence is that in many traplike scenarios, Alpine Linux users will end up compiling NumPy. This greatly lengthens e.g., container build times, which is a substantial cost in time and money.

This is why I'm inquiring.

Where to look?

For the nightly builds https://pypi.anaconda.org/scientific-python-nightly-wheels
For the rc https://anaconda.org/multibuild-wheels-staging/numpy

Thanks. Is there a reason why they're not up on PyPI?

Is there a reason why they're not up on PyPI?

The rc releases are on PyPI. I wouldn't use the staging repo, I delete stuff there on a regular basis once it is put on PyPI.

Navigation Menu

Repository navigation

ENH: upload musllinux wheels to PyPI #20089

Proposed new feature or change:

12 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

ENH: upload musllinux wheels to PyPI #20089

Description

Proposed new feature or change:

Activity

mattip commented on Oct 11, 2021

rgommers commented on Oct 11, 2021

gaby commented on Jan 6, 2022

rgommers commented on Jan 6, 2022

tgross35 commented on Mar 15, 2022

mattip commented on Mar 15, 2022

rgommers commented on Mar 15, 2022

mattip commented on Mar 15, 2022

tgross35 commented on Mar 15, 2022

mattip commented on May 3, 2022

12 remaining items

mattip commented on May 17, 2023

rgommers commented on May 17, 2023

sxm51 commented on Jun 8, 2023

rgommers commented on Jun 8, 2023

sxm51 commented on Jun 9, 2023

modem7 commented on Jun 9, 2023

sanmai-NL commented on Jun 9, 2023

andyfaff commented on Jun 9, 2023

charris commented on Jun 9, 2023

sanmai-NL commented on Jun 9, 2023

sanmai-NL commented on Jun 9, 2023

mattip commented on Jun 11, 2023

sanmai-NL commented on Jun 11, 2023

charris commented on Jun 11, 2023

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions