Releasing PowerPC (ppc64le) wheels?

@rgommers

Proposed new feature or change:

On many leading supercomputers in the US, PowerPC is used. For example, Summit at OLCF and Sierra/Lassen at LLNL.

Installing numpy on those with pip currently needs to build from source.

Describe the solution you'd like.

I saw you already publish arm64 wheels for Linux on pypi. Could you potentially also publish them for ppc64le?

In my projects, I use the free TravisCI PowerPC runners (no limit/credits because they are part of the Partner Queue) for these deployments via cibuildwheel.

Describe alternatives you've considered.

We currently build ourselves from source. Sometimes, these builds are so long that they often killed by runtime limits on shared/login nodes of HPC systems. Running on exclusive compute nodes for these systems is not always possible without work-arounds, since they might not have internet access for downloads.

Using other package managers such as conda and spack has their own issues, some fundamental and some can be overcome in the future, documented here and here, but they work in a subset of workflows.

Update: I will likely focus on Spack and @rgommers's contributions: scipy/scipy#17054 (comment)

Additional context (e.g. screenshots, GIFs)

scipy/scipy#17054 (comment)

We would be happy to take a PR if you want to work on this. Note that PPC64le on TravisCI goes missing on a regular basis, That could be very frustrating when trying to make a release.

@charris

Thanks @charris, can you tell me where your release logic resides? Is this a separate repo or branch? Are there docs one can read?

@lithomas1

The release logic is scattered around, but we use cibuildwheel for the builds. Some of it is in tools/wheels, some in .travis.yml, and some in .github/workflows/wheels.yml. @lithomas1 did most of the work for this, I'm sure he can offer better advice than I can.

Note that wheels will be built from the PR if you start the commit message with 'BLD:`. They aren't uploaded, but can be tested that way.

@rgommers

Thanks! The fundamental question is how much of a frustration the Travis-CI flakes are for you. I definitely have to restart about 1 in 10 IBM PPC64le builds when I do releases, as you mentioned above. I am not sure if cross-compiles would be an option to avoid this.

Alternatively, if Travis is currently our only option, I am also open to explore more of @rgommers's work in Spack for our users' purposes:
scipy/scipy#17054 (comment)

Restarts are fine, the problem is that the machine is simply missing. I'm not vested in any particular service, whatever works is fine as long as it doesn't take days to run :)

I'm fine with adding the ppc64le wheels.
Wheel building logic on travis starts here

numpy/.travis.yml

Lines 53 to 55 in 2524a53

    
           # Wheel builders 
        
           # Note that Ubuntu focal comes with Python 3.8 and CIBW_BUILD determines 
        
           # the Python used to build the wheels.

Personally, I'm not a huge fan of travis. It was causing headaches for us on pandas when the arm64-gravitron builds were not starting, and the status page didn't say anything even though it was down for everyone I think.

@charris

@charris I think numpy might want to start off maybe with a kind of "best effort" support for ppc64le.

This is as in, if the ppc64le wheels build, then great upload for the release, otherwise, they can be uploaded later.

The not AMD64 architectures can take a while to build and might flake out or hang, so its probably better not to block or prolong a release if anything happens with th builds.

I'm fine with adding the ppc64le wheels. Wheel building logic on travis starts here

numpy/.travis.yml

Lines 53 to 55 in 2524a53

# Wheel builders

# Note that Ubuntu focal comes with Python 3.8 and CIBW_BUILD determines

# the Python used to build the wheels.

Personally, I'm not a huge fan of travis. It was causing headaches for us on pandas when the arm64-gravitron builds were not starting, and the status page didn't say anything even though it was down for everyone I think.

You'd want to copy the logic for arm64 and change the CIBW_BUILD env var, to whatever the cibuildwheel identifier for ppc64le is.

The cibuildwheel docs are a good starting point.
https://cibuildwheel.readthedocs.io/en/stable/options/#build-skip

For numpy, whatever cibuildwheel options are not configured in the .travis.yml script are found in the pyproject.toml file in the root directory.

Hello,

If you need any additional manpower for this task I can offer my help, as a part of tango controls community.

Currently we are working on building wheels for arm, amd, ppc64le architectures using manylinux image. Having numpy wheels available would make things more efficient for us, so I can start working on this numpy ticket ASAP.

Best regards

It should also be possible to build the ppc64le and s390x wheels on GHA using QEMU. THe cibuildwheel documentation lists how this is done: https://cibuildwheel.readthedocs.io/en/stable/faq/#emulation

@charris

@charris I think numpy might want to start off maybe with a kind of "best effort" support for ppc64le.

This is as in, if the ppc64le wheels build, then great upload for the release, otherwise, they can be uploaded later.

This is difficult, we much prefer never to do this. The problem is that pip prefers new sdist's over older binaries, so a pip install pkg-depending-on-numpy is likely going to break the moment we fail to upload a new wheel for ppc64le before a new sdist. Or, perhaps even worse, it silently picks up lapack_lite instead of a serious BLAS library and causes large performance regressions.

Adding CI support for any platform is cheap in that respect, because we can remove it again at any time. Adding wheels is a large commitment. We really need that platform support NEP, I'll get around to writing that soon I think.

For ppc64le, I'd prefer to stick with referring people to conda-forge or Spack, or having them build from source, until there is reliable CI support.

For the CI use case, what can be done is to upload ppc64le wheels to https://anaconda.org/scipy-wheels-nightly/numpy/files. That's fine, and a much smaller commitment. (side note, sdist upload to that bucket is wrong and must be fixed, or we'll see the same problems as with PyPI - in fact, Pandas already had those)

I'll add to that that we moved wheel building for PyPI to https://github.com/numpy/numpy-release and using trusted publishing; we don't want to add self-hosted runners there for security reasons. Which connects to the cross-compilation part of how we'd have to go about it (and we wouldn't run tests for cross compiled wheels); from this mailing list thread:

That said, I'm hoping we will have proper cross compilation support very soon, because PEP 739 support in Meson got merged. So it may actually not be that high-effort after that. The question then is still whether we want to publish wheels on PyPI, since it's a long-term commitment for a limited set of users - hard to estimate how many there are, but less than 1% of the total user base almost certainly.

Then again, we did add Windows arm64 wheels, which have way fewer users than PowerPC. So it's always a judgement call.

Another feeling was that most users are HPC users, and they'd get better performance when building from source (e.g., using Spack)

I don't think this assumption is true. Power users/devs want to install it in the most convenient way - its primarily enterprise data scientists.

we don't want to add self-hosted runners there for security reasons

what are your specific security concerns?

Regarding wheels the general opinion was less positive, because of the larger amount of work, long-term support commitment, and low number of users

Given the fact that Windows arm64 has wheels, how much more work would it take to get ppc64le wheels as well? We can help with some of the development in the repos and of course provide the github actions. Thanks 😸

@rgommers

@rgommers, wanted to check if there’s been any update on allowing GHA self-hosted runners for ppc64le wheel builds — we can provide secure, maintained infrastructure if that helps move things forward. We already use a GHA self-hosted runner for NumPy test jobs successfully.

There is an active discussion at scientific-python/summit-2025-nov#4, that will be updated over the next few days when maintainers of multiple projects are getting together in person. It's also a good overview of the load of wheel building and how many platforms people want added.

After that discussion wraps up, we'll know more here.

@rgommers

@rgommers, Thanks for the update! We’ll keep an eye on the outcome of the Scientific Python Summit discussion.

From our side, PowerPC (ppc64le) has been supported for a long time, and there’s been a funded IBM initiative over the past several months to upstream CI/CD support across key Python projects. IBM is committed to long-term maintenance of this architecture, and our GitHub Actions self-hosted runners are secure and actively maintained.

As an example, the Cryptography project already publishes wheels built using IBM’s self-hosted GHA runners — you can see the implementation here:
🔗 cryptography/.github/workflows/wheel-builder.yml#L86

Given NumPy’s foundational role, enabling ppc64le wheels would unlock consistent support for downstream projects like TensorFlow, PyTorch, SciPy, h5py and many more. We’d be happy to assist with CI configuration, testing, and validation whenever the community is ready to proceed.

@rgommers

Hi @rgommers, just checking in — has there been any update from the Scientific Python Summit discussion regarding ppc64le wheel support.

@sandeepgupta12

@sandeepgupta12 thanks for the extra context and the ping. I wasn't at that event myself, so I don't know - I just pinged for an update. If it wasn't discussed there, I'll try to get a NumPy project decision on this. Either way we'll need to write up a policy with some clear rules on how we decide on these kinds of new platform wheel build requests, so we have a fair and transparent way of deciding on this. scientific-python/summit-2025-nov#4 does have a lot of the relevant content for writing such a policy.

@rgommers

Hi @rgommers,

I checked the update shared by @lesteve in the Scientific Python Summit discussion. Just following up on possible next steps. From the ppc64le side, a few quick points that may help move this forward:

• Platform stability: ppc64le has long-term support, with ongoing IBM-funded maintenance. Our GitHub Actions self-hosted runners are secure, dedicated, and actively maintained for upstream open-source CI.

• Proven infra: Projects like Cryptography already build and publish wheels using our ppc64le runners.

• Usage signals: BigQuery shows non-trivial ppc64le downloads (e.g., ~209 weekly for scikit-learn). For NumPy, this would likely be higher given its central role across the ecosystem.

• Ecosystem impact: NumPy is the main dependency blocker. Enabling ppc64le wheels here would unlock consistent binary support for SciPy, TensorFlow, PyTorch, h5py, and others — and we are already in active discussions with these communities about upstreaming CI support.

We’re ready to help with onboarding ppc64le GHA runners for NumPy, maintaining and monitoring CI, testing prereleases, and contributing documentation for the new platform policy.

Could you share expected next steps or an approximate timeline for the platform-policy draft?

Thanks — happy to collaborate whenever the team is ready.

BigQuery shows non-trivial ppc64le downloads (e.g., ~209 weekly for scikit-learn)

Just to be 100% explicit, this is an extremely small number in absolute terms and also relative to the number of sdist downloads ... this number comes from scientific-python/summit-2025-nov#4 (comment) and is only about sdist downloads. If you look at wheel download we are talking of the order of 30M downloads per week see https://clickpy.clickhouse.com/dashboard/scikit-learn for example.

The elephant in the room, sorry to be blunt about it: if we decide to provide ppc64le wheels, we are going to help mostly IBM customers for free (to be explicit it looks like this is a clear no from a number of projects right now see scikit-learn/scikit-learn#31810 (comment)). What would be our incentive as open-source maintainers to do this? A potential funding discussion would involve a whole different can of worms ...

Cryptography already build and publish wheels

Cryptography uses the limited API and releases 53 wheels @~5MB/wheel per version, 6 of them in the last release are ppc64le. NumPy does not use the limited API, and has (without ppc64le) 73 wheels per version @~11MB/wheel (size varies depending on OpenBLAS). Adding ppc64le would mean ~6-12 more wheels (depending if we release musl and glibc builds, and whether we have 4 or 3 python versions to release). We already hitting PyPI storage limits. And this is only NumPy, I think SciPy, pandas, scikit-learn, etc are similarly pressed for storage on PyPI. So this decision influences not only the projects, but the PSF infrastructure we depend on.

@mtarsel

Well, if you all are data scientists and counting numbers... ;) Most of the arch specific wheels for Power are released today through our own DevPi infrastructure, Marvin Giessing's fury infrastructure, Red Hat's infastructure, or wherever we can get them built and hosted. We have been working to get the full Python Ecosystem enabled for Power 10/11 upstream for the past year or so after our relationship with Anaconda took a shift and we decided to go mainstream. And, for data scientists, IBM Power 10/11 hosts about an Exabyte of active data at Fortune 500 customers (very, very private data) that runs something like >60% of the global economy (well, probably need to factor IBM's Z in there as well with a similar data footprint). There's another 5-6 Exabytes of RAID/offline store of course, but, these customers have a strong interest in using data science to process that information (hopefully for good, I tend to assume that, but...) and they are all active users of the Python ecosystem one way or another. We would love to at least get things integrated from a source perspective (we've been doing well with that across the board) and a build perspective. This thread mentions Travis - we built and offered gratis the backend for Travis on Power to the open source community in partnership with Travis. We are doing the same with GitHub Actions. @mtarsel is the lead for that and onboarding orgs and repos and a furious clip, as well as working to continue to improve both the integrated service and, for those that prefer, runners that they can operate on their own or privately. The discussion on PyPI storage limits is interesting, and we can't solve the global problem - we are looking for <500 wheels (but among the top used arch specific ones) to be available. That's a small percentage of the (I lost count >1M wheels now?)? So, just trying to keep "small" and "large" in perspective here and we would love to help find a path forward to enable more Enterprises to continue to get value out of the Python Ecosystem. We fully agree that it is cool and the place to be. ;)

At least on the scikit-learn side, to me this means we shouldn't be doing it at least for free: scikit-learn/scikit-learn#30973 (comment)

Bringing up this discussion again because I'm not sure what is a blocking adding wheels for ppc64le.

Wheels can be generated using our GHA (no-charge) service, and PR's have been merged showing success.
For example. Python Crypto library is building wheels using our GHA service today.

We are committed to help with the development and maintenance of the ppc64le wheels. How can we move this discussion along? thanks

@mtarsel

@mtarsel thanks for asking again, and sorry this conversation moves slowly. I appreciate it's frustrating to just not get a final answer.

Given that scientific-python/summit-2025-nov#4 didn't really get engagement and I finally have some more bandwidth again, I'm going to try and draft a policy (NEP) with some criteria that can help us decide, and can give clarity to the companies and communities that want to get their platform supported.

I'll put up a rough draft with basic principles first, that should hopefully not take too long.

Wheels can be generated using our GHA (no-charge) service

Yes, we appreciate that. That is a win/win for IBM and NumPy I think - we have complex PowerPC/Z-specific code, so testing it better helps us all.

The cost/support argument is a little different though - you're essentially asking maintainers (a combo of volunteers and folks paid by other companies) to take on a lot of what you were previously paying Anaconda for. It's a nontrivial amount of effort to commit to building and providing long-term support for wheels for any platform. And it's both more work and less useful to the person doing the work if they have to do it for a platform that they don't use themselves and not have easy/local access to for development work.

Okay, first draft complete (gh-30748) and discussion started on the mailing list:
https://mail.python.org/archives/list/[email protected]/thread/UZKO3V7GYT3QNXRCBIYDVVUNA6EXLZVY/

Navigation Menu

Repository navigation

Releasing PowerPC (ppc64le) wheels? #22318

Proposed new feature or change:

Describe the solution you'd like.

Describe alternatives you've considered.

Additional context (e.g. screenshots, GIFs)

11 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	# Wheel builders
	# Note that Ubuntu focal comes with Python 3.8 and CIBW_BUILD determines
	# the Python used to build the wheels.

Releasing PowerPC (ppc64le) wheels? #22318

Description

Proposed new feature or change:

Describe the solution you'd like.

Describe alternatives you've considered.

Additional context (e.g. screenshots, GIFs)

Activity

charris commented on Sep 20, 2022

ax3l commented on Sep 20, 2022

charris commented on Sep 20, 2022

charris commented on Sep 20, 2022

ax3l commented on Sep 20, 2022

charris commented on Sep 20, 2022

lithomas1 commented on Sep 20, 2022

lithomas1 commented on Sep 20, 2022

lithomas1 commented on Sep 20, 2022

spirit1317 commented on Sep 22, 2022

andyfaff commented on Jan 20, 2023

rgommers commented on Jan 20, 2023

11 remaining items

rgommers commented on Sep 11, 2025

mtarsel commented on Sep 15, 2025

sandeepgupta12 commented on Nov 4, 2025

rgommers commented on Nov 4, 2025

sandeepgupta12 commented on Nov 5, 2025

sandeepgupta12 commented on Nov 13, 2025

rgommers commented on Nov 14, 2025

sandeepgupta12 commented on Dec 10, 2025

lesteve commented on Dec 11, 2025

mattip commented on Dec 11, 2025

gerrith3 commented on Dec 11, 2025

adrinjalali commented on Dec 12, 2025

mtarsel commented on Jan 30, 2026

rgommers commented on Jan 30, 2026

rgommers commented on Jan 30, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions