ENH: add riscv64 to the wheel build matrix - #30995

Thanks @gounthar. I appreciate the introduction as well (that's new in our PR template, you may be the first one to use it!).

Note that this won't close the linked issue, because that's asking for wheels on PyPI rather than only a CI job. We don't publish from this repo, so this does only the latter.

Before I trigger a whole battery of CI, let me ask if you have a CI log of a passing build on your own fork? That's the usual way to develop a new job, since you can run just that one job rather than ~70 jobs.

Thanks for the welcome @rgommers!

Here's a passing CI run on our fork: https://github.com/gounthar/numpy/actions/runs/22899916618

It builds the cp312-manylinux_riscv64 wheel using QEMU emulation via docker/setup-qemu-action. The workflow is essentially what this PR adds: a [ubuntu-22.04, manylinux_riscv64, ""] entry in the buildplat matrix with a conditional QEMU setup step.

Happy to adjust anything based on your review.

FWIW, 38 minutes is longer than any other CI job we're currently running.

@ngoldbaum Fair concern on the 38-minute build time: that's QEMU emulation overhead. A native riscv64 build would be significantly faster. 🤞

https://cloud-v.co/ provides free self-hosted GitHub Actions runners on real riscv64 hardware to open source projects. If NumPy is open to a self-hosted runner approach, that would cut the build time substantially and also give more reliable test results than emulation.

In the meantime, is the 38-minute QEMU job acceptable as a starting point, or would you prefer to wait for a native runner solution?

This is really quite slow, and it doesn't add much over this job which tests most of the very platform-specific code we have:

So I'd prefer to only add a new job if we can have a native runner with reasonable stability and performance. I commented at #30216 (comment) with my preferred option.

Makes sense: QEMU is indeed painfully slow for a full test suite. I'll rework this PR to target Cloud-V native runners instead.

I'll install the Cloud-V GitHub App on my fork first to validate the workflow (runner provisioning, caching, build times) and update this PR once it's working reliably.

Sounds good, thanks @gounthar

Things are progressing in #30216 thanks to the help of rise.

Updated this PR to use native riscv64 runners from RISE instead of QEMU emulation.

Build time: ~3.5 minutes on native hardware vs ~38 minutes under QEMU.

The approach was validated on a fork with RISE runners: riseproject-dev#1

Key changes from the original PR:

• Runner: ubuntu-24.04-riscv (native) instead of ubuntu-22.04 (QEMU)
• manylinux image: manylinux_2_39 (first version with riscv64 support)
• cibuildwheel installed via pip (actions/setup-python doesn't have riscv64 pre-built Python yet)

Thanks to @luhenry and the RISE project for providing the native runners and for their support on the riscv64 Python ecosystem.

Thanks for the update, that sounds promising.

Build time: ~3.5 minutes on native hardware vs ~38 minutes under QEMU.

The latest CI log seems to say 18 minutes in total, and the tests didn't get run? Is that run-to-run variation with getting different hardware allocated to the job, or something else?

@rgommers Good catch — let me clarify the numbers:

• cibuildwheel build step: ~3.5 minutes (the actual wheel compilation)
• Total job time: ~18 minutes (includes runner setup, Docker image pull, pip installs, cibuildwheel overhead)

The Docker image pull is a one-time cost that would be cached on subsequent runs. The build step itself is comparable to other architectures.

Regarding tests: the current PR uses a smoke test (import numpy; numpy.show_config()) instead of the full pytest suite because pytest-xdist has a known test collection ordering bug that fails on any platform with multiple workers. It's not riscv64-specific — the same bytes_ vs str_ ordering diff shows up. The actual test results from our BananaPi F3 (running the full suite natively) pass.

Once the RISE runners are installed on numpy's org, we can iterate on getting the full test suite working in CI.

Not quite I think. At least in the raw GitHub log with timestamps, the build lasts from 18:46:21 to 18:58:04, which is 11 min 45 sec.

2026-03-18T18:46:08.2641548Z * Creating isolated environment: venv+pip...
2026-03-18T18:46:08.3243733Z * Installing packages in isolated environment:
2026-03-18T18:46:08.3247866Z   - Cython>=3.0.6
2026-03-18T18:46:08.3250826Z   - meson-python>=0.18.0
2026-03-18T18:46:16.0256161Z * Getting build dependencies for wheel...
2026-03-18T18:46:17.3550138Z * Installing packages in isolated environment:
2026-03-18T18:46:17.3554845Z   - ninja >= 1.8.2
2026-03-18T18:46:21.2471308Z * Building wheel...
2026-03-18T18:46:26.4585476Z + /tmp/build-env-9r0vkz1e/bin/python /project/vendored-meson/meson/meson.py setup /project /project/build -Dbuildtype=release -Db_ndebug=if-release -Db_vscrt=md -Duse-ilp64=true -Dallow-noblas=false --native-file=/project/build/meson-python-native-file.ini
2026-03-18T18:47:29.6619355Z The Meson build system

...

2026-03-18T18:58:04.0656571Z Successfully built numpy-2.5.0.dev0-cp312-cp312-linux_riscv64.whl

And pulling the manylinux_2_39_riscv64 container takes just under 4 minutes.

Either way, we can give it a go though.

@rgommers You're right, I was looking at a different metric. The actual wheel build is ~12 minutes, not 3.5. Thanks for the careful log analysis. Still a big improvement over 38 minutes under QEMU, and the Docker pull will cache on subsequent runs.

Glad we can give it a go! I'm running the full numpy test suite natively on the BananaPi F3 right now (skipping f2py which needs gfortran for test collection). Will share the results once it completes.

@rgommers Full native test suite results from BananaPi F3 (SpacemiT K1, rv64gc):

47,447 passed, 4 failed, 1,301 skipped, 10 errors in 42 minutes.

Pass rate: 99.97%

The 4 failures are NOT riscv64-specific:

• test_cython: needs Cython installed
• 3x test_configtool: numpy config tool path differences (pip install vs system)

The 10 errors are all test_mem_policy.py: test infrastructure issue loading a shared library.

f2py tests were excluded (need gfortran for test collection, separate from numpy core).

numpy works on RISC-V.

Update: installed Cython + meson on the F3; test_cython now passes. The remaining 3 failures are all test_configtool (numpy config tool path differences, not riscv64-specific).

Updated numbers: 47,448 passed, 3 failed, 1,301 skipped, 10 errors: 99.97% pass rate on native riscv64.

@rgommers very good to meet at PyTorch Conf this week! Please let me know how you would want to proceed, and happy to help in any way.

@andyfaff Not yet. The RISE runners app would need to be installed on the numpy org by a maintainer. My fork validation ran on riseproject-dev/numpy where it's already set up. @luhenry can help with any questions on the runner side if needed.

@gounthar the actions/setup-python should now work with /opt/hostedtoolcache once https://github.com/riseproject-dev/riscv-runner-images/actions/runs/24685283231/job/72207968562 finishes.

Has the RISE Github app already been connected?

@rgommers would know more about that AFAIU

Simplifying the change at https://github.com/luhenry/numpy/tree/add-riscv64-wheels. I'm debugging an issue with the python in the venv that doesn't have the proper RPATH, fixing the binaries at https://github.com/riseproject-dev/python-versions/commits/. Once I've verified it works on my fork, I'll update the base image and everything should "just work". That will take a day or two, given the many versions and the base image would need to be rebuilt. I'll keep posting here with updates.

Getting there, down to 34 minutes in this job after including test selection changes in gh-31420:

Build and test phase are then taking about as long. We may indeed want to consider changing this job to use ccache next, that should bring it down to 20-25 minutes which is reasonable.

I think we don't necessarily need to keep things in wheels.yml anyway, because if we do decide to promote RISC-V to Tier 2, we'll need to cross-compile (or use QEMU, but that'll be very slow again) since we don't want to use self-hosted runners or any caching on the numpy-release repo for supply chain security reasons (see its README for details).

@rgommers the first line did it:

root@k1:~# source .venv/bin/activate
(root) root@k1:~# python3
Python 3.12.3 (main, Nov  6 2024, 18:32:19) [GCC 13.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.random.seed(1)
>>> shape = (50, 50)
>>> t = [-1, -2]
>>> a = np.random.randn(*shape)
>>> a = np.matmul(a.transpose(t).conj(), a)
>>> a = np.asarray(a, dtype=np.float64)
>>> c = np.linalg.cholesky(a, upper=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/root/.venv/lib/python3.12/site-packages/numpy/linalg/_linalg.py", line 888, in cholesky
    r = gufunc(a, signature=signature)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/.venv/lib/python3.12/site-packages/numpy/linalg/_linalg.py", line 148, in _raise_linalgerror_nonposdef
    raise LinAlgError("Matrix is not positive definite")
numpy.linalg.LinAlgError: Matrix is not positive definite

I see it also on the BananaPi F3:

(armbian) armbian@bananapif3:~$ python3
Python 3.12.3 (main, Mar  3 2026, 12:15:18) [GCC 13.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.random.seed(1)
>>> shape = (50, 50)
>>> t = [-1, -2]
>>> a = np.random.randn(*shape)
>>> a = np.matmul(a.transpose(t).conj(), a)
>>> a = np.asarray(a, dtype=np.float64)
>>> c = np.linalg.cholesky(a, upper=True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/armbian/.venv/lib/python3.12/site-packages/numpy/linalg/_linalg.py", line 888, in cholesky
    r = gufunc(a, signature=signature)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/armbian/.venv/lib/python3.12/site-packages/numpy/linalg/_linalg.py", line 148, in _raise_linalgerror_nonposdef
    raise LinAlgError("Matrix is not positive definite")
numpy.linalg.LinAlgError: Matrix is not positive definite
>>> np.__version__
'2.5.0.dev0+git20260512.d7735ce'

That linalg issue could still be in NumPy (due to differences in the random numbers generated or the np.matmul precision), or in OpenBLAS. The test is sensitive to numerical precision issues, see gh-8385 and gh-13658 for much older occurrences of this same error.

I ran this on some new RISC-V hardware,SpacemiT K3. I ran this in a Fedora 43 container.

That was my setup:

$ python3.12 -m venv numpy_venv_3.12
$ source numpy_venv_3.12/bin/activate
$ pip install --upgrade pip
$ pip install meson pytest hypothesis
$ unzip cp312-manylinux_riscv64-.zip -d numpy_artifact
$ pip install numpy_artifact/*.whl

Ran NumPy's bundled test suite; it failed with a bunch of tests

$ { time python3.12 -c 'import numpy; numpy.test()'; } 
NumPy version 2.5.0.dev0+git20260512.d7735ce
NumPy CPU features:   RVV*
[...]
../numpy_venv_3.12/lib64/python3.12/site-packages/numpy/linalg/_linalg.py:888: in cholesky
    r = gufunc(a, signature=signature)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        a          = array([[[10.73433257, -4.87682749, -3.1554569 , -4.59975348,
          2.10303633, -1.24351932,  3.03068619, -4.149521...      ,  0.        ,
          0.        ,  0.        ,  0.        ,  0.        ,
         -7.0796601 ,  9.4854158 ]]])
        gufunc     = <ufunc 'cholesky_up'>
        result_t   = <class 'numpy.float64'>
        signature  = 'd->d'
        t          = <class 'numpy.float64'>
        upper      = True
        wrap       = <built-in method __array_wrap__ of numpy.ndarray object at 0x3f97825fb0>
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

err = 'invalid value', flag = 8

    def _raise_linalgerror_nonposdef(err, flag):
>       raise LinAlgError("Matrix is not positive definite")
E       numpy.linalg.LinAlgError: Matrix is not positive definite

err        = 'invalid value'
flag       = 8

../numpy_venv_3.12/lib64/python3.12/site-packages/numpy/linalg/_linalg.py:148: LinAlgError
=============================== warnings summary ===============================
_core/tests/test_records.py::TestFromrecords::test_recarray_conflict_fields
  /root/numpy_venv_3.12/lib64/python3.12/site-packages/numpy/_core/records.py:462: DeprecationWarning: Setting the shape on a NumPy array has been deprecated in NumPy 2.5.
  As an alternative, you can create a new view using np.reshape (with copy=False if needed).
    ret = object.__setattr__(self, attr, val)

lib/tests/test_function_base.py::TestQuantile::test_nan_err[err1-O-weights0]
lib/tests/test_function_base.py::TestQuantile::test_nan_err[err1-O-weights1]
  /root/numpy_venv_3.12/lib64/python3.12/site-packages/numpy/lib/_function_base_impl.py:4505: RuntimeWarning: invalid value encountered in less
    if np.any(weights < 0):

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=========================== short test summary info ============================
FAILED linalg/tests/test_linalg.py::TestCholesky::test_basic_property[False-float32-shape3]
FAILED linalg/tests/test_linalg.py::TestCholesky::test_basic_property[False-float32-shape4]
FAILED linalg/tests/test_linalg.py::TestCholesky::test_basic_property[False-float64-shape3]
FAILED linalg/tests/test_linalg.py::TestCholesky::test_basic_property[False-float64-shape4]
FAILED linalg/tests/test_linalg.py::TestCholesky::test_basic_property[True-float32-shape3]
FAILED linalg/tests/test_linalg.py::TestCholesky::test_basic_property[True-float32-shape4]
FAILED linalg/tests/test_linalg.py::TestCholesky::test_basic_property[True-float64-shape3]
FAILED linalg/tests/test_linalg.py::TestCholesky::test_basic_property[True-float64-shape4]
ERROR _core/tests/test_mem_policy.py::test_set_policy - subprocess.CalledProc...
ERROR _core/tests/test_mem_policy.py::test_default_policy_singleton - subproc...
ERROR _core/tests/test_mem_policy.py::test_policy_propagation - subprocess.Ca...
ERROR _core/tests/test_mem_policy.py::test_context_locality - subprocess.Call...
ERROR _core/tests/test_mem_policy.py::test_thread_locality - subprocess.Calle...
ERROR _core/tests/test_mem_policy.py::test_switch_owner[0] - subprocess.Calle...
ERROR _core/tests/test_mem_policy.py::test_switch_owner[1] - subprocess.Calle...
ERROR _core/tests/test_mem_policy.py::test_switch_owner[None] - subprocess.Ca...
ERROR _core/tests/test_mem_policy.py::test_owner_is_base - subprocess.CalledP...
8 failed, 45267 passed, 1792 skipped, 2845 deselected, 31 xfailed, 2 xpassed, 3 warnings, 9 errors in 478.95s (0:07:58)

I've attached the full log here (232KB):
numpy_test_results_on_k3.log

Thanks @kashyapc! Could you post a traceback on one of the test_mem_policy.py ones (I assume they're all the same)? EDIT: the link to the log file doesn't work for me

Reproduced on my BananaPi F3 (SpaceMiT K1, rv64gc). Walking back my earlier "0 accuracy failures" comment, that was based on a different setup (system numpy with system OpenBLAS), not this PR's wheel. Sorry for the noise.

Running rgommers' minimal repro with the wheel from runs/25744398974:

numpy 2.5.0.dev0+git20260512.d7735ce
BLAS: scipy-openblas 0.3.33.0.0 (DYNAMIC_ARCH, riscv64_generic)

a is symmetric: True
min eigenvalue = -2.779433e+00      # mathematically should be >= 0
max eigenvalue = 1.949916e+02
LinAlgError: Matrix is not positive definite

Same machine, swapping in system numpy 2.4.3 with system OpenBLAS 0.3.29 (libopenblas0 0.3.29+ds-3, Debian 13 trixie): cholesky succeeds.

So cholesky is correctly rejecting a matrix that the matmul step computed wrong. The bug is upstream of cholesky, in OpenBLAS GEMM/SYRK on K1. Either a regression between 0.3.29 and 0.3.33, or a packaging difference between Debian's build and scipy-openblas's. Both targets compile to riscv64_generic, so it's not vector-related.

Lines up with what eshattow showed: Pine64 Star64 (StarFive JH-7110, SiFive U74) doesn't reproduce this, while threexc's Muse Pi Pro and my F3 (both K1) do. K1-specific so far.

I can bisect 0.3.30 through 0.3.32 and open an upstream OpenBLAS issue once I've narrowed the range. Will report back.

Bisected on the BananaPi F3 (LD_PRELOAD swap of libscipy_openblas64_*.so, same numpy wheel throughout):

Clean break between OpenBLAS 0.3.31 and 0.3.33 (no 0.3.32 in scipy-openblas64 PyPI). The K1 has no RVV, so DYNAMIC_ARCH dispatches to riscv64_generic kernels. 250 commits between the v0.3.31 and v0.3.33 tags.

Filing an upstream OpenBLAS issue with the repro and this bisect range. Will link back here.

Upstream issue filed: OpenMathLib/OpenBLAS#5811

Bisected on the BananaPi F3 (LD_PRELOAD swap of libscipy_openblas64_*.so, same numpy wheel throughout):
scipy-openblas64 embedded OpenBLAS min eigval cholesky
0.3.31.22.1 0.3.31.dev +1.405e-02 OK
0.3.31.126.1 0.3.31.dev +1.405e-02 OK
0.3.31.126.5 0.3.31.dev +1.405e-02 OK
0.3.31.159.0 0.3.31.159.0 +1.405e-02 OK
0.3.31.188.0 0.3.31.188.0 +1.405e-02 OK
0.3.33.0.0 0.3.33.0.0 -2.779e+00 LinAlgError

Clean break between OpenBLAS 0.3.31 and 0.3.33 (no 0.3.32 in scipy-openblas64 PyPI). The K1 has no RVV, so DYNAMIC_ARCH dispatches to riscv64_generic kernels. 250 commits between the v0.3.31 and v0.3.33 tags.

Filing an upstream OpenBLAS issue with the repro and this bisect range. Will link back here.

What do you mean by "no RVV"? The K1 is advertised as RVA22 + RVV 1.0.

Getting there, down to 34 minutes in this job after including test selection changes in gh-31420:

Build and test phase are then taking about as long. We may indeed want to consider changing this job to use ccache next, that should bring it down to 20-25 minutes which is reasonable.

Of course if 34 minutes is tolerance for a little while, we can also decide to just merge this as is now, so we get some hands-on experience. If new hardware is coming in the summer that's 2-3x faster, that will address the performance and it'll be fine with no further changes needed.

Currently the other two slowest jobs:

• Linux Qemu tests / riscv64: varies between 21 and 31 min. in logs I checked
• Linux SIMD tests / old gcc: 24 min.

We can turn off the Qemu riscv64 test if this new one is stable, so the overall extra runtime before that new hardware arrives only takes a minor hit.

There's something to say for any of these options:

1. Merge as is, wait for new hardware
2. Merge as is, immediately work on a follow-up PR to move the job away from wheels.yml and use ccache
3. Don't merge because 34 minutes is too slow, work on a job with ccache now

@andyfaff any preferences?
@luhenry any preferences taking into account the load on your infra?

I think options 2 or 3 are preferable .

@luhenry any preferences taking into account the load on your infra?

Looking at the data, there are around ~300-350 wheels.yml workflow runs / 30day. At 30min per run, the load on the infra infrastructure would be negligible. (we are currently at the 300-500 runs/day and the infra is far from peak capacity).

There's something to say for any of these options:

1. Merge as is, wait for new hardware
2. Merge as is, immediately work on a follow-up PR to move the job away from wheels.yml and use ccache
3. Don't merge because 34 minutes is too slow, work on a job with ccache now

To me, the main question is what you, as maintainers, are happy to take on. We should make the load as small as possible for everyday's maintainers and contributors, and riscv64 shouldn't be a pain point. I don't think we are in such a rush (we all want to see it integrated!) that we can't work on some workarounds before merging (split out of wheels.yml? ccache? all of the above?).

Integrating ccache would in my experience be quite a focused change with:

1. use https://github.com/hendrikmuhs/ccache-action to manage the cache save/restore to GHA Cache
2. setup ccache inside the container: CIBW_BEFORE_ALL_LINUX: 'yum install -y ccache' should do?
3. setup the ccache environment variables:

CIBW_ENVIRONMENT_LINUX: >-
  CC="ccache gcc"
  CXX="ccache g++"
  CCACHE_DIR=/host/tmp/ccache
  CCACHE_COMPILERCHECK=content
  CCACHE_BASEDIR=/project

4. profit?

It's worth trying out, I can give it a try today or tomorrow.

Okay, option (3) it is:)

1. use https://github.com/hendrikmuhs/ccache-action to manage the cache save/restore to GHA Cache

Interesting, I didn't see that before. We normally use regular https://github.com/actions/cache and set CCACHE_DIR, which this action seems to be doing under the hood. We only have one usage in NumPy right now, here in macos.yml. And that looks outdated/suboptimal, we want to run the job on merges to main and only store caches there I think.

In SciPy we recently overhauled this and made sure the behavior was optimal for cache hits and storage consumption. It uses a generic composite action (this one) and then there's still two job steps per job that uses ccache, plus setting CCACHE_DIR:

https://github.com/scipy/scipy/blob/930870e57a31074e8f80e5f35a30d74bd1386563/.github/workflows/linux.yml#L162-L166

https://github.com/scipy/scipy/blob/930870e57a31074e8f80e5f35a30d74bd1386563/.github/workflows/linux.yml#L205-L209

If https://github.com/hendrikmuhs/ccache-action works the same, I think we can use that.

2. setup ccache inside the container: CIBW_BEFORE_ALL_LINUX: 'yum install -y ccache' should do?

Agreed.

3. setup the ccache environment variables:

That isn't even necessary (except for CCACHE_DIR probably). As long as ccache is on the PATH, meson will use it automatically.

It's worth trying out, I can give it a try today or tomorrow.

That would be great, thanks. I'd suggest creating a new linux_riscv64.yml, with contents stripped down from wheels.yml plus the extra push: branches: main trigger.

Interesting, I didn't see that before. We normally use regular https://github.com/actions/cache and set CCACHE_DIR, which this action seems to be doing under the hood. We only have one usage in NumPy right now, here in macos.yml. And that looks outdated/suboptimal, we want to run the job on merges to main and only store caches there I think.

The main advantage I saw to this https://github.com/hendrikmuhs/ccache-action is that you can selectively save. You can save on workflows to main, and never save on workflows on pull requests for example. This way you have a good "hot" cache but don't pollute it with many from pull requests.

That isn't even necessary (except for CCACHE_DIR probably). As long as ccache is on the PATH, meson will use it automatically.

We would want to mount the ccache dir from the host and so need to point the ccache inside the container to that directory. So if meson automatically picks it up it's a lot easier indeed.

You're right, I had this completely wrong. The K1 X60 cores do implement RVV 1.0 (the ISA string in /proc/cpuinfo shows _v_ along with the Zve/Zvfh/Zvkt extensions, VLEN=256).

Re-checked with OPENBLAS_VERBOSE=2 on the F3:

=== scipy-openblas64==0.3.31.188.0 ===
Core: riscv64_zvl256b

=== scipy-openblas64==0.3.33.0.0 ===
Core: riscv64_zvl256b

So OpenBLAS DYNAMIC_ARCH dispatches to the riscv64_zvl256b kernels on the K1, not generic. The bisect range is unchanged (0.3.31.188.0 OK, 0.3.33.0.0 broken), but the regression is somewhere in the vector kernels (or something they call), not the scalar generic path.

That actually fits the earlier observation that the Pine64 Star64 (SiFive U74, no RVV, dispatches to riscv64_generic) does not reproduce. Same regression range, different kernel path, different result, and the failing path is the vectorized one.

Updating OpenMathLib/OpenBLAS#5811 with the corrected dispatch info.

Thanks @kashyapc! Could you post a traceback on one of the test_mem_policy.py ones (I assume they're all the same)? EDIT: the link to the log file doesn't work for me

@rgommers Hey, can you please try again the link in my comment here. (I fixed the formatting, but the same link worked for me.)

As for traceback, I'll let you know once I get it.

The 8 TestCholesky::test_basic_property failures in @kashyapc's K3 run are a known OpenBLAS regression, not a numpy or wheel defect. I filed it as OpenMathLib/OpenBLAS#5811: DGEMM/SYRK regressed between scipy-openblas64 0.3.31 and 0.3.33 on the riscv64 vector kernel path, so A.T @ A comes back with a negative eigenvalue and Cholesky correctly rejects the result as non-PSD. It reproduces on the K1 (zvl256b dispatch) and the K3 signature matches exactly. I bisected it to OpenBLAS git history between v0.3.31 and v0.3.33; the repro and bisect range are in that issue.

The 9 test_mem_policy.py errors look like a different thing. They're subprocess.CalledProcessError on the in-subprocess library load, which is the path @rgommers is already digging into, not the linalg math.

So nothing in this PR's diff is implicated by either bucket: the build is clean and the job is green. The Cholesky path will stay red on K1/K3 hardware until the OpenBLAS fix lands, and I'm tracking that at #5811.

The 9 test_mem_policy.py errors look like a different thing. They're subprocess.CalledProcessError on the in-subprocess library load, which is the path @rgommers is already digging into, not the linalg math.

The relevant part of the traceback is this:

>           raise CalledProcessError(retcode, cmd)
E           subprocess.CalledProcessError: Command '['meson', 'setup', '--vsenv', '..', '--native-file=/tmp/pytest-of-root/pytest-0/test_set_policy0/mem_policy/.mesonpy-native-file.ini']' returned non-zero exit status 1.

cmd        = ['meson', 'setup', '--vsenv', '..', '--native-file=/tmp/pytest-of-root/pytest-0/test_set_policy0/mem_policy/.mesonpy-native-file.ini']
kwargs     = {'cwd': PosixPath('/tmp/pytest-of-root/pytest-0/test_set_policy0/mem_policy/build')}
popenargs  = (['meson', 'setup', '--vsenv', '..', '--native-file=/tmp/pytest-of-root/pytest-0/test_set_policy0/mem_policy/.mesonpy-native-file.ini'],)
retcode    = 1

/usr/lib64/python3.12/subprocess.py:413: CalledProcessError
---------------------------- Captured stdout setup -----------------------------
The Meson build system
Version: 1.11.1
Source dir: /tmp/pytest-of-root/pytest-0/test_set_policy0/mem_policy
Build dir: /tmp/pytest-of-root/pytest-0/test_set_policy0/mem_policy/build
Build type: native build
Project name: foo
Project version: undefined
C compiler for the host machine: cc (gcc 15.2.1 "cc (GCC) 15.2.1 20260123 (Red Hat 15.2.1-7)")
C linker for the host machine: cc ld.bfd 2.45.1-4
Host machine cpu family: riscv64
Host machine cpu: riscv64
Program python found: YES (/root/numpy_venv_3.12/bin/python3.12)
Found pkg-config: YES (/usr/bin/pkg-config) 2.3.0
Run-time dependency python found: NO  (tried pkgconfig and sysconfig)

../meson.build:3:3: ERROR: Python dependency not found

A full log can be found at /tmp/pytest-of-root/pytest-0/test_set_policy0/mem_policy/build/meson-logs/meson-log.txt

At first sight that looks like an incomplete Python install, e.g. python-config is missing, or python-dev which provides Python.h (my guess is the latter). I don't think it requires a change in numpy.

Confirming @rgommers's read with a data point from native riscv64. On the BPI-F3 (SpaceMiT K1, Armbian trixie), the same cp312 wheel from this PR's CI (2.5.0.dev0+git20260512.d7735ce) runs test_mem_policy.py clean: 9 passed, 1 skipped in 18.7s, native, not QEMU. That is the exact set of tests that errored on @kashyapc's K3 run.

The only difference is the Python: the venv here uses a standalone 3.12 that ships Python.h, so meson resolves the python dependency. The Fedora container python3.12 had no devel package, which is what the meson setup failure was. So this is environment, not riscv64 or the wheel, and nothing for numpy to change.

Submitted a PR for adding ccache support: #31474

Given feedback at 1 and 2, I suggest we add a linux_riscv64.yml file and extract that wheel building steps into it. @gounthar want to have a look, or should I do it? Happy either way. If you want to do it, simply copy/paste what I've done at #31474, all good for me.

Go for it, Ludovic. You've already got the working version in #31474, so it makes sense for you to extract it rather than me copying it over. Once linux_riscv64.yml is up I'll run it against the Banana Pi F3 (native K1) and review. Thanks for offering either way.

Thank you for being understanding, https://github.com/numpy/numpy/blob/main/.github/workflows/linux-ibm.yml is an example to base the file off.

gh-31488 was merged and incorporated most of this PR, so I'll close this now. Thanks all!