BUG: linalg.inv crashing on windows arm64

Describe the issue:

The linalg.inv function is showing consistent crashes with the numpy 2.3.1 and 2.3.2 wheels for win_arm64 (Windows on Arm, WoA) installed from PyPI with Windows fatal exception: access violation.

The traceback in SciPy CI points at this line:

numpy/numpy/linalg/_linalg.py

Line 669 in c6b5258

ainv = _umath_linalg.inv(a, signature=signature)

Reproduce the code example:

Here is a reproducer:


@pytest.mark.parametrize('dtype', [np.complex64, np.complex128])
@pytest.mark.parametrize('mtype', ['symm', 'herm'])  # matrix type
def test_numpy_inv_crash_win_arm64(dtype, mtype):
    rng = np.random.default_rng(1723059677121834)
    n = 20
    A = rng.random((n, n)) + rng.random((n, n))*1j
    if np.issubdtype(dtype, np.floating):
        A = A.real
    A = A.astype(dtype)
    A = A + A.T if mtype == 'symm' else A + A.conj().T
    ref = np.linalg.inv(A)


It's crashing for all combinations of `complex64`/`complex128` and symmetric/hermitian matrices. `float32`/`float64` dtypes are working fine.

Error message:

This ran in SciPy CI with `pytest-xdist` with 2 workers.

Windows fatal exception: access violation
...

 Current thread 0x00001f34 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.12.10\arm64\Lib\site-packages\numpy\linalg\_linalg.py", line 669 in inv
  File "C:\a\scipy\scipy\build-install\Lib\site-packages\scipy\linalg\tests\test_lapack.py", line 3527 in test_numpy_inv_crash_win_arm64
  File "C:\hostedtoolcache\windows\Python\3.12.10\arm64\Lib\site-packages\_pytest\python.py", line 157 in pytest_pyfunc_call
  File "C:\hostedtoolcache\windows\Python\3.12.10\arm64\Lib\site-packages\pluggy\_callers.py", line 121 in _multicall
...

Python and NumPy Versions:

Python 3.11/3.12
NumPy 2.3.1/2.3.2

Runtime Environment:

(can't easily provide this, I don't have local hardware)

Context for the issue:

The SciPy tests can be skipped for now, so it's not blocking.

@rgommers

@rgommers - do these look like OpenBLAS errors do you think? Would it be worth testing with lapack-liite? @MugundanMCW @khmyznikov - do you have any interest in testing here?

Could be, but could equally well be inside numpy. Ideally this is reproduced locally and then a stack trace generated. Or even in CI, since it reproducibly crashes there and it seems like the "Windows way" of getting a stack trace is to modify the code - once that's done, may as well push it to CI.

Switching to lapack_lite by passing -Csetup-args=blas=none to a build is a reasonable proxy, but doesn't get you any more info than whether it's more or less likely to be in OpenBLAS or in NumPy.

Currently giving it a go with Arm's blas to see what happens.

Well, I guess that answers the question as to whether this is numpy or openblas:

(venv) E:\test>pytest repro.py
================================================= test session starts =================================================
platform win32 -- Python 3.13.5, pytest-8.4.1, pluggy-1.6.0
rootdir: E:\test
collected 4 items

repro.py ....                                                                                                    [100%]

================================================== 4 passed in 0.08s ==================================================

>>> np.show_config()
Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: C:/Program\ Files/Arm\ Performance\ Libraries/armpl_25.07/include/
    lib directory: C:/Program\ Files/Arm\ Performance\ Libraries/armpl_25.07/lib/
    name: armpl_lp64
    openblas configuration: unknown
    pc file directory: C:/Users/andrew/pkgconf
    version: '25.07'
  lapack:
    detection method: pkgconfig
    found: true
    include directory: C:/Program\ Files/Arm\ Performance\ Libraries/armpl_25.07/include/
    lib directory: C:/Program\ Files/Arm\ Performance\ Libraries/armpl_25.07/lib/
    name: armpl_lp64
    openblas configuration: unknown
    pc file directory: C:/Users/andrew/pkgconf
    version: '25.07'
Compilers:
  c:
    commands: clang-cl.exe
    linker: lld-link
    name: clang-cl
    version: 20.1.8
  c++:
    commands: clang-cl.exe
    linker: lld-link
    name: clang-cl
    version: 20.1.8
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 3.1.2
Machine Information:
  build:
    cpu: aarch64
    endian: little
    family: aarch64
    system: windows
  host:
    cpu: aarch64
    endian: little
    family: aarch64
    system: windows
Python Information:
  path: E:\numpy\venv\Scripts\python.exe
  version: '3.13'
SIMD Extensions:
  baseline:
  - NEON
  - NEON_FP16
  - NEON_VFPV4
  - ASIMD
  not found:
  - ASIMDHP
  - ASIMDFHM

.....it also passes with the pypi wheel. so either the reproducer is bad or something else is up.

>>> np.show_config()
Build Dependencies:
  blas:
    detection method: pkgconfig
    found: true
    include directory: C:/Users/runneradmin/AppData/Local/Temp/cibw-run-xil685ik/cp313-win_arm64/build/venv/Lib/site-packages/scipy_openblas32/include
    lib directory: C:/Users/runneradmin/AppData/Local/Temp/cibw-run-xil685ik/cp313-win_arm64/build/venv/Lib/site-packages/scipy_openblas32/lib
    name: scipy-openblas
    openblas configuration: OpenBLAS 0.3.30 NO_AFFINITY ARMV8 MAX_THREADS=4
    pc file directory: C:/a/numpy/numpy/.openblas
    version: 0.3.30
  lapack:
    detection method: pkgconfig
    found: true
    include directory: C:/Users/runneradmin/AppData/Local/Temp/cibw-run-xil685ik/cp313-win_arm64/build/venv/Lib/site-packages/scipy_openblas32/include
    lib directory: C:/Users/runneradmin/AppData/Local/Temp/cibw-run-xil685ik/cp313-win_arm64/build/venv/Lib/site-packages/scipy_openblas32/lib
    name: scipy-openblas
    openblas configuration: OpenBLAS 0.3.30 NO_AFFINITY ARMV8 MAX_THREADS=4
    pc file directory: C:/a/numpy/numpy/.openblas
    version: 0.3.30
Compilers:
  c:
    commands: clang-cl
    linker: lld-link
    name: clang-cl
    version: 20.1.6
  c++:
    commands: clang-cl
    linker: lld-link
    name: clang-cl
    version: 20.1.6
  cython:
    commands: cython
    linker: cython
    name: cython
    version: 3.1.2
Machine Information:
  build:
    cpu: aarch64
    endian: little
    family: aarch64
    system: windows
  host:
    cpu: aarch64
    endian: little
    family: aarch64
    system: windows
Python Information:
  path: C:\Users\runneradmin\AppData\Local\Temp\build-env-p02s09hg\Scripts\python.exe
  version: '3.13'
SIMD Extensions:
  baseline:
  - NEON
  - NEON_FP16
  - NEON_VFPV4
  - ASIMD
  not found:
  - ASIMDHP
  - ASIMDFHM

@theAeon

Thanks for digging in @theAeon! It may well be either hardware or compiler specific. It reproducibly crashes on the GitHub windows-11-arm runners. In scipy/scipy#23133, @MugundanMCW had mixed success trying to reproduce locally, e.g see this comment and this comment.

It could be specific to an architecture-specific kernel in OpenBLAS. With export OPENBLAS_CORETYPE=<arm-family-name> you can force one, with the names given in OpenBLAS's TargetList.txt. I'm not quite sure if all of them are relevant though.

@Mousius as our resident Arm expert, do you have any other suggestions here?

Suppose theres also the uarch question-this is an oryon chip (x elite), not sure what the runner chips are (or the other local machines)

@theAeon

@theAeon - sorry - this is off-topic - but are the ARM performance libraries an option for PyPI wheels? Are they faster than OpenBLAS? Do they have a compatible license? (Probably current ARM PF license). Can we ship the libraries with the wheel, do you know?

Copying my comment from scipy/scipy#23133 (comment): It's not open source, so I doubt that this will be acceptable. The EULA seems to permit redistribution, but no source code available is a deal breaker I think.

We can always discuss of course, and it'd be nice to see what the binary size is with ArmPL (hopefully close to OpenBLAS and nothing like MKL), but previously we have not wanted to consider this. Maybe once we have variant wheels, it's more of an option to ship alternative BLAS libraries.

Suppose theres also the uarch question-this is an oryon chip (x elite), not sure what the runner chips are (or the other local machines)

So it seems (i can't be sure) that the github runners are older azure arm machines w/ ampere altra chips, which are modified NeoVerse N1. Which are fairly different in terms of simd instruction set support.

Does the CI have logs of which SIMD features are enabled? I think that may be the key here.

Does the CI have logs of which SIMD features are enabled? I think that may be the key here.

For numpy itself yes, but I already checked and those were the same (it's under SIMD Extensions: in one of the comments you posted above). For OpenBLAS you can't tell, but you can actually select which ones get dynamically selected at runtime - which is why I suggested the OPENBLAS_CORETYPE experiment above. If it's an OpenBLAS Neoverse1-specific kernel, then there's a good chance you can reproduce the failure locally by doing export OPENBLAS_CORETYPE=NEOVERSEN1 before running the reproducer.

then there's a good chance you can reproduce the failure locally by doing export OPENBLAS_CORETYPE=NEOVERSEN1 before running the reproducer.

~~Well, I can't reproduce that locally as the oryon chips (and the large majority of consumer facing arm chips that i'm aware of) don't have the SVE extension set that neoversen1 does.~~

scratch that, neoversen1 doesn't actually have sve haha

If someone forces the CI to run with OPENBLAS_CORETYPE=ARMV8, however, that would probably work.

@theAeon

@theAeon I use the Qualcomm Snapdragon X-Elite chipset for testing SciPy.

so the same chip I'm on.

Bizarre.

I'm seeing segfaults on Windows ARM64 with float64 arrays in np.linalg.inv too.

@bashtage

@bashtage can you provide more details: what does numpy.show_runtime() show?

This is what I see on the CI runner:

 [{'numpy_version': '2.4.4',
    'python': '3.13.12 (tags/v3.13.12:1cbe481, Feb  3 2026, 19:08:18) [MSC '
              'v.1944 64 bit (ARM64)]',
    'uname': uname_result(system='Windows', node='runnervmm4i28', release='11', version='10.0.26200', machine='ARM64')},
   {'simd_extensions': {'baseline': ['NEON', 'NEON_FP16', 'NEON_VFPV4', 'ASIMD'],
                        'found': [],
                        'not_found': ['ASIMDHP', 'ASIMDDP', 'ASIMDFHM']}},
   {'ignore_floating_point_errors_in_matmul': False}]

I can add that we are also seeing failures on solve on Windows-ARM64.

Maybe connected to #31618?

Can someone with an appropriate machine check if this was fixed?

@mattip

Sure @mattip

@MugundanMCW

Hi @MugundanMCW , any updates on this?

I did a rather large exprimental session tonight using CI runners and Claude. What was found:

The bug continues to exist.
The bug in inv is triggered by setting OPENBLAS_NUM_THREADS=1
Dimension of the array matters, with 23 and 31 problematic.

The full report is in the next post.

numpy.linalg crashes on Windows on ARM: findings from 2026-11-04

These experiments ran on GitHub Actions windows-11-arm runners in arch PRs #1003 to #1007. None of these PRs is for merging.

Important

In the OpenBLAS of numpy's win_arm64 wheels, np.linalg.inv and np.linalg.solve crash only for 23×23 to 31×31 matrices, and only when OpenBLAS starts with 1 thread. The routine that crashes is LAPACK's dgesv. Every numpy with win_arm64 wheels has the bug, from 2.3.0 to the current nightly. arch itself is fine. With 4 threads, or no limit, nothing crashes.

Summary

Question	Answer
Which calls?	`inv` and `solve` (1 or 3 right-hand sides). Never `det`, `cholesky`, `qr`, `eigh`, `svd`, `lstsq` or `matmul`.
Which sizes?	n = 23 to 31 only. 1–22, 32–64 and every 8th size to 160 pass.
Does the data matter?	No. `inv(np.eye(23))` crashes.
Which routine?	`dgesv`. Its parts, `dgetrf` then `dgetrs`, don't crash.
Which thread counts?	1 thread crashes. 2 threads hang at 23–31. 3 threads hang at 31. 4 threads, or none set, run fine.
Which numpy versions?	Every one with `win_arm64` wheels: 2.3.0 to the nightly 2.6.0.dev0 (2026-11-03).
SciPy?	SciPy 1.18.1 bundles its own OpenBLAS, which never crashes.
Stack overflow?	No. A 64 MB and a 256 KB thread stack give the same crashes.
faulthandler?	No. The same crashes happen without `-X faulthandler`.
Deterministic?	Yes. Every run gave the same results.

Minimal reproducer

set OPENBLAS_NUM_THREADS=1
python -c "import numpy as np; np.linalg.inv(np.eye(23))"

This exits with 0xC0000005 (Windows fatal exception: access violation). Without OPENBLAS_NUM_THREADS, or with it set to 4, it runs fine.

Setup


Runner	GitHub Actions `windows-11-arm`, image `windows-11-vs2026-arm64` 20261101.203.1
OS	Windows 11 Enterprise 10.0.26200
CPU	`ARMv8 (64-bit) Family 8 Model D49, MICROSOFT CORPORATION`. Part 0xD49 is Neoverse N2. 4 cores.
Python	3.13.17, 3.14.9 and 3.14.10, ARM64 (`win-arm64`)
numpy	PyPI `win_arm64` wheels, and the nightly from `scientific-python-nightly-wheels`
numpy 2.5.4's BLAS	`scipy-openblas` 0.3.34.dev, `NO_AFFINITY ARMV8 MAX_THREADS=24`, LP64, clang-cl 22.1.8
OpenBLAS kernels	`openblas_get_corename()` returns `ARMV8`. The build has ARMV8 kernels only.

Findings

1. Calls and sizes (#1004)

All with OPENBLAS_NUM_THREADS=1 and numpy 2.5.4.

Call	Crashes at
`inv`, identity or random positive definite matrix	23–31
`solve`, `b` with 1 or 3 columns	23–31
`det`, `cholesky`, `qr`, `eigh`, `svd`, `lstsq`	never
`matmul`, matrix and vector	never

Each size was called 3 times. Every size that crashed also crashed when run alone, in a new process.

2. The routine: `dgesv` (#1006)

These calls go straight into numpy's own OpenBLAS (numpy.libs/scipy_openblas-*.dll) with ctypes. ctypes catches the access violation and reports its address.

Routine	Crashes at	Error
`dgesv`, `nrhs` 1 or 3	23–31	`access violation reading 0x0000000000000024`, every time
`dgetrf` then `dgetrs` on the same matrix, as `dgesv` does	never
`dgetrf`, `dgetrs`, `dgetri`, `dpotrf`	never
`dtrsm`, `dlaswp`, `dgemm`, `dgemv`	never

Note

Reading address 0x24 means a member of a struct is read through a null pointer. numpy's inv and solve call gesv, and det calls only getrf, which fits the calls that crash.

3. Threads (#1004, #1006)

What matters is the number of threads when OpenBLAS starts, not only when it's called.

`OPENBLAS_NUM_THREADS` at start	`openblas_set_num_threads` before the calls	Result
1	—	💥 crashes at 23–31
1	4	✅ no crash
2	—	⏳ hangs at 23–31
3	—	⏳ hangs at 31
4	—	✅ no crash
4	1	✅ no crash
unset (4 cores)	—	✅ no crash
unset (4 cores)	1	✅ no crash

det never hangs with 2 or 3 threads. The hang map covered sizes 1–40, and a call counted as hung after 15 s.

4. numpy versions (#1005)

1 thread. Each row is the first or last release of a minor version with win_arm64 wheels.

numpy	Bundled OpenBLAS	`inv`	`solve`
2.3.0	0.3.29.dev (`MAX_THREADS=4`)	💥 23–31	✅
2.3.5	0.3.30	💥 23–31	✅
2.4.1	0.3.30	💥 23–31	✅
2.4.6	0.3.31.dev	💥 23–31	💥 23–31
2.5.0	0.3.33.dev	💥 23–31	💥 23–31
2.5.4	0.3.34.dev	💥 23–31	💥 23–31
2.6.0.dev0+git20261103.dc266dd (nightly)	0.3.34.dev	💥 23–31	💥 23–31

The whole table is on Python 3.13. 2.5.4 and the nightly were also run on Python 3.14, with the same results. The bug isn't fixed in numpy's main branch. That answers the question left open on numpy#29442 since July.

5. What makes no difference (#1003, #1007)

SciPy's OpenBLAS: SciPy 1.18.1 bundles its own copy, OpenBLAS 0.3.31.dev, also built for ARMV8. It never crashes, with 1 or 4 threads. That covers scipy.linalg.inv, solve, lu_factor, lu_solve and cho_factor, and the wrappers dgesv, dgetrf, dgetrs, dgetri, dpotrf, dtrsm, dlaswp, dgemm and dgemv. numpy 2.4.6 bundles the same OpenBLAS version, and it does crash.
The stack: running the calls in a thread with a 64 MB or a 256 KB stack gives the same crashes at 23–31.
faulthandler: the crashes are the same with and without -X faulthandler.
OPENBLAS_CORETYPE: ARMV8, NEOVERSEN1 and CORTEXA57 change nothing.

6. arch (#1003)

First run: the other CI limits OpenBLAS to 1 thread, and this run did too. With pytest-xdist, 31,565 tests passed, 457 were skipped and 14 crashed.
Where they crashed: all 14 crashes are in the unit root code. 12 are inv in DynamicOLS._cov, of 23×23 to 30×30 matrices. 2 are solve in the ADF and DFGLS lag search, on their first 23×23 system.
Alone: each of the 14 tests also crashes when run on its own. The matrices saved before each call crash numpy alone.
The rest of arch: it builds as a win_arm64 wheel, its compiled extensions load, and the univariate (13,198) and multiple comparison (771) tests pass.

What this means for arch on Windows on ARM

Tip

Don't limit the BLAS threads on Windows on ARM. #1003 now runs the whole suite with no thread limit. It fails if any *_NUM_THREADS variable is set, or if numpy's OpenBLAS reports fewer than 2 threads. The jobs cover Python 3.13 and 3.14 with numba, and 3.15 without it. Python 3.12 can't be tested, because statsmodels has win_arm64 wheels only for 3.13 and later.

Whole suite without a thread limit (run 41928225500): passes. The 14 tests that crashed with 1 thread now pass.

Python	numba	Tests	Slow tests
3.13.17	0.68.1	31,579 passed, 457 skipped	784 passed, 1 skipped
3.14.9	0.68.1	31,579 passed, 457 skipped	784 passed, 1 skipped
3.15.1	none (no `win_arm64` wheels)	30,480 passed, 462 skipped	784 passed, 1 skipped

#1003 is now "CI: Test on Windows on ARM". It adds this job and release notes. The Azure pipeline and the wheel builds are restored, so the change against main is only the workflow and the release notes.

How the experiments ran

ci/github-actions/winarm_blas.py is the same script in all four experiment PRs. It runs every call in a child process:

Walk: a child calls each size in turn and prints start CALL N before each call.
Restart: when a child crashes, or hangs past the timeout, the size is recorded and a new child goes on with the next size.
Confirm: every size that crashed is then tried alone, in a new process.
No other BLAS calls: the test matrices are built with einsum, which doesn't call BLAS. So the call under test is the only BLAS call.
Self-test: a call named selftest crashes at size 5 on purpose, which shows that crashes are found on ARM.

A job takes 30–60 seconds. The first size map in #1003 started a process for every call and size, and allowed 900 s per hang. It hit the 2-thread hangs, ran for over an hour, and was cancelled at the 2-hour job limit.

PRs, runs and saved data

PR	Experiment	Runs
#1003	arch's tests and crash isolation; now the Windows on ARM CI job	41890241078, 41900587692, 41900945622, 41901664770, 41928225500
#1004	numpy.linalg calls, by number of threads	41919056366, 41921007879
#1005	numpy versions	41919060481
#1006	OpenBLAS routines, `openblas_set_num_threads`	41919064548, 41919898415
#1007	SciPy's OpenBLAS, the stack, faulthandler	41919073449

All job logs, results tables, saved crash matrices and copies of the numpy issues are in reports/numpy-winarm64-linalg/ in the release folder. A draft comment for numpy#29442 is ready, and nothing has been posted to numpy yet.

Open questions

Why does dgesv crash when dgetrf followed by dgetrs doesn't? Inside OpenBLAS, dgesv is those two calls.
Why only sizes 23 to 31, and why only when OpenBLAS starts with 1 thread?
Why does SciPy's copy of OpenBLAS work? numpy 2.4.6 bundles the same version, 0.3.31.dev, and it crashes.
Do the hangs with 2 and 3 threads belong in the same numpy issue, or in an OpenBLAS issue?

It also appears that using the latest SciPy may be a viable workaround for projects that already depend on SciPy.

One difference between SciPy and NumPy is that SciPy uses 32-bit interfaces (scipy-openblas32), NumPy uses 64-bit interfaces (scipy-openblas64).

Navigation Menu

Repository navigation

BUG: `linalg.inv` crashing on windows arm64 #29442

Describe the issue:

Reproduce the code example:

Error message:

Python and NumPy Versions:

Runtime Environment:

Context for the issue:

5 remaining items

numpy.linalg crashes on Windows on ARM: findings from 2026-11-04

Summary

Minimal reproducer

Setup

Findings

1. Calls and sizes (#1004)

2. The routine: `dgesv` (#1006)

3. Threads (#1004, #1006)

4. numpy versions (#1005)

5. What makes no difference (#1003, #1007)

6. arch (#1003)

What this means for arch on Windows on ARM

How the experiments ran

Open questions

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

BUG: linalg.inv crashing on windows arm64 #29442

Description

Describe the issue:

Reproduce the code example:

Error message:

Python and NumPy Versions:

Runtime Environment:

Context for the issue:

Activity

matthew-brett commented on Jul 26, 2025

rgommers commented on Jul 26, 2025

theAeon commented on Jul 28, 2025

theAeon commented on Jul 28, 2025

theAeon commented on Jul 28, 2025

rgommers commented on Jul 28, 2025

theAeon commented on Jul 28, 2025

matthew-brett commented on Jul 28, 2025

rgommers commented on Jul 28, 2025

theAeon commented on Jul 28, 2025

rgommers commented on Jul 28, 2025

theAeon commented on Jul 28, 2025

theAeon commented on Jul 28, 2025

5 remaining items

MugundanMCW commented on Jul 29, 2025

theAeon commented on Jul 29, 2025

bashtage commented on Mar 30, 2026

mattip commented on Mar 30, 2026

bashtage commented on Mar 30, 2026

bashtage commented on Apr 1, 2026

mattip commented on Jun 13, 2026

mattip commented on Jul 5, 2026

MugundanMCW commented on Jul 5, 2026

chinazhangchao commented on Sep 9, 2026

bashtage commented on Nov 4, 2026

bashtage commented on Nov 4, 2026

numpy.linalg crashes on Windows on ARM: findings from 2026-11-04

Summary

Minimal reproducer

Setup

Findings

1. Calls and sizes (#1004)

2. The routine: dgesv (#1006)

3. Threads (#1004, #1006)

4. numpy versions (#1005)

5. What makes no difference (#1003, #1007)

6. arch (#1003)

What this means for arch on Windows on ARM

How the experiments ran

Open questions

bashtage commented on Nov 4, 2026

mattip commented on Nov 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

BUG: `linalg.inv` crashing on windows arm64 #29442

2. The routine: `dgesv` (#1006)