Skip to content

CI: Windows on ARM experiment: SciPy, stack and faulthandler (not for merging) - #1007

Closed
bashtage wants to merge 3 commits into
mainfrom
a90-winarm-scipy
Closed

bashtage wants to merge 3 commits into
mainfrom
a90-winarm-scipy

Conversation

@bashtage

@bashtage bashtage commented Nov 4, 2026 •

Copy link
Copy Markdown
Owner

Experimental, not for merging. One of four experiments on the access violations in numpy.linalg on Windows on ARM that #1003 found. With OPENBLAS_NUM_THREADS=1, inv and solve of small positive definite matrices crash in numpy 2.5.4's OpenBLAS, and numpy alone reproduces it. Each experiment has its own pull request, so that they run at the same time and each can change without restarting the others: #1003 (arch's tests), #1004 (the calls and threads), #1005 (the numpy versions), #1006 (the OpenBLAS routines) and #1007 (SciPy, the stack and faulthandler).

Like #1003, this removes the Azure pipeline (azure-pipelines.yml) and the wheel builds (cron-test.yml), so that no tests run here. arch is not built: the job installs only numpy (and SciPy where needed) from wheels.

What this one runs

  • SciPy's OpenBLAS, with 1 and 4 threads. SciPy bundles its own copy. The calls are scipy.linalg's inv, solve, lu_factor, lu_solve and cho_factor, and the LAPACK and BLAS wrappers dgetrf, dgetrs, dgesv, dgetri, dpotrf, dtrsm, dlaswp, dgemm and dgemv.
  • The stack: numpy's inv, solve and det in a thread with a 64 MB stack and one with a 256 KB stack, with 1 thread. If the crash is a stack overflow, the large stack stops it and the small one makes it happen at smaller sizes.
  • faulthandler: the same calls without -X faulthandler, which every earlier run used.

numpy 2.5.4, SciPy 1.18.1, Python 3.14.

How it works

ci/github-actions/winarm_blas.py, the same in all four pull requests, makes every call in a child process. A child walks through the sizes (1 to 64, then every 8th to 160) and prints start CALL N before each call. When it crashes or hangs (60 s), the size is recorded and a new child goes on with the next size. Each size that crashed is then tried alone, in a new process. So a map takes minutes rather than a process for every size, which is what made the size map in #1003 so slow. The matrices are built without BLAS (einsum), so the call under test is the only BLAS call.

The job passes when the map is complete, whether or not anything crashed. The results are in the job summary, in the log as they come, and in the report.md artifact. Each report starts with the builds and what OpenBLAS reports at run time (kernel, threads, configuration) and the CPU.

Checked locally (x64)

  • actionlint and check-jsonschema (GitHub workflow schema) report no problems. The script is clean under ruff, black, isort, flake8, bandit and pylint.
  • Every command of the four workflows runs to the end on x64, in under a minute in total, with no crashes. selftest crashes at size 5 and is found, the walk goes on, and the crash is found again alone.
  • The ctypes calls into numpy's OpenBLAS (ILP64 on x64) and SciPy's (LP64) check their results without BLAS.

🤖 Generated with Claude Code

The experiments on the numpy.linalg crashes on Windows on ARM (see #1003)
run in their own pull requests. Remove the Azure pipeline and the wheel
builds, the workflows that run the tests, so that these pull requests do
not use the CI queues. Not for merging.
Maps the sizes at which numpy.linalg calls, SciPy's LAPACK and BLAS, and
the LAPACK and BLAS routines of numpy's OpenBLAS (called directly with
ctypes) crash. A child process walks through the sizes, and after a crash
or a hang a new child continues with the next size, so a map takes
minutes rather than a process for every size. Each size that crashed is
then tried alone. Options set the OpenBLAS threads in the environment or
at run time, the stack of the thread that makes the calls, and
faulthandler. Not for merging.
Maps the crashes of scipy.linalg and its LAPACK and BLAS wrappers, which
use SciPy's own OpenBLAS, and of numpy.linalg in a thread with a 64 MB
and a 256 KB stack, and without faulthandler. Not for merging.
bashtage added a commit that referenced this pull request Nov 4, 2026
The map ran a process for every call and size and took more than an hour,
so the job would time out. #1004 to #1007 map the crashes in minutes,
each in its own pull request. Not for merging.

result["scipy"] = scipy.__version__
result["scipy openblas"] = runtime("scipy")
except ImportError:
bashtage added a commit that referenced this pull request Nov 4, 2026
numpy.linalg.inv and solve crash in the OpenBLAS of numpy's win_arm64
wheels only when OpenBLAS starts with 1 thread (see #1004 to #1007), so
do not set OPENBLAS_NUM_THREADS, OMP_NUM_THREADS or MKL_NUM_THREADS, and
fail if any thread limit is set or OpenBLAS reports fewer than 2 threads.
Run the tests that the other CI runs with pytest-xdist, then the slow
tests, on Python 3.13 and 3.14 with numba and 3.15 without it (no
win_arm64 wheels). statsmodels has no win_arm64 wheels for 3.12. Remove
the crash isolation, which moved to #1004 to #1007.
@bashtage bashtage closed this Nov 5, 2026
@bashtage
bashtage deleted the a90-winarm-scipy branch November 11, 2026 13:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants