Repository navigation
CI: Test on Windows on ARM - #1003
Merged
Merged
Conversation
Build the wheel on the windows-11-arm runner with Python 3.14 and test the installed package. The dependencies, numba included, are installed from win_arm64 wheels only. The compiled extensions are imported directly, since arch falls back to the pure Python code when they cannot be imported.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #1003 +/- ##
=======================================
Coverage 99.44% 99.44%
=======================================
Files 152 152
Lines 34110 34110
Branches 2586 2586
=======================================
Hits 33920 33920
Misses 114 114
Partials 76 76
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The first run crashed with access violations in numpy.linalg.inv and solve, in the OpenBLAS of numpy's win_arm64 wheel, in 14 unit root tests. Run only the univariate tests, and then the multiple comparison tests if they pass, in a single process with the name of each test. Show the BLAS that numpy loads. ci/github-actions/windows_arm_linalg.py runs each test that crashed in its own process and saves the arguments of every inv and solve before the call, so the matrix of a crash is kept. It passes each of these matrices, and random matrices of sizes 1 to 160, to inv or solve in processes that only import numpy, with other OpenBLAS kernels and with 1 and 4 threads. The results are in the job summary and the matrices are uploaded.
Remove the Azure pipeline and the wheel builds, which run the tests, so that only the Windows on ARM job runs the tests on this branch. Not for merging.
bashtage
changed the title
Load numpy with importlib, since it is only imported so that threadpoolctl finds its BLAS. Mark the subprocess import and call, which only run this script with its own arguments and without a shell. Replace the asserts with an error.
The crashes reproduce with numpy alone on the saved matrices, with 1 OpenBLAS thread and not with 4. Add a step that calls inv of the identity, and inv and solve of a random positive definite matrix, at sizes 1 to 64 and every 8th to 160 with 1 thread, each in its own process, and then with other numbers of threads at the sizes that crash. Show the processor, Windows version and runner image. Record the last call of a sweep rather than the end of the traceback.
Add det (the LU factorization alone) and cholesky to the map of the sizes that crash, and SciPy's inv, lu_factor and lu_solve, which use SciPy's own OpenBLAS, so that a crash can be placed in getrf or the solve, and in numpy or OpenBLAS. Rerun the calls that crash in a thread with a 64 MB stack to rule out a stack overflow. Show SciPy's build configuration.
This was referenced Nov 4, 2026
numpy.linalg.inv and solve crash in the OpenBLAS of numpy's win_arm64 wheels only when OpenBLAS starts with 1 thread (see #1004 to #1007), so do not set OPENBLAS_NUM_THREADS, OMP_NUM_THREADS or MKL_NUM_THREADS, and fail if any thread limit is set or OpenBLAS reports fewer than 2 threads. Run the tests that the other CI runs with pytest-xdist, then the slow tests, on Python 3.13 and 3.14 with numba and 3.15 without it (no win_arm64 wheels). statsmodels has no win_arm64 wheels for 3.12. Remove the crash isolation, which moved to #1004 to #1007.
bashtage
changed the title
They were removed while the Windows on ARM job was an experiment. They are the same as on main.
Note the Windows on ARM job, which does not limit the BLAS threads, and that arch builds and passes its tests on Windows on ARM with Python 3.13 and later, but crashes in NumPy's OpenBLAS with OPENBLAS_NUM_THREADS=1.
bashtage
changed the title
This was referenced Nov 4, 2026
arch-wheels builds and tests win_arm64 wheels for Python 3.13 to 3.15. Also note that OMP_NUM_THREADS=1 limits OpenBLAS to one thread, too.
bashtage
marked this pull request as ready for review
November 5, 2026 11:11
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds a GitHub Actions job that builds arch and runs the whole test suite on Windows on ARM (ARM64), on GitHub's
windows-11-armrunner. arch builds, and every test passes on Python 3.13, 3.14 and 3.15, with no BLAS thread limit.Results
Run 41928225500, numpy 2.5.4, SciPy 1.18.1, pandas 3.1.1, statsmodels 0.15.0, matplotlib 3.11.2:
win_arm64wheels)numpy's OpenBLAS reported 4 threads in every job, one for each of the runner's cores.
The job
.github/workflows/windows-arm.ymlruns on Python 3.13 and 3.14 with numba, and on 3.15 without it. Python 3.12 can't be tested, because statsmodels only haswin_arm64wheels from 3.13. For each Python, the job:win-arm64), so that it can't test an x64 Python running under emulation;--only-binary=:all:), so arch is the only package that is compiled;-m "not slow", with pytest-xdist), then the slow tests, whether or not the first ones pass.It runs on pushes to main and on pull requests, and a new push to a pull request cancels the run before it.
Why there's no thread limit
The other CI sets
OPENBLAS_NUM_THREADS,OMP_NUM_THREADSandMKL_NUM_THREADSto 1. With that limit, the first run of this job passed 31,565 tests, and 14 tests crashed innumpy.linalg.invandsolve, all in the unit root code. The experiments in #1004 to #1007 traced the crashes to the OpenBLAS of numpy'swin_arm64wheels (numpy/numpy#29442):invandsolvecrash for 23×23 to 31×31 matrices, with an access violation in LAPACK'sdgesv;win_arm64wheels has the bug, from 2.3.0 to the nightly.Without the limit, the 14 tests pass.
Release notes
The 9.0 notes change in three places:
OPENBLAS_NUM_THREADS=1orOMP_NUM_THREADS=1.Wheels
The release wheels are built in arch-wheels, and bashtage/arch-wheels#31 adds
win_arm64wheels for Python 3.13 to 3.15. All 27 of its jobs pass (run 41930418282). Each of the three Windows on ARM jobs builds and repairs awin_arm64wheel and tests it: 30,180 passed and 1,546 skipped, without the slow tests and with no BLAS thread limit. The notes assume that #31 is merged before 9.0 is released.Not included
The branch's history includes the experiments: commits that removed the Azure pipeline and the wheel builds and added scripts that isolated the crashes. Later commits restore and remove them, so the change against main is only the workflow and the release notes.
🤖 Generated with Claude Code