Repository navigation
Closed
Labels
CIItems related to the CI tools such as CircleCI, GitHub Actions or AzureItems related to the CI tools such as CircleCI, GitHub Actions or AzuredefectA clear bug or issue that prevents SciPy from being installed or used as expectedA clear bug or issue that prevents SciPy from being installed or used as expectedscipy.optimize
Milestone
Description
In gh-23607 I just noticed that the new win-arm64 job is failing with a crash (from this CI log):
Windows fatal exception: access violation
Thread 0x00001ec0 (most recent call first):
File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\execnet\gateway_base.py", line 534 in read
File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\execnet\gateway_base.py", line 567 in from_io
File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\execnet\gateway_base.py", line 1160 in _thread_receiver
File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\execnet\gateway_base.py", line 341 in run
File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\execnet\gateway_base.py", line 411 in _perform_spawn
Current thread 0x000018c0 (most recent call first):
File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\scipy\optimize\_minpack_py.py", line 439 in leastsq
File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\scipy\optimize\_root.py", line 323 in _root_leastsq
File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\scipy\optimize\_root.py", line 255 in root
File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\scipy\optimize\tests\test_minpack.py", line 257 in test_pressure_network_no_gradient
File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\_pytest\python.py", line 157 in pytest_pyfunc_call
File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\pluggy\_callers.py", line 121 in _multicall
File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec
...
...............[gw0] node down: Not properly terminated
F
replacing crashed worker gw0
...
=========================== short test summary info ===========================
FAILED optimize/tests/test_minpack.py::TestRootLM::test_pressure_network_no_gradient
Note that ASan is flagging that exact same test, see #23429 (comment). So it may be easier to address that ASan issue on Linux or macOS than to try and debug on Windows arm64.
Metadata
Metadata
Assignees
Labels
CIItems related to the CI tools such as CircleCI, GitHub Actions or AzureItems related to the CI tools such as CircleCI, GitHub Actions or AzuredefectA clear bug or issue that prevents SciPy from being installed or used as expectedA clear bug or issue that prevents SciPy from being installed or used as expectedscipy.optimize
Activity
nickodell commentedon Sep 15, 2025
@rgommers Do you know how to build SciPy with ASan? I tried the following, on Linux:
But I get this error:
I couldn't find any SciPy documentation about building with ASan, so I may be doing something incorrectly.
lucascolley commentedon Sep 15, 2025
@ngoldbaum I noticed numpy/numpy#29079, I assume the steps will be similar for SciPy?
ilayn commentedon Sep 15, 2025
I did not know that you could use
--asannice to know. It would be much easier to change only the meson.build file to include asan to that particular translation unit. Otherwise you'll get everything tested. You can use c_args and linker_args fields in the meson fields.But that message tells you that you have to use
export LD_PRELOAD_PATH=<path-to-asan-lib>; spin <command>. I don't remember by heart since I don't have any computer around.ngoldbaum commentedon Sep 15, 2025
@lucascolley argh - now I feel guilty for never writing those docs!
The trick is you need to build your entire Python stack, including CPython itself, using address sanitizer.
Here's a CI job where NumPy runs the unit tests under both address and undefined behavior sanitizer, using a build of CPython with address and undefined behavior sanitizers enabled:
https://github.com/numpy/numpy/blob/5f8296630d0102029175164a76fc1eae6da3fd20/.github/workflows/compiler_sanitizers.yml#L23-L74
You can just skip the UBSan bits if all you care about is ASan.
If your entire stack was built with sanitizer instrumentation, there should be no need to use
LD_PRELOADor similar.lucascolley commentedon Sep 15, 2025
thanks @ngoldbaum ! Seems feasible to do that with Pixi then, just need to figure out what https://github.com/pyenv/pyenv/blob/master/plugins/python-build/bin/pyenv-install is doing under the hood.
rgommers commentedon Sep 15, 2025
Here is a working CI job that builds CPython, NumPy and SciPy with ASan: https://github.com/scipy/scipy/compare/main...rgommers:scipy:ci-asan?expand=1 on macOS. It's similar to the NumPy job, but a few tweaks were needed (yay for Fortran).
I was planning to work through the failures or add suppressions for them, so that we can add it as a new job, but I don't think I'll have time for that in the next couple of weeks. But it's usable as is when you're just running the tests that you're interested in.
ngoldbaum commentedon Sep 15, 2025
Nothing special - just running
./configure; make; make installunder a bunch of complicated shell scripts.lucascolley commentedon Sep 15, 2025
I at least see some output after setting
LD_PRELOAD:that's just with this diff:
ngoldbaum commentedon Sep 15, 2025
This will work too and you'll be able to see any issues that are "local" to SciPy.
I would disable leak detection using ASAN_OPTIONS though because it will have a lot of false-positives. Any allocation that goes through
PyMem_Mallocor similar won't be instrumented.lucascolley commentedon Sep 15, 2025
thanks, adding
env.ASAN_OPTIONS = "detect_leaks=0"leaves us with seemingly no "local" issues for the tests @nickodell was interested in:rgommers commentedon Nov 25, 2025
We now have macOS-based ASan CI job (xref gh-23936). It currently silences 4 issues, one of which is in
minpackand may explain this crash.