Skip to content

BUG: optimize: crash in win-arm64 CI job in optimize.root(method='lm') #23612

Closed

Description

@rgommers

In gh-23607 I just noticed that the new win-arm64 job is failing with a crash (from this CI log):

Windows fatal exception: access violation

Thread 0x00001ec0 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\execnet\gateway_base.py", line 534 in read
  File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\execnet\gateway_base.py", line 567 in from_io
  File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\execnet\gateway_base.py", line 1160 in _thread_receiver
  File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\execnet\gateway_base.py", line 341 in run
  File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\execnet\gateway_base.py", line 411 in _perform_spawn

Current thread 0x000018c0 (most recent call first):
  File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\scipy\optimize\_minpack_py.py", line 439 in leastsq
  File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\scipy\optimize\_root.py", line 323 in _root_leastsq
  File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\scipy\optimize\_root.py", line 255 in root
  File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\scipy\optimize\tests\test_minpack.py", line 257 in test_pressure_network_no_gradient
  File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\_pytest\python.py", line 157 in pytest_pyfunc_call
  File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\pluggy\_callers.py", line 121 in _multicall
  File "C:\hostedtoolcache\windows\Python\3.11.9\arm64\Lib\site-packages\pluggy\_manager.py", line 120 in _hookexec

...

...............[gw0] node down: Not properly terminated
F
replacing crashed worker gw0

...

=========================== short test summary info ===========================
 FAILED optimize/tests/test_minpack.py::TestRootLM::test_pressure_network_no_gradient

Note that ASan is flagging that exact same test, see #23429 (comment). So it may be easier to address that ASan issue on Linux or macOS than to try and debug on Windows arm64.

Activity

  1. added
    defectA clear bug or issue that prevents SciPy from being installed or used as expected
    on Sep 13, 2025
  2. added
    CIItems related to the CI tools such as CircleCI, GitHub Actions or Azure
    on Sep 13, 2025
  3. nickodell commented on Sep 15, 2025

    @nickodell
    Member

    Note that ASan is flagging that exact same test, see 23429. So it may be easier to address that ASan issue on Linux or macOS than to try and debug on Windows arm64.

    @rgommers Do you know how to build SciPy with ASan? I tried the following, on Linux:

    rm -rf build build-install/
    spin build --asan
    spin test -t scipy.optimize.tests.test_nnls
    

    But I get this error:

    (scipy-dev) nodell@scipy-dev:~/scipy$ spin test -t scipy.optimize.tests.test_nnls
    Invoking `build` prior to running tests:
    $ meson compile -j 6 -C build
    INFO: autodetecting backend as ninja
    INFO: calculating backend command to run: /home/nodell/.miniforge3/envs/scipy-dev/bin/ninja -C /home/nodell/scipy/build -j 6
    ninja: Entering directory `/home/nodell/scipy/build'
    [2/338] Generating subprojects/highs/src/HConfig.h with a custom command
    $ meson install --only-changed -C build --destdir ../build-install --tags=runtime,python-runtime,tests,devel
    $ export PYTHONPATH="/home/nodell/scipy/build-install/usr/lib/python3.13/site-packages"
    $ /home/nodell/.miniforge3/envs/scipy-dev/bin/python3.13 -P -c 'import scipy'
    ==270868==ASan runtime does not come first in initial library list; you should either link runtime to your application or manually preload it with LD_PRELOAD.
    As a sanity check, we tried to import scipy.
    Stopping. Please investigate the build error.
    

    I couldn't find any SciPy documentation about building with ASan, so I may be doing something incorrectly.

  4. lucascolley commented on Sep 15, 2025

    @lucascolley
    Member

    @ngoldbaum I noticed numpy/numpy#29079, I assume the steps will be similar for SciPy?

  5. ilayn commented on Sep 15, 2025

    @ilayn
    Member

    I couldn't find any SciPy documentation about building with ASan, so I may be doing something incorrectly.

    I did not know that you could use --asan nice to know. It would be much easier to change only the meson.build file to include asan to that particular translation unit. Otherwise you'll get everything tested. You can use c_args and linker_args fields in the meson fields.

    But that message tells you that you have to use export LD_PRELOAD_PATH=<path-to-asan-lib>; spin <command>. I don't remember by heart since I don't have any computer around.

  6. ngoldbaum commented on Sep 15, 2025

    @ngoldbaum
    Contributor

    @lucascolley argh - now I feel guilty for never writing those docs!

    The trick is you need to build your entire Python stack, including CPython itself, using address sanitizer.

    Here's a CI job where NumPy runs the unit tests under both address and undefined behavior sanitizer, using a build of CPython with address and undefined behavior sanitizers enabled:

    https://github.com/numpy/numpy/blob/5f8296630d0102029175164a76fc1eae6da3fd20/.github/workflows/compiler_sanitizers.yml#L23-L74

    You can just skip the UBSan bits if all you care about is ASan.

    If your entire stack was built with sanitizer instrumentation, there should be no need to use LD_PRELOAD or similar.

  7. lucascolley commented on Sep 15, 2025

    @lucascolley
    Member

    thanks @ngoldbaum ! Seems feasible to do that with Pixi then, just need to figure out what https://github.com/pyenv/pyenv/blob/master/plugins/python-build/bin/pyenv-install is doing under the hood.

  8. rgommers commented on Sep 15, 2025

    @rgommers
    MemberAuthor

    Here is a working CI job that builds CPython, NumPy and SciPy with ASan: https://github.com/scipy/scipy/compare/main...rgommers:scipy:ci-asan?expand=1 on macOS. It's similar to the NumPy job, but a few tweaks were needed (yay for Fortran).

    I was planning to work through the failures or add suppressions for them, so that we can add it as a new job, but I don't think I'll have time for that in the next couple of weeks. But it's usable as is when you're just running the tests that you're interested in.

  9. ngoldbaum commented on Sep 15, 2025

    @ngoldbaum
    Contributor

    what https://github.com/pyenv/pyenv/blob/master/plugins/python-build/bin/pyenv-install is doing under the hood.

    Nothing special - just running ./configure; make; make install under a bunch of complicated shell scripts.

  10. lucascolley commented on Sep 15, 2025

    @lucascolley
    Member

    I at least see some output after setting LD_PRELOAD:

    pixi-dev-scipystack/scipy on 🎋 asan [🔨] via 🧚 v0.55.0 took 2s
    ❯ pixi run test-asan
    ✨ Pixi task (test-asan in asan): spin test --no-build
    {snip}
    =================================================================
    ==31565==ERROR: LeakSanitizer: detected memory leaks
    
    Direct leak of 3824 byte(s) in 4 object(s) allocated from:
        #0 0x704a970b1cba in malloc ../../../../libsanitizer/asan/asan_malloc_linux.cpp:69
        #1 0x606b414f7274 in _PyMem_RawMalloc /usr/local/src/conda/python-3.12.11/Objects/obmalloc.c:45
        #2 0x606b414f7274 in PyMem_RawMalloc /usr/local/src/conda/python-3.12.11/Objects/obmalloc.c:662
        #3 0x606b414f7274 in _PyObject_Malloc /usr/local/src/conda/python-3.12.11/Objects/obmalloc.c:1569
        #4 0x606b414f7274 in PyObject_Malloc /usr/local/src/conda/python-3.12.11/Objects/obmalloc.c:801
        #5 0x606b414f7274 in new_keys_object /usr/local/src/conda/python-3.12.11/Objects/dictobject.c:644
        #6 0x606b414f7274 in dictresize /usr/local/src/conda/python-3.12.11/Objects/dictobject.c:1453
    
    SUMMARY: AddressSanitizer: 3824 byte(s) leaked in 4 allocation(s).

    that's just with this diff:

    diff --git a/scipy/pixi.toml b/scipy/pixi.toml
    index 9276c4b..b326529 100644
    --- a/scipy/pixi.toml
    +++ b/scipy/pixi.toml
    @@ -66,4 +66,18 @@ depends-on = "build"
    
    
    +[feature.asan]
    +platforms = ["linux-64"]
    +
    +[feature.asan.tasks.build-asan]
    +cmd = "spin build --asan --setup-args=-Dblas=blas --setup-args=-Dlapack=lapack --setup-args=-Duse-g77-abi=true"
    +cwd = "scipy"
    +env = { CC = "ccache $CC", CXX = "ccache $CXX", FC = "ccache $FC" }
    +
    +[feature.asan.tasks.test-asan]
    +cmd = "spin test --no-build"
    +cwd = "scipy"
    +depends-on = "build-asan"
    +env.LD_PRELOAD = "/home/lucas/ghq/github.com/rgommers/pixi-dev-scipystack/scipy/.pixi/envs/asan/x86_64-conda-linux-gnu/lib/libasan.so"
    +
    +
     [feature.test.tasks.test]
     cmd = "spin test --no-build"
    @@ -433,4 +447,5 @@ depends-on = "build-editable"
     [environments]
     default = ["build", "test", "typing", "bench", "openblas", "lint"]
    +asan = ["asan"]
  11. ngoldbaum commented on Sep 15, 2025

    @ngoldbaum
    Contributor

    after setting LD_PRELOAD

    This will work too and you'll be able to see any issues that are "local" to SciPy.

    I would disable leak detection using ASAN_OPTIONS though because it will have a lot of false-positives. Any allocation that goes through PyMem_Malloc or similar won't be instrumented.

  12. lucascolley commented on Sep 15, 2025

    @lucascolley
    Member

    thanks, adding env.ASAN_OPTIONS = "detect_leaks=0" leaves us with seemingly no "local" issues for the tests @nickodell was interested in:

    pixi-dev-scipystack/scipy on 🎋 asan [🔨] via 🧚 v0.55.0 took 6s
    ❯ pixi run test-asan -t scipy.optimize.tests.test_nnls
    {snip}
    =========================================================================================== 14 passed in 0.98s ============================================================================================
  13. rgommers commented on Nov 25, 2025

    @rgommers
    MemberAuthor

    We now have macOS-based ASan CI job (xref gh-23936). It currently silences 4 issues, one of which is in minpack and may explain this crash.

  14. added this to the 1.18.0 milestone on Jan 10, 2026
  15. modified the milestones: 1.18.0, 1.17.1 on Jan 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    CIItems related to the CI tools such as CircleCI, GitHub Actions or AzuredefectA clear bug or issue that prevents SciPy from being installed or used as expectedscipy.optimize

    Type

    No type

    Projects

    No projects

      Milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions