Skip to content

Conversation

@mattip
Copy link
Member

@mattip mattip commented Oct 22, 2025

Closes #29391 by patching OpenBLAS. Hopefully by the time 0.3.31 is released we can remove the patch.

Also use spin build -- -Dpkg_config_path= instead of setting PKG_CONFIG_PATH in the normal (non-wheel building) CI.

@mattip mattip added the 09 - Backport-Candidate PRs tagged should be backported label Oct 22, 2025
@github-actions github-actions bot added the 36 - Build Build related PR label Oct 22, 2025
python -c"import scipy_openblas32 as ob32; print(ob32.get_pkg_config())" > ./.openblas/scipy-openblas.pc
echo "PKG_CONFIG_PATH=${{ github.workspace }}/.openblas" >> $GITHUB_ENV
ld_library_path=$(python -c"import scipy_openblas32 as ob32; print(ob32.get_lib_dir())")
echo "LD_LIBRARY_PATH=$ld_library_path" >> $GITHUB_ENV
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure why we have LD_LIBRARY_PATH here, leftover cruft?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, leftover I believe, it was needed once upon a time.

@mattip
Copy link
Member Author

mattip commented Oct 23, 2025

Exploring the crash on macos-x86_64 over at MacPython/openblas-libs#219. Any macos compilation experts are welcome to chime in there what might be wrong.

TERM: xterm-256color
run:
spin build -- --werror -Dallow-noblas=false
spin build -- --werror -Dallow-noblas=false -Dpkg_config_path=${PWD}/.openblas
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice change

@mattip
Copy link
Member Author

mattip commented Oct 30, 2025

The build to watch is "Wheel builder / Build wheel cp311-macosx_x86_64-openblas" which failed with the previous scipy_openblas wheel.

@mattip
Copy link
Member Author

mattip commented Oct 30, 2025

The ubuntu_ubisan CI run seems flaky, I see occasional failures elsewhere.

spin==0.15
# Keep this in sync with ci_requirements.txt
scipy-openblas32==0.3.30.0.1
scipy-openblas32==0.3.30.0.6
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note this does not change the base OpenBLAS commit used, only patches out the not-quite-ready code. OpenBLAS HEAD has also reverted that change, so the patch will not be needed for OpenBLAS 0.3.31. That is why the version here only changes the build from 1 to 6.

@rgommers rgommers added this to the 2.4.0 release milestone Oct 30, 2025
@rgommers
Copy link
Member

The issue you mentioned above (MacPython/openblas-libs#219) is still open - are you happy for this to go in now?

@mattip
Copy link
Member Author

mattip commented Oct 30, 2025

are you happy for this to go in now?

Yes. PR MacPython/openblas-libs#225 fixed MacPython/openblas-libs#219, as "proven" in the passing "Build wheel cp311-macosx_x86_64-openblas" CI run here so I closed that issue.

@rgommers
Copy link
Member

Recording the unrelated ASan failure for posterity:

 numpy/_core/tests/test_multiarray.py::TestArgmaxArgminCommon::test_object_with_NULLs[argmin-vals1] PASSED
Fatal Python error: Segmentation fault
Details
<Cannot show all threads while the GIL is disabled>
Stack (most recent call first):
  File "/home/runner/work/numpy/numpy/build-install/usr/lib/python3.14t/site-packages/numpy/_core/fromnumeric.py", line 55 in _wrapfunc
  File "/home/runner/work/numpy/numpy/build-install/usr/lib/python3.14t/site-packages/numpy/_core/fromnumeric.py", line 1319 in argmax
  File "/home/runner/work/numpy/numpy/build-install/usr/lib/python3.14t/site-packages/numpy/_core/tests/test_multiarray.py", line 5082 in test_combinations
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/_pytest/python.py", line 1788 in runtest
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/_pytest/runner.py", line 341 in from_call
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/_pytest/main.py", line 349 in pytest_runtestloop
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/_pytest/main.py", line 324 in _main
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/_pytest/main.py", line 270 in wrap_session
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/_pytest/main.py", line 317 in pytest_cmdline_main
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/pluggy/_callers.py", line 121 in _multicall
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/pluggy/_hooks.py", line 512 in __call__
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/_pytest/config/__init__.py", line 166 in main
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/_pytest/config/__init__.py", line 189 in console_main
  File "/home/runner/.pyenv/versions/3.14.0t/lib/python3.14t/site-packages/pytest/__main__.py", line 5 in <module>
  File "<frozen runpy>", line 88 in _run_code
  File "<frozen runpy>", line 198 in _run_module_as_main

Current thread's C stack trace (most recent call first):
  Binary file "/lib/x86_64-linux-gnu/libasan.so.8", at +0x831e0 [0x7fdced0831e0]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at _Py_DumpStack+0xb6 [0x7fdcec8a8c56]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at +0x8e3363 [0x7fdcec8e3363]
  Binary file "/lib/x86_64-linux-gnu/libc.so.6", at +0x45330 [0x7fdcebc45330]
  Binary file "/home/runner/work/numpy/numpy/build-install/usr/lib/python3.14t/site-packages/numpy/_core/_multiarray_umath.cpython-314t-x86_64-linux-gnu.so", at +0x4b046a2 [0x7fdca3d046a2]
  Binary file "/home/runner/work/numpy/numpy/build-install/usr/lib/python3.14t/site-packages/numpy/_core/_multiarray_umath.cpython-314t-x86_64-linux-gnu.so", at +0x4b56555 [0x7fdca3d56555]
  Binary file "/home/runner/work/numpy/numpy/build-install/usr/lib/python3.14t/site-packages/numpy/_core/_multiarray_umath.cpython-314t-x86_64-linux-gnu.so", at +0x31d691e [0x7fdca23d691e]
  Binary file "/home/runner/work/numpy/numpy/build-install/usr/lib/python3.14t/site-packages/numpy/_core/_multiarray_umath.cpython-314t-x86_64-linux-gnu.so", at +0x3457b6b [0x7fdca2657b6b]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at +0x340b69 [0x7fdcec340b69]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at _PyEval_EvalFrameDefault+0x20dc2 [0x7fdcec1ea822]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at +0x6fc2dd [0x7fdcec6fc2dd]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at PyObject_Vectorcall+0xfc [0x7fdcec33c79c]
  Binary file "/home/runner/work/numpy/numpy/build-install/usr/lib/python3.14t/site-packages/numpy/_core/_multiarray_umath.cpython-314t-x86_64-linux-gnu.so", at +0x31bb9c6 [0x7fdca23bb9c6]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at PyObject_Vectorcall+0xfc [0x7fdcec33c79c]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at _PyEval_EvalFrameDefault+0x30c1 [0x7fdcec1ccb21]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at +0x6fc2dd [0x7fdcec6fc2dd]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at +0x34690c [0x7fdcec34690c]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at +0x340b69 [0x7fdcec340b69]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at _PyEval_EvalFrameDefault+0x20dc2 [0x7fdcec1ea822]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at +0x6fc2dd [0x7fdcec6fc2dd]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at +0x34126c [0x7fdcec34126c]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at +0x34156d [0x7fdcec34156d]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at +0x53537d [0x7fdcec53537d]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at _PyObject_MakeTpCall+0x131 [0x7fdcec33c0a1]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at _PyEval_EvalFrameDefault+0x1d2df [0x7fdcec1e6d3f]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at +0x6fc2dd [0x7fdcec6fc2dd]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at +0x34126c [0x7fdcec34126c]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at +0x34156d [0x7fdcec34156d]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at +0x53537d [0x7fdcec53537d]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at PyObject_Call+0xb9 [0x7fdcec340fe9]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at _PyEval_EvalFrameDefault+0x20dc2 [0x7fdcec1ea822]
  Binary file "/home/runner/.pyenv/versions/3.14.0t/lib/libpython3.14t.so.1.0", at +0x6fc2dd [0x7fdcec6fc2dd]
  <truncated rest of calls>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, numpy._core._multiarray_tests, numpy._core._rational_tests, numpy._core._umath_tests, cython.cimports.libc.math, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._generator, numpy.random._mt19937, numpy.random._philox, numpy.random._sfc64, numpy.random.mtrand, numpy._core._struct_ufunc_tests, numpy._core._simd, numpy._core._operand_flag_tests, numpy.linalg.lapack_lite, checks, mem_policy (total: 21)
AddressSanitizer:DEADLYSIGNAL

Copy link
Member

@rgommers rgommers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In it goes, thanks Matti!

@rgommers rgommers merged commit 56243c9 into numpy:main Oct 30, 2025
75 of 76 checks passed
charris pushed a commit to charris/numpy that referenced this pull request Nov 3, 2025
@charris charris removed the 09 - Backport-Candidate PRs tagged should be backported label Nov 3, 2025
charris added a commit that referenced this pull request Nov 3, 2025
BLD: update scipy-openblas, use -Dpkg_config_path (#30049)
cakedev0 pushed a commit to cakedev0/numpy that referenced this pull request Dec 5, 2025
IndifferentArea pushed a commit to IndifferentArea/numpy that referenced this pull request Dec 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

36 - Build Build related PR

Projects

None yet

Development

Successfully merging this pull request may close these issues.

BUG: np.dot produces incorrect results when run concurrently

3 participants