Repository navigation
Description
I had been testing numpy v2.5.0rc1 in conda-forge and that worked fine; for v2.5.0 GA, CI is pretty red though, due to a test added in #31704.
Details
=================================== FAILURES ===================================
_________________________ test_blas_gemm_thread_safety _________________________
[gw0] linux -- Python 3.12.13 $SRC_DIR_run_env/bin/python
def test_blas_gemm_thread_safety():
# gh-31618: concurrently run transpose and no-transpose GEMM variants to
# exercise possible thread safety issues due to lock sharding between
# kernels, see OpenBLAS issue #5836.
num_threads = 8
num_iters = 10
M = 512 * 512
rng = np.random.default_rng(0x9e3779b9)
no_trans = rng.random((M, 4)) # C-contiguous -> NoTrans GEMM
no_trans_w = rng.random((4, 2))
trans = rng.random((2, M)).T # F-contiguous -> Trans GEMM
trans_w = rng.random((2, 2))
expected_no_trans = no_trans @ no_trans_w
expected_trans = trans @ trans_w
mismatches = 0
lock = threading.Lock()
def closure(i, b):
nonlocal mismatches
count = 0
for _ in range(num_iters):
b.wait()
if i % 2:
ok = np.array_equal(no_trans @ no_trans_w, expected_no_trans)
else:
ok = np.array_equal(trans @ trans_w, expected_trans)
if not ok:
count += 1
with lock:
mismatches += count
run_threaded(closure, num_threads, pass_count=True, pass_barrier=True)
blas_name, blas_version = _detected_blas()
if mismatches and _openblas_predates_gemm_fix(blas_name, blas_version):
pytest.xfail(
f"OpenBLAS version ({blas_version}) predates first OpenBLAS "
"version with a fix (0.3.33.112)"
)
> assert mismatches == 0, (
f"{mismatches} concurrent matmul results were corrupted "
f"({blas_name} {blas_version})"
)
E AssertionError: 1 concurrent matmul results were corrupted (blas 3.9.0)
E assert 1 == 0
M = 262144
blas_name = 'blas'
blas_version = '3.9.0'
closure = <function test_blas_gemm_thread_safety.<locals>.closure at 0x7f3d07ddd4e0>
expected_no_trans = array([[0.82917678, 0.64631505],
[1.12787757, 0.61713878],
[0.58626928, 0.29928036],
...,
[0.30331501, 0.19477312],
[1.02426229, 0.7088999 ],
[0.79192141, 0.50717405]], shape=(262144, 2))
expected_trans = array([[1.22946894, 0.72557492],
[0.37233322, 0.61671449],
[0.47452178, 0.67690663],
...,
[0.89373242, 0.66193472],
[0.83352945, 0.72978179],
[0.78604996, 0.22733695]], shape=(262144, 2))
lock = <unlocked _thread.lock object at 0x7f3d0e54e540>
mismatches = 1
no_trans = array([[0.53808463, 0.40540176, 0.97098426, 0.29884582],
[0.21645565, 0.85018773, 0.82101307, 0.79576803],
...0.82990353, 0.84376674, 0.61073338],
[0.56083043, 0.88714949, 0.28678986, 0.73171138]],
shape=(262144, 4))
no_trans_w = array([[0.02597612, 0.29835088],
[0.14194139, 0.17054754],
[0.57574152, 0.39776106],
[0.66462428, 0.10178165]])
num_iters = 10
num_threads = 8
rng = Generator(PCG64) at 0x7F3D0552F060
trans = array([[0.52190499, 0.99709558],
[0.66401165, 0.01960259],
[0.70724405, 0.10255883],
...,
[0.55080277, 0.62915218],
[0.65700102, 0.50680001],
[0.0321854 , 0.80573585]], shape=(262144, 2))
trans_w = array([[0.5325608 , 0.92152787],
[0.95429448, 0.24533749]])
../test_run_env/lib/python3.12/site-packages/numpy/_core/tests/test_multithreading.py:149: AssertionError
=============================== warnings summary ===============================
We have the latest available openblas release, which however is 0.3.33, not 0.3.33.112. AFAICT from MacPython/openblas-libs#273, this makes numpy depend on a development version of openblas, which I would consider quite unfriendly to distributions.
To the credit of @mattip, this failure shouldn't show up for older openblas versions, but for some reason, the test skip for this
numpy/numpy/_core/tests/test_multithreading.py
Lines 143 to 147 in 8680b48
| if mismatches and _openblas_predates_gemm_fix(blas_name, blas_version): | |
| pytest.xfail( | |
| f"OpenBLAS version ({blas_version}) predates first OpenBLAS " | |
| "version with a fix (0.3.33.112)" | |
| ) |
does not trigger in our case; although the condition in
numpy/numpy/_core/tests/test_multithreading.py
Lines 97 to 104 in 8680b48
| def _openblas_predates_gemm_fix(name, version): | |
| if 'openblas' not in name: | |
| return False | |
| try: | |
| parsed = tuple(int(p) for p in version.split('.')) | |
| except ValueError: | |
| return False | |
| return parsed < (0, 3, 33, 112) |
should hold despite the mismatched tuple lengths, i.e.
(0, 3, 33) < (0, 3, 33, 112) is True in my python 3.12 REPL. So presumably this must go wrong either due to the name or a failure to parse/populate the version tuple.
I'll also note that AssertionError: 1 concurrent matmul results were corrupted (blas 3.9.0) picks up an incorrect version (in some sense). We currently compile against BLAS/LAPACK 3.9 in conda-forge, but we have (the equivalent of) 3.11 at runtime
│ libblas ┆ 3.11.0 ┆ 8_haddc8a3_openblas ┆ conda-forge ┆ 18.40 KiB │
│ libcblas ┆ 3.11.0 ┆ 8_hd72aa62_openblas ┆ conda-forge ┆ 18.38 KiB │
│ libopenblas ┆ 0.3.33 ┆ pthreads_h9d3fd7e_0 ┆ conda-forge ┆ 4.88 MiB │
Activity
mattip commentedon Jun 22, 2026
Hmm. The test failure is a real existing problem in the versions of OpenBLAS in use, and will, like in previous versions, trip up users who use NumPy in multithreaded applications.
With that, you could patch
_openblas_predates_gemm_fixin conda-forge to always return True in order to skip the test and get the new release out, which would make 2.5 consistent with previous NumPy versions.h-vetinari commentedon Jun 22, 2026
I don't doubt that this fixes a bug; though since the problem is pre-existing for a long time, I wonder if this is really worth depending on a development version of openblas. Be that as it may, the problem is actually in
numpy/numpy/_core/tests/test_multithreading.py
Lines 92 to 94 in 8680b48
in the sense that
np.show_configwill misdetect not just the version of conda-forge's blas stack (see above), but also fail to detect the underlying implementation (openblas/MKL/etc.):In other words, the
'openblas' in namein_openblas_predates_gemm_fixcannot succeed in conda-forge's setup.ngoldbaum commentedon Jun 22, 2026
Sorry, my fault for that bug! I'll try to send in a fix for the BLAS detection ASAP.
ngoldbaum commentedon Jun 22, 2026
I opened #31714. @h-vetinari I think that should unblock conda-forge, let me know if you think I missed something.
h-vetinari commentedon Jun 22, 2026
Thanks a lot @ngoldbaum! I think there's probably more that could be done w.r.t. detecting the BLAS implementation in conda-ish environments, but that might be worth a separate issue?
ngoldbaum commentedon Jun 22, 2026
Yeah, I bet we could make np.show_config have nicer output on conda even if threadpoolctl isn't installed.