BUG: np.dot is thread-unsafe for non-contiguous input with OpenBLAS #31618
Description
Over in scikit-image/scikit-image#8212, the scikit-image developers are seeing random failures in a test that does some NumPy operations involving OpenBLAS.
Consider the following script:
import threading
import numpy as np
THREADS, ITERS = 12, 50
N = 512 * 512
coords = np.indices((512, 512), dtype=float).reshape(2, -1).T # non-contiguous
radial = np.random.rand(N, 4)
A = np.random.rand(2, 2)
W = np.random.rand(4, 2)
compute = lambda: coords @ A + radial @ W
expected = compute()
mismatches = 0
lock = threading.Lock()
barrier = threading.Barrier(THREADS)
def worker():
global mismatches
count = 0
for _ in range(ITERS):
barrier.wait() # line up all threads on the BLAS call
if not np.array_equal(compute(), expected):
count += 1
with lock:
mismatches += count
threads = [threading.Thread(target=worker) for _ in range(THREADS)]
for t in threads:
t.start()
for t in threads:
t.join()
total = THREADS * ITERS
print(f"{mismatches} mismatches / {total} calls "
f"({'RACE DETECTED' if mismatches else 'clean'})")Running this script with the following bash wrapper:
cd /tmp/repro && echo "=== FREE-THREADED build (3.14t) — 3 runs ===" && for i in 1 2 3; do t0=$(date +%s.%N); out=$(/tmp/venv-ft/bin/python tps_openblas_race.py 2>/dev/null); t1=$(date +%s.%N); echo " run $i: $out [$(echo "$t1-$t0"|bc)s]"; done
echo " control OPENBLAS_NUM_THREADS=1: $(OPENBLAS_NUM_THREADS=1 /tmp/venv-ft/bin/python tps_openblas_race.py 2>/dev/null)"
Outputs on the free-threaded build:
run 1: 15 mismatches / 600 calls (RACE DETECTED) [5.6s]
run 2: 17 mismatches / 600 calls (RACE DETECTED) [5.8s]
run 3: 16 mismatches / 600 calls (RACE DETECTED) [5.9s]
control OPENBLAS_NUM_THREADS=1: 0 mismatches / 600 calls (clean)
And on the GIL-enabled build:
run 1: 41 mismatches / 600 calls (RACE DETECTED) [4.4s]
run 2: 35 mismatches / 600 calls (RACE DETECTED) [4.2s]
run 3: 43 mismatches / 600 calls (RACE DETECTED) [4.1s]
control OPENBLAS_NUM_THREADS=1: 0 mismatches / 600 calls (clean)
So this isn't particular to the free-threaded build.
np.show_config() says:
name: scipy-openblas
version: 0.3.31.188.0
config: OpenBLAS 0.3.31.188.0 USE64BITINT DYNAMIC_ARCH NO_AFFINITY MAX_THREADS=64
threadpoolctl.threadpool_info() outputs:
openblas | libscipy_openblas | v0.3.31.188.0 | num_threads=4 | threading=pthreads
Full np.show_config() output:
Details
{
"Compilers": {
"c": { "name": "gcc", "linker": "ld.bfd", "version": "14.2.1", "commands": "cc" },
"cython": { "name": "cython", "linker": "cython", "version": "3.2.4", "commands": "cython" },
"c++": { "name": "gcc", "linker": "ld.bfd", "version": "14.2.1", "commands": "c++" }
},
"Machine Information": {
"host": { "cpu": "x86_64", "family": "x86_64", "endian": "little", "system": "linux" },
"build": { "cpu": "x86_64", "family": "x86_64", "endian": "little", "system": "linux" }
},
"Build Dependencies": {
"blas": {
"name": "scipy-openblas",
"found": true,
"version": "0.3.31.188.0",
"detection method": "pkgconfig",
"include directory": "/opt/_internal/cpython-3.14.0-nogil/lib/python3.14t/site-packages/scipy_openblas64/include",
"lib directory": "/opt/_internal/cpython-3.14.0-nogil/lib/python3.14t/site-packages/scipy_openblas64/lib",
"openblas configuration": "OpenBLAS 0.3.31.188.0 USE64BITINT DYNAMIC_ARCH NO_AFFINITY SkylakeX MAX_THREADS=64",
"pc file directory": "/project/.openblas"
},
"lapack": {
"name": "scipy-openblas",
"found": true,
"version": "0.3.31.188.0",
"detection method": "pkgconfig",
"include directory": "/opt/_internal/cpython-3.14.0-nogil/lib/python3.14t/site-packages/scipy_openblas64/include",
"lib directory": "/opt/_internal/cpython-3.14.0-nogil/lib/python3.14t/site-packages/scipy_openblas64/lib",
"openblas configuration": "OpenBLAS 0.3.31.188.0 USE64BITINT DYNAMIC_ARCH NO_AFFINITY SkylakeX MAX_THREADS=64",
"pc file directory": "/project/.openblas"
}
},
"Python Information": {
"path": "/tmp/build-env-h9ocvpix/bin/python",
"version": "3.14"
},
"SIMD Extensions": {
"baseline": ["X86_V2"],
"found": ["X86_V3", "X86_V4"],
"not found": ["AVX512_ICL", "AVX512_SPR"]
}
}
I think this is an OpenBLAS bug but since we vendor OpenBLAS in our wheels and I don't know how to reproduce this without NumPy I'm creating an issue here first.
I used Claude to generate the test case.
Ping @martin-frbg @mattip @rgommers
Reacted by Sebastian Berg
Activity
[-]np.dot is thread-unsafe for non-contiguous input with OpenBLAS[/-][+]BUG: np.dot is thread-unsafe for non-contiguous input with OpenBLAS[/+]ngoldbaum commentedon Jun 13, 2026
I opened OpenMathLib/OpenBLAS#5836 with a pure C reproducer
mattip commentedon Jun 15, 2026
The OpenBLAS issue was solved so I started MacPython/openblas-libs#273 to create an appropriate scipy-openblas release.
ngoldbaum commentedon Jun 15, 2026
We should probably add a multithreaded regression test to the NumPy tests so we catch it just in case other BLAS libraries ever suffer from the same problem.
mattip commentedon Jun 15, 2026
I am a little hesitant to add a 5 sec test to the suite for this case. Don't we already have a similar test?
ngoldbaum commentedon Jun 15, 2026
I can make it run faster, I think. We don't yet have any similar tests in
test_multithreading, I think.mattip commentedon Jun 15, 2026
Feel free to add a test to #31649