Skip to content

TST: test_blas_gemm_thread_safety depends on dev. version of openblas #31708

Closed

Description

@h-vetinari

I had been testing numpy v2.5.0rc1 in conda-forge and that worked fine; for v2.5.0 GA, CI is pretty red though, due to a test added in #31704.

Details
=================================== FAILURES ===================================
_________________________ test_blas_gemm_thread_safety _________________________
[gw0] linux -- Python 3.12.13 $SRC_DIR_run_env/bin/python
    def test_blas_gemm_thread_safety():
        # gh-31618: concurrently run transpose and no-transpose GEMM variants to
        # exercise possible thread safety issues due to lock sharding between
        # kernels, see OpenBLAS issue #5836.
        num_threads = 8
        num_iters = 10
        M = 512 * 512
    
        rng = np.random.default_rng(0x9e3779b9)
        no_trans = rng.random((M, 4))            # C-contiguous -> NoTrans GEMM
        no_trans_w = rng.random((4, 2))
        trans = rng.random((2, M)).T             # F-contiguous -> Trans GEMM
        trans_w = rng.random((2, 2))
        expected_no_trans = no_trans @ no_trans_w
        expected_trans = trans @ trans_w
    
        mismatches = 0
        lock = threading.Lock()
    
        def closure(i, b):
            nonlocal mismatches
            count = 0
            for _ in range(num_iters):
                b.wait()
                if i % 2:
                    ok = np.array_equal(no_trans @ no_trans_w, expected_no_trans)
                else:
                    ok = np.array_equal(trans @ trans_w, expected_trans)
                if not ok:
                    count += 1
            with lock:
                mismatches += count
    
        run_threaded(closure, num_threads, pass_count=True, pass_barrier=True)
    
        blas_name, blas_version = _detected_blas()
        if mismatches and _openblas_predates_gemm_fix(blas_name, blas_version):
            pytest.xfail(
                f"OpenBLAS version ({blas_version}) predates first OpenBLAS "
                "version with a fix (0.3.33.112)"
            )
    
>       assert mismatches == 0, (
            f"{mismatches} concurrent matmul results were corrupted "
            f"({blas_name} {blas_version})"
        )
E       AssertionError: 1 concurrent matmul results were corrupted (blas 3.9.0)
E       assert 1 == 0
M          = 262144
blas_name  = 'blas'
blas_version = '3.9.0'
closure    = <function test_blas_gemm_thread_safety.<locals>.closure at 0x7f3d07ddd4e0>
expected_no_trans = array([[0.82917678, 0.64631505],
       [1.12787757, 0.61713878],
       [0.58626928, 0.29928036],
       ...,
       [0.30331501, 0.19477312],
       [1.02426229, 0.7088999 ],
       [0.79192141, 0.50717405]], shape=(262144, 2))
expected_trans = array([[1.22946894, 0.72557492],
       [0.37233322, 0.61671449],
       [0.47452178, 0.67690663],
       ...,
       [0.89373242, 0.66193472],
       [0.83352945, 0.72978179],
       [0.78604996, 0.22733695]], shape=(262144, 2))
lock       = <unlocked _thread.lock object at 0x7f3d0e54e540>
mismatches = 1
no_trans   = array([[0.53808463, 0.40540176, 0.97098426, 0.29884582],
       [0.21645565, 0.85018773, 0.82101307, 0.79576803],
    ...0.82990353, 0.84376674, 0.61073338],
       [0.56083043, 0.88714949, 0.28678986, 0.73171138]],
      shape=(262144, 4))
no_trans_w = array([[0.02597612, 0.29835088],
       [0.14194139, 0.17054754],
       [0.57574152, 0.39776106],
       [0.66462428, 0.10178165]])
num_iters  = 10
num_threads = 8
rng        = Generator(PCG64) at 0x7F3D0552F060
trans      = array([[0.52190499, 0.99709558],
       [0.66401165, 0.01960259],
       [0.70724405, 0.10255883],
       ...,
       [0.55080277, 0.62915218],
       [0.65700102, 0.50680001],
       [0.0321854 , 0.80573585]], shape=(262144, 2))
trans_w    = array([[0.5325608 , 0.92152787],
       [0.95429448, 0.24533749]])
../test_run_env/lib/python3.12/site-packages/numpy/_core/tests/test_multithreading.py:149: AssertionError
=============================== warnings summary ===============================

We have the latest available openblas release, which however is 0.3.33, not 0.3.33.112. AFAICT from MacPython/openblas-libs#273, this makes numpy depend on a development version of openblas, which I would consider quite unfriendly to distributions.

To the credit of @mattip, this failure shouldn't show up for older openblas versions, but for some reason, the test skip for this

if mismatches and _openblas_predates_gemm_fix(blas_name, blas_version):
pytest.xfail(
f"OpenBLAS version ({blas_version}) predates first OpenBLAS "
"version with a fix (0.3.33.112)"
)

does not trigger in our case; although the condition in
def _openblas_predates_gemm_fix(name, version):
if 'openblas' not in name:
return False
try:
parsed = tuple(int(p) for p in version.split('.'))
except ValueError:
return False
return parsed < (0, 3, 33, 112)

should hold despite the mismatched tuple lengths, i.e. (0, 3, 33) < (0, 3, 33, 112) is True in my python 3.12 REPL. So presumably this must go wrong either due to the name or a failure to parse/populate the version tuple.

I'll also note that AssertionError: 1 concurrent matmul results were corrupted (blas 3.9.0) picks up an incorrect version (in some sense). We currently compile against BLAS/LAPACK 3.9 in conda-forge, but we have (the equivalent of) 3.11 at runtime

 │ libblas               ┆ 3.11.0    ┆ 8_haddc8a3_openblas  ┆ conda-forge ┆  18.40 KiB │
 │ libcblas              ┆ 3.11.0    ┆ 8_hd72aa62_openblas  ┆ conda-forge ┆  18.38 KiB │
 │ libopenblas           ┆ 0.3.33    ┆ pthreads_h9d3fd7e_0  ┆ conda-forge ┆   4.88 MiB │

Activity

  1. mattip commented on Jun 22, 2026

    @mattip
    Member

    Hmm. The test failure is a real existing problem in the versions of OpenBLAS in use, and will, like in previous versions, trip up users who use NumPy in multithreaded applications.

    With that, you could patch _openblas_predates_gemm_fix in conda-forge to always return True in order to skip the test and get the new release out, which would make 2.5 consistent with previous NumPy versions.

  2. h-vetinari commented on Jun 22, 2026

    @h-vetinari
    ContributorAuthor

    I don't doubt that this fixes a bug; though since the problem is pre-existing for a long time, I wonder if this is really worth depending on a development version of openblas. Be that as it may, the problem is actually in

    def _detected_blas():
    blas = np.show_config('dicts').get('Build Dependencies', {}).get('blas', {})
    return blas.get('name', 'unknown'), blas.get('version', 'unknown')

    in the sense that np.show_config will misdetect not just the version of conda-forge's blas stack (see above), but also fail to detect the underlying implementation (openblas/MKL/etc.):

    >>> pprint(np.show_config('dicts').get('Build Dependencies', {}))
    {'blas': {'detection method': 'pkgconfig',
              'found': True,
              'include directory': 'E:/miniforge/envs/dev/Library/include',
              'lib directory': 'E:/miniforge/envs/dev/Library/lib',
              'name': 'blas',                                                       # <---
              'openblas configuration': 'unknown',
              'pc file directory': 'E:/miniforge/envs/dev/Library/lib/pkgconfig',
              'version': '3.9.0'},
     'lapack': {'detection method': 'pkgconfig',
                'found': True,
                'include directory': 'E:/miniforge/envs/dev/Library/include',
                'lib directory': 'E:/miniforge/envs/dev/Library/lib',
                'name': 'lapack',                                                   # <---
                'openblas configuration': 'unknown',
                'pc file directory': 'E:/miniforge/envs/dev/Library/lib/pkgconfig',
                'version': '3.9.0'}}

    In other words, the 'openblas' in name in _openblas_predates_gemm_fix cannot succeed in conda-forge's setup.

  3. ngoldbaum commented on Jun 22, 2026

    @ngoldbaum
    Member

    Sorry, my fault for that bug! I'll try to send in a fix for the BLAS detection ASAP.

  4. ngoldbaum commented on Jun 22, 2026

    @ngoldbaum
    Member

    I opened #31714. @h-vetinari I think that should unblock conda-forge, let me know if you think I missed something.

  5. h-vetinari commented on Jun 22, 2026

    @h-vetinari
    ContributorAuthor

    Thanks a lot @ngoldbaum! I think there's probably more that could be done w.r.t. detecting the BLAS implementation in conda-ish environments, but that might be worth a separate issue?

  6. ngoldbaum commented on Jun 22, 2026

    @ngoldbaum
    Member

    Yeah, I bet we could make np.show_config have nicer output on conda even if threadpoolctl isn't installed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions