TST: a flaky test on macOS arm64 (RuntimeWarning: invalid value encountered in matmul)

@mattip

Seen in gh-30160
@mattip reported seeing this independently too

Basically, the test (sometimes) fails as

FAILED numpy/linalg/tests/test_linalg.py::TestQR::test_stacked_inputs[float32-outer_size2-size1] - RuntimeWarning: invalid value encountered in matmul

I'm able to reproduce this locally (although unreliably), so I'll attempt a bisection.
possibly related to gh-29820

Hum, looks like I was insanely lucky when I reproduced it locally after only ~5 attempts. I was not yet able to hit the bug (which I assume is a race) a single time since, despite dozens of subsequent attempts. Would be great to know if there's anything I can do to increase the likelyhood of hitting the race.

Ah, got it a second time. I tried both Python 3.11 and 3.14 and I think both my lucky tries were on 3.11.

here's the script I'm running

set -euxo pipefail

uv venv -c -p 3.11
source .venv/bin/activate

uv pip install -r requirements/build_requirements.txt -r requirements/test_requirements.txt
spin build --clean
spin test -j2 -- numpy/linalg --timeout=30

(I'm building from scratch every time because I have no idea if this bug may be affected (possibly hidden) by partial rebuilds)

Documenting the failure:

CI run failure

$ /Library/Frameworks/Python.framework/Versions/3.11/bin/python -P -m pytest -n 2 -m 'not slow' numpy/linalg --timeout=600 --durations=10
============================= test session starts ==============================
platform darwin -- Python 3.11.9, pytest-7.4.0, pluggy-1.6.0
rootdir: /Users/runner/work/numpy/numpy
configfile: pytest.ini
plugins: xdist-3.8.0, timeout-2.4.0, cov-4.1.0, hypothesis-6.142.2
timeout: 600.0s
timeout method: signal
timeout func_only: False
created: 2/2 workers
2 workers [459 items]

..................................x..................................... [ 15%]
........................................................................ [ 31%]
......................................................................... [ 47%]
........................................................................ [ 62%]
....................................................................F.... [ 78%]
.........................................................s...........s.. [ 94%]
.....................x...                                                [100%]
=================================== FAILURES ===================================
____________ TestQR.test_stacked_inputs[float32-outer_size2-size1] _____________
[gw1] darwin -- Python 3.11.9 /Library/Frameworks/Python.framework/Versions/3.11/bin/python

self = <numpy.linalg.tests.test_linalg.TestQR object at 0x106b9abd0>
outer_size = (2, 3, 4), size = (4, 3), dt = <class 'numpy.float32'>

    @pytest.mark.parametrize("size", [
        (3, 4), (4, 3), (4, 4),
        (3, 0), (0, 3)])
    @pytest.mark.parametrize("outer_size", [
        (2, 2), (2,), (2, 3, 4)])
    @pytest.mark.parametrize("dt", [
        np.single, np.double,
        np.csingle, np.cdouble])
    def test_stacked_inputs(self, outer_size, size, dt):
    
        rng = np.random.default_rng(123)
        A = rng.normal(size=outer_size + size).astype(dt)
        B = rng.normal(size=outer_size + size).astype(dt)
>       self.check_qr_stacked(A)

A          = array([[[[[-0.9891214 , -0.36778665,  1.2879252 ],
          [ 0.19397442,  0.9202309 ,  0.5771038 ],
          [-0.63...         [-0.38024592, -1.7452345 , -0.07476547],
          [-0.6747657 , -1.7015473 ,  0.31218666]]]]], dtype=float32)
B          = array([[[[[-1.05055904e+00,  7.18370795e-01, -8.97489429e-01],
          [ 9.77046371e-01,  4.25423503e-01,  1.0860990...72074080e-01, -1.03079081e+00],
          [-8.12349617e-01,  1.14889920e+00,  1.33000448e-01]]]]],
      dtype=float32)
dt         = <class 'numpy.float32'>
outer_size = (2, 3, 4)
rng        = Generator(PCG64) at 0x106BF4200
self       = <numpy.linalg.tests.test_linalg.TestQR object at 0x106b9abd0>
size       = (4, 3)

numpy/linalg/tests/test_linalg.py:1862: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <numpy.linalg.tests.test_linalg.TestQR object at 0x106b9abd0>
a = array([[[[[-0.9891214 , -0.36778665,  1.2879252 ],
          [ 0.19397442,  0.9202309 ,  0.5771038 ],
          [-0.63...         [-0.38024592, -1.7452345 , -0.07476547],
          [-0.6747657 , -1.7015473 ,  0.31218666]]]]], dtype=float32)

    def check_qr_stacked(self, a):
        # This test expects the argument `a` to be an ndarray or
        # a subclass of an ndarray of inexact type.
        a_type = type(a)
        a_dtype = a.dtype
        m, n = a.shape[-2:]
        k = min(m, n)
    
        # mode == 'complete'
        q, r = linalg.qr(a, mode='complete')
        assert_(q.dtype == a_dtype)
        assert_(r.dtype == a_dtype)
        assert_(isinstance(q, a_type))
        assert_(isinstance(r, a_type))
        assert_(q.shape[-2:] == (m, m))
        assert_(r.shape[-2:] == (m, n))
        assert_almost_equal(matmul(q, r), a)
        I_mat = np.identity(q.shape[-1])
        stack_I_mat = np.broadcast_to(I_mat,
                        q.shape[:-2] + (q.shape[-1],) * 2)
>       assert_almost_equal(matmul(swapaxes(q, -1, -2).conj(), q), stack_I_mat)
E       RuntimeWarning: invalid value encountered in matmul

I_mat      = array([[1., 0., 0., 0.],
       [0., 1., 0., 0.],
       [0., 0., 1., 0.],
       [0., 0., 0., 1.]])
a          = array([[[[[-0.9891214 , -0.36778665,  1.2879252 ],
          [ 0.19397442,  0.9202309 ,  0.5771038 ],
          [-0.63...         [-0.38024592, -1.7452345 , -0.07476547],
          [-0.6747657 , -1.7015473 ,  0.31218666]]]]], dtype=float32)
a_dtype    = dtype('float32')
a_type     = <class 'numpy.ndarray'>
k          = 3
m          = 4
n          = 3
q          = array([[[[[-0.800965  ,  0.23105948,  0.45674476,  0.3105653 ],
          [ 0.1570755 , -0.798724  ,  0.38201502,  0.4...46, -0.29233274, -0.7230267 ],
          [-0.6851794 , -0.5540031 , -0.4423038 ,  0.16726339]]]]],
      dtype=float32)
r          = array([[[[[ 1.2349122 ,  0.13444547, -0.3794004 ],
          [ 0.        , -1.1256865 ,  0.18784489],
          [ 0.  ...         [ 0.        ,  0.        ,  0.5484351 ],
          [ 0.        ,  0.        ,  0.        ]]]]], dtype=float32)
self       = <numpy.linalg.tests.test_linalg.TestQR object at 0x106b9abd0>
stack_I_mat = array([[[[[1., 0., 0., 0.],
          [0., 1., 0., 0.],
          [0., 0., 1., 0.],
          [0., 0., 0., 1.]],

    ...],

         [[1., 0., 0., 0.],
          [0., 1., 0., 0.],
          [0., 0., 1., 0.],
          [0., 0., 0., 1.]]]]])

numpy/linalg/tests/test_linalg.py:1824: RuntimeWarning

I don't see any changes in the details of the runner from the start of the CI job in the "Set up job" step across recent CI jobs. It is using macos-14.

and for reference I'm running macos 15.7.1

I'm going to try and loop over my reprod script, see if I can reliably get a failure in a reasonable amount of time.

@neutrinoceros

@neutrinoceros do you have a M4 mac? We added code to ignore FPEs when Acclerate uses SME, but this seems to be something else. @Developer-Ecosystem-Engineering could you take a look?

My mac is an M2. Is it expected to make a difference ?

with gh-29820 being resolved, I retried this on the latest state of main, and I can still reproduce the error.

@neutrinoceros

Thanks @neutrinoceros. I can get it to reproduce as well on a M2 macmini. I think if we are using Accelerate we should always call numpy._core._multiarray_umath._blas_supports_fpe(False). The problem is how to detect Accelerate. Another option is to give up and always call that when we are on macOS arm64.

naive question; what's the expected gain from not always calling Accelerate on this target ?

what's the expected gain from not always calling Accelerate on this target

Not sure I understand. Accelerate gives a performance boost over OpenBLAS, it is supplied by Apple much like MKL is supplied by Intel. Both closed source libraries can match the hardware more optimally than open source solutions can.

In #30102 we assumed the warnings would be reliably emitted by the python-level matmul test. It seems this is not true, the check is flaky and so the FPE status is not ignored. I opened #30255 to always ignore FPE status in BLAS calls when Accelerate is used.

maybe my question didn't make sense; I'm a bit out of my depth here. In any case, thanks for addressing the issue !

Navigation Menu

Repository navigation

TST: a flaky test on macOS arm64 (RuntimeWarning: invalid value encountered in matmul) #30162

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

TST: a flaky test on macOS arm64 (RuntimeWarning: invalid value encountered in matmul) #30162

Description

Activity

neutrinoceros commented on Nov 6, 2025

neutrinoceros commented on Nov 6, 2025

mattip commented on Nov 6, 2025

neutrinoceros commented on Nov 6, 2025

mattip commented on Nov 9, 2025

neutrinoceros commented on Nov 9, 2025

neutrinoceros commented on Nov 17, 2025

mattip commented on Nov 18, 2025

neutrinoceros commented on Nov 18, 2025

mattip commented on Nov 18, 2025

neutrinoceros commented on Nov 18, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions