Skip to content

BUG: Running test suite in parallel causes ThreadSanitizer data races #30085

New issue
New issue
Closed
Closed
BUG: Running test suite in parallel causes ThreadSanitizer data races#30085
Labels
00 - Bug39 - free-threadingPRs and issues related to support for free-threading CPython (a.k.a. no-GIL, PEP 703)

Description

@bwhitt7
bwhitt7
opened on Oct 28, 2025

Describe the issue:

When running the test suite under pytest-run-parallel and ThreadSanitizer, many tests cause data races. The affected tests (~80 test) are listed in this spreadsheet. Below are some tests I thought were notable.

Tests cause a crash, with no output (some of these overlap with ASan errors I found previously, and likely have to do with the tests crashing due to memory issues):

  • numpy/_core/tests/test_arrayprint.py::test_multithreaded_array_printing
  • numpy/_core/tests/test_multiarray.py::TestCreation::test_zeros_big
  • numpy/_core/tests/test_regression.py::TestRegression::test_to_ctypes
  • numpy/_core/tests/test_shape_base.py::TestConcatenate::test_huge_list_error

Segmentation fault after data race

  • numpy/_core/tests/test_custom_dtypes.py::TestSFloat::test_wrapped_and_wrapped_reductions

RandomState data race (may be a linux-only bug)

  • numpy/linalg/tests/test_linalg.py::TestCond::test_stacked_singular
  • numpy/_core/tests/test_umath.py::TestLog::test_log_strides
  • numpy/linalg/tests/test_linalg.py::TestCond::test_stacked_singular

dec_addstatus data races

  • numpy/_core/tests/test_umath.py::TestRationalFunctions::test_decimal
  • numpy/lib/tests/test_function_base.py::TestAverage::test_object_dtype
  • numpy/lib/tests/test_function_base.py::TestAverage::test_object_no_weights

Caused test failure

  • numpy/lib/tests/test_io.py::TestSaveTxt::test_large_zip
  • numpy/testing/tests/test_utils.py::TestAssertNoGcCycles::test_asserts

Steps to reproduce

  • Use Python 3.14t with ThreadSanitizer enabled.
    • Steps on how to do this can be found here.
    • Currently using these TSAN options export TSAN_OPTIONS=suppressions=/numpy_suppressions.txt:allocator_may_return_null=1:halt_on_error=0
    • Using the suppression file from here
  • Build NumPy using clang, ThreadSanitizer and without openBLAS
    • spin build --clean -- -Db_sanitize=thread -Dbuildtype=debugoptimized -Dallow-noblas=true
  • Run test suite under pytest-run-parallel
    • spin test -p auto -sv

Error message:

I have the traceback files for all TSan warnings I ran into. Don't want to post them all here and make a mess, but I can provide any of them if needed.

Python and NumPy Versions:

NumPy: 2.4.0.dev0+git20251003.da5c632
Python: 3.14.0rc3+ free-threading build (heads/3.14:460ad1aa1ba, Sep 22 2025, 23:51:20) [Clang 18.1.3 (1ubuntu1)]

Runtime Environment:

[{'numpy_version': '2.4.0.dev0+git20251003.da5c632',
'python': '3.14.0rc3+ free-threading build (heads/3.14:460ad1aa1ba, Sep 22 '
'2025, 23:51:20) [Clang 18.1.3 (1ubuntu1)]',
'uname': uname_result(system='Linux', node='SPACE', release='6.6.87.2-microsoft-standard-WSL2', version='#1 SMP PREEMPT_DYNAMIC Thu Jun 5 18:30:46 UTC 2025', machine='x86_64')},
{'simd_extensions': {'baseline': ['SSE', 'SSE2', 'SSE3'],
'found': ['SSSE3',
'SSE41',
'POPCNT',
'SSE42',
'AVX',
'F16C',
'FMA3',
'AVX2'],
'not_found': ['AVX512F',
'AVX512CD',
'AVX512_KNL',
'AVX512_SKX',
'AVX512_CLX',
'AVX512_CNL',
'AVX512_ICL']}},
{'architecture': 'Haswell',
'filepath': '/usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so',
'internal_api': 'openblas',
'num_threads': 24,
'prefix': 'libopenblas',
'threading_layer': 'pthreads',
'user_api': 'blas',
'version': '0.3.26'}]

Context for the issue:

No response

Activity

  1. bwhitt7
    added
    00 - Bug
    on Oct 28, 2025
  2. DasoTD
    added a commit that references this issue on Oct 30, 2025

    Fix ThreadSanitizer data race conditions in NumPy core

    e492d61
  3. jorenham
    added
    39 - free-threadingPRs and issues related to support for free-threading CPython (a.k.a. no-GIL, PEP 703)
    on Nov 16, 2025

  4. kumaraditya303 commented on Nov 17, 2025

    @kumaraditya303
    kumaraditya303
    on Nov 17, 2025
    Contributor

    dec_addstatus data races

    These races are from decimal module in stdlib, linking python/cpython#141148 as the upstream tracking issue.

  5. kumaraditya303 commented on Nov 17, 2025

    @kumaraditya303
    kumaraditya303
    on Nov 17, 2025
    Contributor

    There are a few data races in hashtable implementation as well like:

    ==================
WARNING: ThreadSanitizer: data race (pid=4159)
  Write of size 8 at 0x000300062280 by thread T62:
    #0 PyArrayIdentityHash_SetItem npy_hashtable.cpp:224 (_multiarray_umath.cpython-314t-darwin.so:arm64+0x2e4f20)
    #1 promote_and_get_info_and_ufuncimpl(_tagPyUFuncObject*, tagPyArrayObject* const*, PyArray_DTypeMeta**, PyArray_DTypeMeta**, unsigned char) dispatching.cpp:962 (_multiarray_umath.cpython-314t-darwin.so:arm64+0x2fabc4)
    #2 promote_and_get_ufuncimpl dispatching.cpp:1101 (_multiarray_umath.cpython-314t-darwin.so:arm64+0x2f7fe8)
    #3 ufunc_generic_fastcall ufunc_object.c:4683 (_multiarray_umath.cpython-314t-darwin.so:arm64+0x304050)
    #4 ufunc_generic_vectorcall ufunc_object.c:4766 (_multiarray_umath.cpython-314t-darwin.so:arm64+0x30184c)
    #5 PyObject_Vectorcall <null> (libpython3.14t.dylib:arm64+0x8ab8c)
    #6 _PyEval_EvalFrameDefault <null> (libpython3.14t.dylib:arm64+0x27ce94)
    #7 _PyEval_Vector <null> (libpython3.14t.dylib:arm64+0x278c90)
    #8 _PyFunction_Vectorcall <null> (libpython3.14t.dylib:arm64+0x8b1cc)
    #9 method_vectorcall <null> (libpython3.14t.dylib:arm64+0x8f638)
    #10 _PyObject_Call <null> (libpython3.14t.dylib:arm64+0x8ae40)
    #11 PyObject_Call <null> (libpython3.14t.dylib:arm64+0x8aeb4)
    #12 _PyEval_EvalFrameDefault <null> (libpython3.14t.dylib:arm64+0x27fbc0)
    #13 _PyEval_Vector <null> (libpython3.14t.dylib:arm64+0x278c90)
    #14 _PyFunction_Vectorcall <null> (libpython3.14t.dylib:arm64+0x8b1cc)
    #15 method_vectorcall <null> (libpython3.14t.dylib:arm64+0x8f638)
    #16 context_run <null> (libpython3.14t.dylib:arm64+0x2c3eec)
    #17 _PyEval_EvalFrameDefault <null> (libpython3.14t.dylib:arm64+0x283168)
    #18 _PyEval_Vector <null> (libpython3.14t.dylib:arm64+0x278c90)
    #19 _PyFunction_Vectorcall <null> (libpython3.14t.dylib:arm64+0x8b1cc)
    #20 method_vectorcall <null> (libpython3.14t.dylib:arm64+0x8f638)
    #21 _PyObject_Call <null> (libpython3.14t.dylib:arm64+0x8ae40)
    #22 PyObject_Call <null> (libpython3.14t.dylib:arm64+0x8aeb4)
    #23 thread_run <null> (libpython3.14t.dylib:arm64+0x416b08)
    #24 pythread_wrapper <null> (libpython3.14t.dylib:arm64+0x368cac)

  Previous read of size 8 at 0x000300062280 by thread T64:
    #0 PyArrayIdentityHash_GetItem npy_hashtable.cpp:240 (_multiarray_umath.cpython-314t-darwin.so:arm64+0x2e509c)
    #1 ufunc_generic_fastcall ufunc_object.c:4425 (_multiarray_umath.cpython-314t-darwin.so:arm64+0x3034a4)
    #2 ufunc_generic_vectorcall ufunc_object.c:4766 (_multiarray_umath.cpython-314t-darwin.so:arm64+0x30184c)
    #3 PyObject_Vectorcall <null> (libpython3.14t.dylib:arm64+0x8ab8c)
    #4 _PyEval_EvalFrameDefault <null> (libpython3.14t.dylib:arm64+0x27ce94)
    #5 _PyEval_Vector <null> (libpython3.14t.dylib:arm64+0x278c90)
    #6 _PyFunction_Vectorcall <null> (libpython3.14t.dylib:arm64+0x8b1cc)
    #7 method_vectorcall <null> (libpython3.14t.dylib:arm64+0x8f638)
    #8 _PyObject_Call <null> (libpython3.14t.dylib:arm64+0x8ae40)
    #9 PyObject_Call <null> (libpython3.14t.dylib:arm64+0x8aeb4)
    #10 _PyEval_EvalFrameDefault <null> (libpython3.14t.dylib:arm64+0x27fbc0)
    #11 _PyEval_Vector <null> (libpython3.14t.dylib:arm64+0x278c90)
    #12 _PyFunction_Vectorcall <null> (libpython3.14t.dylib:arm64+0x8b1cc)
    #13 method_vectorcall <null> (libpython3.14t.dylib:arm64+0x8f638)
    #14 context_run <null> (libpython3.14t.dylib:arm64+0x2c3eec)
    #15 _PyEval_EvalFrameDefault <null> (libpython3.14t.dylib:arm64+0x283168)
    #16 _PyEval_Vector <null> (libpython3.14t.dylib:arm64+0x278c90)
    #17 _PyFunction_Vectorcall <null> (libpython3.14t.dylib:arm64+0x8b1cc)
    #18 method_vectorcall <null> (libpython3.14t.dylib:arm64+0x8f638)
    #19 _PyObject_Call <null> (libpython3.14t.dylib:arm64+0x8ae40)
    #20 PyObject_Call <null> (libpython3.14t.dylib:arm64+0x8aeb4)
    #21 thread_run <null> (libpython3.14t.dylib:arm64+0x416b08)
    #22 pythread_wrapper <null> (libpython3.14t.dylib:arm64+0x368cac)


SUMMARY: ThreadSanitizer: data race npy_hashtable.cpp:224 in PyArrayIdentityHash_SetItem
==================

    From the backtrace, the data race happens in ufunc_generic_fastcall which further calls PyArrayIdentityHash_GetItem but it doesn't acquire the read lock of the shared mutex like it is done in promote_and_get_info_and_ufuncimpl_with_locking.

    numpy/numpy/_core/src/umath/ufunc_object.c

    Lines 4338 to 4345 in 02c8c3d

    // so that the the dtype gets registered and things will work next time.
    PyArray_DTypeMeta *op_dtypes[2] = {NPY_DTYPE(dt), NULL};
    PyObject *info = PyArrayIdentityHash_GetItem( // borrowed reference.
    (PyArrayIdentityHash *)ufunc->_dispatch_cache,
    (PyObject **)op_dtypes);
    if (info == NULL) {
    goto bail;
    }

    numpy/numpy/_core/src/umath/dispatching.cpp

    Lines 984 to 991 in 02c8c3d

    std::shared_mutex *mutex = ((std::shared_mutex *)((PyArrayIdentityHash *)ufunc->_dispatch_cache)->mutex);
    NPY_BEGIN_ALLOW_THREADS
    mutex->lock_shared();
    NPY_END_ALLOW_THREADS
    PyObject *info = PyArrayIdentityHash_GetItem(
    (PyArrayIdentityHash *)ufunc->_dispatch_cache,
    (PyObject **)op_dtypes);
    mutex->unlock_shared();

  6. ngoldbaum commented on Nov 18, 2025

    @ngoldbaum
    ngoldbaum
    on Nov 18, 2025
    Member

    Not a data race per se, but I noticed a failure on the pytest-run-parallel CI I haven't seen before: https://github.com/numpy/numpy/actions/runs/19474581693/job/55730908994?pr=30234

    ==================================== ERRORS ====================================
_________________ ERROR at call of TestPickling.test_roundtrip _________________

self = <test_multiarray.TestPickling object at 0x2e0142e8510>

    def test_roundtrip(self):
        for proto in range(2, pickle.HIGHEST_PROTOCOL + 1):
            carray = np.array([[2, 9], [7, 0], [3, 8]])
            DATA = [
                carray,
                np.transpose(carray),
                np.array([('xxx', 1, 2.0)], dtype=[('a', (str, 3)), ('b', int),
                                                   ('c', float)])
            ]
    
            refs = [weakref.ref(a) for a in DATA]
            for a in DATA:
                assert_equal(
                        a, pickle.loads(pickle.dumps(a, protocol=proto)),
                        err_msg=f"{a!r}")
            del a, DATA, carray
            break_cycles()
            # check for reference leaks (gh-12793)
            for ref in refs:
>               assert ref() is None
E               AssertionError: assert array([('xxx', 1, 2.)], dtype=[('a', '<U3'), ('b', '<i8'), ('c', '<f8')]) is None
E                +  where array([('xxx', 1, 2.)], dtype=[('a', '<U3'), ('b', '<i8'), ('c', '<f8')]) = <weakref at 0x2e0b40d4ce0; to 'numpy.ndarray' at 0x2e0b41a0850>()

proto      = 4
ref        = <weakref at 0x2e0b40d4ce0; to 'numpy.ndarray' at 0x2e0b41a0850>
refs       = [<weakref at 0x2e0b40d4c20; dead>, <weakref at 0x2e0b40d2ca0; dead>, <weakref at 0x2e0b40d4ce0; to 'numpy.ndarray' at 0x2e0b41a0850>]
self       = <test_multiarray.TestPickling object at 0x2e0142e8510>

numpy/_core/tests/test_multiarray.py:4674: AssertionError

    That test is checking explicitly for reference leaks, so perhaps it's thread-unsafe because there's a (transient) reference from a global or something along those lines and no one noticed until now.

  7. kumaraditya303 commented on Nov 18, 2025

    @kumaraditya303
    kumaraditya303
    on Nov 18, 2025
    Contributor

    That test is checking explicitly for reference leaks, so perhaps it's thread-unsafe because there's a (transient) reference from a global or something along those lines and no one noticed until now.

    I think it is because the test relies on break_cycles to clear references which just calls gc.collect(). If another thread has requested gc then that function is a no-op and doesn't break cycles. See python/cpython#137682

  8. ngoldbaum commented on Nov 18, 2025

    @ngoldbaum
    ngoldbaum
    on Nov 18, 2025
    Member

    So it's a rare race, we should mark it as thread-unsafe. Thanks for taking a look.

  9. ngoldbaum commented on Dec 2, 2025

    @ngoldbaum
    ngoldbaum
    on Dec 2, 2025
    Member

    I looked a little closer at the numpy.random races. I think we need to use critical sections in the Cython NumPy RNG implementation to make sure no new races are possible on the free-threaded build.

    Basically, whenever the internal state of the RNG is read from or mutated, we need to take a critical section. Here's how I modified RandomState.set_state:

    diff --git a/numpy/random/mtrand.pyx b/numpy/random/mtrand.pyx
index 8e7f437641..55cc7a3fd3 100644
--- a/numpy/random/mtrand.pyx
+++ b/numpy/random/mtrand.pyx
@@ -381,9 +381,13 @@ def set_state(self, state):
                     st['has_gauss'] = state[3]
                     st['gauss'] = state[4]

-        self._aug_state.gauss = st.get('gauss', 0.0)
-        self._aug_state.has_gauss = st.get('has_gauss', 0)
-        self._bit_generator.state = st
+        cdef double gauss = st.get('gauss', 0.0)
+        cdef int has_gauss = st.get('has_gauss', 0)
+
+        with cython.crticial_section(self):
+            self._aug_state.gauss = gauss
+            self._aug_state.has_gauss = has_gauss
+            self._bit_generator.state = st

    Moving the st.get calls outside of the critical section avoids calling back into the CPython C API while the critical section is held.

    Now I need to audit the rest of the Cython code to find all the spots the internal state is read from or modified...

  10. kumaraditya303 commented on Dec 11, 2025

    @kumaraditya303
    kumaraditya303
    on Dec 11, 2025
    Contributor

    I ran the full test suite under TSAN and all of the data races inside numpy are fixed now. There are three remaining data races which happens when running under TSAN:

    • partial_vectorcall_fallback
    ==================
WARNING: ThreadSanitizer: data race (pid=46428)
  Read of size 8 at 0x0003023a4978 by thread T449:
    #0 object_vacall call.c:819 (libpython3.14t.dylib:arm64+0x8d0e8)
    #1 PyObject_CallFunctionObjArgs call.c:926 (libpython3.14t.dylib:arm64+0x8d498)
    #2 array_str strfuncs.c:66 (_multiarray_umath.cpython-314t-darwin.so:arm64+0x166d50)
    #3 unicode_vectorcall unicodeobject.c:15640 (libpython3.14t.dylib:arm64+0x1ea12c)
    #4 PyObject_Vectorcall call.c:327 (libpython3.14t.dylib:arm64+0x8af00)
    #5 _PyEval_EvalFrameDefault generated_cases.c.h:1619 (libpython3.14t.dylib:arm64+0x27caa4)
    #6 _PyEval_Vector ceval.c:2083 (libpython3.14t.dylib:arm64+0x2787cc)
    #7 _PyFunction_Vectorcall call.c (libpython3.14t.dylib:arm64+0x8b54c)
    #8 method_vectorcall classobject.c:65 (libpython3.14t.dylib:arm64+0x8f920)
    #9 _PyObject_Call call.c:348 (libpython3.14t.dylib:arm64+0x8b100)
    #10 PyObject_Call call.c:373 (libpython3.14t.dylib:arm64+0x8b234)
    #11 _PyEval_EvalFrameDefault generated_cases.c.h:2654 (libpython3.14t.dylib:arm64+0x27f854)
    #12 _PyEval_Vector ceval.c:2083 (libpython3.14t.dylib:arm64+0x2787cc)
    #13 _PyFunction_Vectorcall call.c (libpython3.14t.dylib:arm64+0x8b54c)
    #14 method_vectorcall classobject.c:73 (libpython3.14t.dylib:arm64+0x8f9c0)
    #15 context_run context.c:728 (libpython3.14t.dylib:arm64+0x2c3bbc)
    #16 _PyEval_EvalFrameDefault generated_cases.c.h:3744 (libpython3.14t.dylib:arm64+0x282b08)
    #17 _PyEval_Vector ceval.c:2083 (libpython3.14t.dylib:arm64+0x2787cc)
    #18 _PyFunction_Vectorcall call.c (libpython3.14t.dylib:arm64+0x8b54c)
    #19 method_vectorcall classobject.c:73 (libpython3.14t.dylib:arm64+0x8f9c0)
    #20 _PyObject_Call call.c:348 (libpython3.14t.dylib:arm64+0x8b1c0)
    #21 PyObject_Call call.c:373 (libpython3.14t.dylib:arm64+0x8b234)
    #22 thread_run _threadmodule.c:359 (libpython3.14t.dylib:arm64+0x4199b4)
    #23 pythread_wrapper thread_pthread.h:242 (libpython3.14t.dylib:arm64+0x369898)

  Previous write of size 8 at 0x0003023a4978 by thread T452:
    #0 partial_vectorcall_fallback _functoolsmodule.c:378 (libpython3.14t.dylib:arm64+0x428380)
    #1 partial_vectorcall _functoolsmodule.c:393 (libpython3.14t.dylib:arm64+0x427cc0)
    #2 object_vacall call.c:819 (libpython3.14t.dylib:arm64+0x8d104)
    #3 PyObject_CallFunctionObjArgs call.c:926 (libpython3.14t.dylib:arm64+0x8d498)
    #4 array_str strfuncs.c:66 (_multiarray_umath.cpython-314t-darwin.so:arm64+0x166d50)
    #5 unicode_vectorcall unicodeobject.c:15640 (libpython3.14t.dylib:arm64+0x1ea12c)
    #6 PyObject_Vectorcall call.c:327 (libpython3.14t.dylib:arm64+0x8af00)
    #7 _PyEval_EvalFrameDefault generated_cases.c.h:1619 (libpython3.14t.dylib:arm64+0x27caa4)
    #8 _PyEval_Vector ceval.c:2083 (libpython3.14t.dylib:arm64+0x2787cc)
    #9 _PyFunction_Vectorcall call.c (libpython3.14t.dylib:arm64+0x8b54c)
    #10 method_vectorcall classobject.c:65 (libpython3.14t.dylib:arm64+0x8f920)
    #11 _PyObject_Call call.c:348 (libpython3.14t.dylib:arm64+0x8b100)
    #12 PyObject_Call call.c:373 (libpython3.14t.dylib:arm64+0x8b234)
    #13 _PyEval_EvalFrameDefault generated_cases.c.h:2654 (libpython3.14t.dylib:arm64+0x27f854)
    #14 _PyEval_Vector ceval.c:2083 (libpython3.14t.dylib:arm64+0x2787cc)
    #15 _PyFunction_Vectorcall call.c (libpython3.14t.dylib:arm64+0x8b54c)
    #16 method_vectorcall classobject.c:73 (libpython3.14t.dylib:arm64+0x8f9c0)
    #17 context_run context.c:728 (libpython3.14t.dylib:arm64+0x2c3bbc)
    #18 _PyEval_EvalFrameDefault generated_cases.c.h:3744 (libpython3.14t.dylib:arm64+0x282b08)
    #19 _PyEval_Vector ceval.c:2083 (libpython3.14t.dylib:arm64+0x2787cc)
    #20 _PyFunction_Vectorcall call.c (libpython3.14t.dylib:arm64+0x8b54c)
    #21 method_vectorcall classobject.c:73 (libpython3.14t.dylib:arm64+0x8f9c0)
    #22 _PyObject_Call call.c:348 (libpython3.14t.dylib:arm64+0x8b1c0)
    #23 PyObject_Call call.c:373 (libpython3.14t.dylib:arm64+0x8b234)
    #24 thread_run _threadmodule.c:359 (libpython3.14t.dylib:arm64+0x4199b4)
    #25 pythread_wrapper thread_pthread.h:242 (libpython3.14t.dylib:arm64+0x369898)

    This is already fixed in CPython upstream main branch but was fixed as a result of performance optimization so will not be backported to 3.14.

    • dec_addstatus
    ==================
WARNING: ThreadSanitizer: data race (pid=46428)
  Write of size 4 at 0x0003012c41bc by thread T17600:
    #0 dec_addstatus _decimal.c:605 (_decimal.cpython-314t-darwin.so:arm64+0x98e4)
    #1 PyDecType_FromLongExact _decimal.c:2444 (_decimal.cpython-314t-darwin.so:arm64+0x9c2c)
    #2 dec_new _decimal.c:3070 (_decimal.cpython-314t-darwin.so:arm64+0x4ee8)
    #3 type_call typeobject.c:2309 (libpython3.14t.dylib:arm64+0x18bb84)
    #4 _PyObject_MakeTpCall call.c:242 (libpython3.14t.dylib:arm64+0x8a3ac)
    #5 PyObject_Vectorcall call.c:327 (libpython3.14t.dylib:arm64+0x8af9c)
    #6 _PyEval_EvalFrameDefault generated_cases.c.h:1619 (libpython3.14t.dylib:arm64+0x27caa4)
    #7 _PyEval_Vector ceval.c:2083 (libpython3.14t.dylib:arm64+0x2787cc)
    #8 _PyFunction_Vectorcall call.c (libpython3.14t.dylib:arm64+0x8b54c)
    #9 method_vectorcall classobject.c:73 (libpython3.14t.dylib:arm64+0x8f9c0)
    #10 _PyObject_Call call.c:348 (libpython3.14t.dylib:arm64+0x8b1c0)
    #11 PyObject_Call call.c:373 (libpython3.14t.dylib:arm64+0x8b234)
    #12 _PyEval_EvalFrameDefault generated_cases.c.h:2654 (libpython3.14t.dylib:arm64+0x27f854)
    #13 _PyEval_Vector ceval.c:2083 (libpython3.14t.dylib:arm64+0x2787cc)
    #14 _PyFunction_Vectorcall call.c (libpython3.14t.dylib:arm64+0x8b54c)
    #15 method_vectorcall classobject.c:73 (libpython3.14t.dylib:arm64+0x8f9c0)
    #16 context_run context.c:728 (libpython3.14t.dylib:arm64+0x2c3bbc)
    #17 _PyEval_EvalFrameDefault generated_cases.c.h:3744 (libpython3.14t.dylib:arm64+0x282b08)
    #18 _PyEval_Vector ceval.c:2083 (libpython3.14t.dylib:arm64+0x2787cc)
    #19 _PyFunction_Vectorcall call.c (libpython3.14t.dylib:arm64+0x8b54c)
    #20 method_vectorcall classobject.c:73 (libpython3.14t.dylib:arm64+0x8f9c0)
    #21 _PyObject_Call call.c:348 (libpython3.14t.dylib:arm64+0x8b1c0)
    #22 PyObject_Call call.c:373 (libpython3.14t.dylib:arm64+0x8b234)
    #23 thread_run _threadmodule.c:359 (libpython3.14t.dylib:arm64+0x4199b4)
    #24 pythread_wrapper thread_pthread.h:242 (libpython3.14t.dylib:arm64+0x369898)

  Previous write of size 4 at 0x0003012c41bc by thread T17597:
    #0 dec_addstatus _decimal.c:605 (_decimal.cpython-314t-darwin.so:arm64+0x98e4)
    #1 PyDecType_FromLongExact _decimal.c:2444 (_decimal.cpython-314t-darwin.so:arm64+0x9c2c)
    #2 dec_new _decimal.c:3070 (_decimal.cpython-314t-darwin.so:arm64+0x4ee8)
    #3 type_call typeobject.c:2309 (libpython3.14t.dylib:arm64+0x18bb84)
    #4 _PyObject_MakeTpCall call.c:242 (libpython3.14t.dylib:arm64+0x8a3ac)
    #5 PyObject_Vectorcall call.c:327 (libpython3.14t.dylib:arm64+0x8af9c)
    #6 _PyEval_EvalFrameDefault generated_cases.c.h:1619 (libpython3.14t.dylib:arm64+0x27caa4)
    #7 _PyEval_Vector ceval.c:2083 (libpython3.14t.dylib:arm64+0x2787cc)
    #8 _PyFunction_Vectorcall call.c (libpython3.14t.dylib:arm64+0x8b54c)
    #9 method_vectorcall classobject.c:73 (libpython3.14t.dylib:arm64+0x8f9c0)
    #10 _PyObject_Call call.c:348 (libpython3.14t.dylib:arm64+0x8b1c0)
    #11 PyObject_Call call.c:373 (libpython3.14t.dylib:arm64+0x8b234)
    #12 _PyEval_EvalFrameDefault generated_cases.c.h:2654 (libpython3.14t.dylib:arm64+0x27f854)
    #13 _PyEval_Vector ceval.c:2083 (libpython3.14t.dylib:arm64+0x2787cc)
    #14 _PyFunction_Vectorcall call.c (libpython3.14t.dylib:arm64+0x8b54c)
    #15 method_vectorcall classobject.c:73 (libpython3.14t.dylib:arm64+0x8f9c0)
    #16 context_run context.c:728 (libpython3.14t.dylib:arm64+0x2c3bbc)
    #17 _PyEval_EvalFrameDefault generated_cases.c.h:3744 (libpython3.14t.dylib:arm64+0x282b08)
    #18 _PyEval_Vector ceval.c:2083 (libpython3.14t.dylib:arm64+0x2787cc)
    #19 _PyFunction_Vectorcall call.c (libpython3.14t.dylib:arm64+0x8b54c)
    #20 method_vectorcall classobject.c:73 (libpython3.14t.dylib:arm64+0x8f9c0)
    #21 _PyObject_Call call.c:348 (libpython3.14t.dylib:arm64+0x8b1c0)
    #22 PyObject_Call call.c:373 (libpython3.14t.dylib:arm64+0x8b234)
    #23 thread_run _threadmodule.c:359 (libpython3.14t.dylib:arm64+0x4199b4)
    #24 pythread_wrapper thread_pthread.h:242 (libpython3.14t.dylib:arm64+0x369898)

    This is tracked upstream in python/cpython#141148 and I am looking into this.

    • __Pyx_AddTraceback
    ==================
WARNING: ThreadSanitizer: data race (pid=68622)
  Read of size 8 at 0x00032a1d0a98 by thread T3206:
    #0 PyFrame_New frameobject.c:2137 (python.exe:arm64+0x10012ae18)
    #1 __Pyx_AddTraceback mtrand.pyx.c:35760 (mtrand.cpython-314t-darwin.so:arm64+0x73c4)
    #2 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x1000daa4c)
    #3 _PyEval_EvalFrameDefault generated_cases.c.h:1619 (python.exe:arm64+0x10034df40)
    #4 _PyEval_Vector ceval.c:2083 (python.exe:arm64+0x10032542c)
    #5 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000daf64)
    #6 _PyObject_VectorcallTstate pycore_call.h:169 (python.exe:arm64+0x1000e0cf8)
    #7 method_vectorcall classobject.c:73 (python.exe:arm64+0x1000df0b8)
    #8 context_run context.c:728 (python.exe:arm64+0x100395d64)
    #9 _PyEval_EvalFrameDefault generated_cases.c.h:3744 (python.exe:arm64+0x10034c1b4)
    #10 _PyEval_Vector ceval.c:2083 (python.exe:arm64+0x10032542c)
    #11 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000daf64)
    #12 _PyObject_VectorcallTstate pycore_call.h:169 (python.exe:arm64+0x1000e0cf8)
    #13 method_vectorcall classobject.c:73 (python.exe:arm64+0x1000df0b8)
    #14 _PyVectorcall_Call call.c:273 (python.exe:arm64+0x1000da900)
    #15 _PyObject_Call call.c:348 (python.exe:arm64+0x1000dab40)
    #16 PyObject_Call call.c:373 (python.exe:arm64+0x1000dad50)
    #17 thread_run _threadmodule.c:359 (python.exe:arm64+0x100546b0c)
    #18 pythread_wrapper thread_pthread.h:242 (python.exe:arm64+0x1004759b4)

  Previous write of size 8 at 0x00032a1d0a98 by thread T3208:
    #0 _PyUnicode_InternMortal unicodeobject.c:16082 (python.exe:arm64+0x100278620)
    #1 _PyCode_New codeobject.c:748 (python.exe:arm64+0x1000e26b8)
    #2 PyCode_NewEmpty codeobject.c:1000 (python.exe:arm64+0x1000e3e20)
    #3 __Pyx_AddTraceback mtrand.pyx.c:35747 (mtrand.cpython-314t-darwin.so:arm64+0x7230)
    #4 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x1000daa4c)
    #5 _PyEval_EvalFrameDefault generated_cases.c.h:1619 (python.exe:arm64+0x10034df40)
    #6 _PyEval_Vector ceval.c:2083 (python.exe:arm64+0x10032542c)
    #7 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000daf64)
    #8 _PyObject_VectorcallTstate pycore_call.h:169 (python.exe:arm64+0x1000e0cf8)
    #9 method_vectorcall classobject.c:73 (python.exe:arm64+0x1000df0b8)
    #10 context_run context.c:728 (python.exe:arm64+0x100395d64)
    #11 _PyEval_EvalFrameDefault generated_cases.c.h:3744 (python.exe:arm64+0x10034c1b4)
    #12 _PyEval_Vector ceval.c:2083 (python.exe:arm64+0x10032542c)
    #13 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000daf64)
    #14 _PyObject_VectorcallTstate pycore_call.h:169 (python.exe:arm64+0x1000e0cf8)
    #15 method_vectorcall classobject.c:73 (python.exe:arm64+0x1000df0b8)
    #16 _PyVectorcall_Call call.c:273 (python.exe:arm64+0x1000da900)
    #17 _PyObject_Call call.c:348 (python.exe:arm64+0x1000dab40)
    #18 PyObject_Call call.c:373 (python.exe:arm64+0x1000dad50)
    #19 thread_run _threadmodule.c:359 (python.exe:arm64+0x100546b0c)
    #20 pythread_wrapper thread_pthread.h:242 (python.exe:arm64+0x1004759b4)

    This is a cython bug and is tracked upstream in cython/cython#7402

  11. ngoldbaum commented on Dec 11, 2025

    @ngoldbaum
    ngoldbaum
    on Dec 11, 2025
    Member

    Awesome, given that and that there are upstream issues to track all the remaining races, let's close this.

    It's still far too slow to add this to CI, so people will only do this as a one-off for now. Let's not mess with the TSan suppression file until someone needs this, on the theory that the upstream issues will eventually be fixed.

  12. ngoldbaum
    closed this as completedon Dec 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    00 - Bug39 - free-threadingPRs and issues related to support for free-threading CPython (a.k.a. no-GIL, PEP 703)

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions