Repository navigation
[BUG] data races in code object caching and __Pyx_AddTraceback #7402
Description
Describe the bug
While running the tests of numpy under TSAN the following data races are reported:
WARNING: ThreadSanitizer: data race (pid=68622)
Atomic read of size 8 at 0x0003202908e0 by thread T2129:
#0 _Py_DecRefShared object.c:430 (python.exe:arm64+0x1001a8944)
#1 __Pyx_AddTraceback mtrand.pyx.c:35758 (mtrand.cpython-314t-darwin.so:arm64+0x7574)
#2 method_vectorcall classobject.c:65 (python.exe:arm64+0x1000df080)
#3 _PyVectorcall_Call call.c:285 (python.exe:arm64+0x1000da8ac)
#4 _PyObject_Call call.c:348 (python.exe:arm64+0x1000dab40)
#5 PyObject_Call call.c:373 (python.exe:arm64+0x1000dad50)
#6 _PyEval_EvalFrameDefault generated_cases.c.h:2654 (python.exe:arm64+0x10033fb70)
#7 _PyEval_Vector ceval.c:2083 (python.exe:arm64+0x10032542c)
#8 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000daf64)
#9 _PyObject_VectorcallTstate pycore_call.h:169 (python.exe:arm64+0x1000e0cf8)
#10 method_vectorcall classobject.c:65 (python.exe:arm64+0x1000df080)
#11 _PyVectorcall_Call call.c:285 (python.exe:arm64+0x1000da8ac)
#12 _PyObject_Call call.c:348 (python.exe:arm64+0x1000dab40)
#13 PyObject_Call call.c:373 (python.exe:arm64+0x1000dad50)
#14 _PyEval_EvalFrameDefault generated_cases.c.h:2654 (python.exe:arm64+0x10033fb70)
#15 _PyEval_Vector ceval.c:2083 (python.exe:arm64+0x10032542c)
#16 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000daf64)
#17 _PyObject_VectorcallTstate pycore_call.h:169 (python.exe:arm64+0x1000e0cf8)
#18 method_vectorcall classobject.c:73 (python.exe:arm64+0x1000df0b8)
#19 _PyVectorcall_Call call.c:273 (python.exe:arm64+0x1000da900)
#20 _PyObject_Call call.c:348 (python.exe:arm64+0x1000dab40)
#21 PyObject_Call call.c:373 (python.exe:arm64+0x1000dad50)
#22 _PyEval_EvalFrameDefault generated_cases.c.h:2654 (python.exe:arm64+0x10033fb70)
#23 _PyEval_Vector ceval.c:2083 (python.exe:arm64+0x10032542c)
#24 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000daf64)
#25 _PyObject_VectorcallTstate pycore_call.h:169 (python.exe:arm64+0x1000e0cf8)
#26 method_vectorcall classobject.c:73 (python.exe:arm64+0x1000df0b8)
#27 context_run context.c:728 (python.exe:arm64+0x100395d64)
#28 _PyEval_EvalFrameDefault generated_cases.c.h:3744 (python.exe:arm64+0x10034c1b4)
#29 _PyEval_Vector ceval.c:2083 (python.exe:arm64+0x10032542c)
#30 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000daf64)
#31 _PyObject_VectorcallTstate pycore_call.h:169 (python.exe:arm64+0x1000e0cf8)
#32 method_vectorcall classobject.c:73 (python.exe:arm64+0x1000df0b8)
#33 _PyVectorcall_Call call.c:273 (python.exe:arm64+0x1000da900)
#34 _PyObject_Call call.c:348 (python.exe:arm64+0x1000dab40)
#35 PyObject_Call call.c:373 (python.exe:arm64+0x1000dad50)
#36 thread_run _threadmodule.c:359 (python.exe:arm64+0x100546b0c)
#37 pythread_wrapper thread_pthread.h:242 (python.exe:arm64+0x1004759b4)
Previous write of size 8 at 0x0003202908e0 by thread T2130:
#0 _PyObject_SetDeferredRefcount object.c:2585 (python.exe:arm64+0x1001af9f8)
#1 _PyObject_AssignUniqueId uniqueid.c:96 (python.exe:arm64+0x10047f3d8)
#2 _PyCode_New codeobject.c:754 (python.exe:arm64+0x1000e2d8c)
#3 PyCode_NewEmpty codeobject.c:1000 (python.exe:arm64+0x1000e3e20)
#4 __Pyx_AddTraceback mtrand.pyx.c:35747 (mtrand.cpython-314t-darwin.so:arm64+0x7230)
#5 method_vectorcall classobject.c:65 (python.exe:arm64+0x1000df080)
#6 _PyVectorcall_Call call.c:285 (python.exe:arm64+0x1000da8ac)
#7 _PyObject_Call call.c:348 (python.exe:arm64+0x1000dab40)
#8 PyObject_Call call.c:373 (python.exe:arm64+0x1000dad50)
#9 _PyEval_EvalFrameDefault generated_cases.c.h:2654 (python.exe:arm64+0x10033fb70)
#10 _PyEval_Vector ceval.c:2083 (python.exe:arm64+0x10032542c)
#11 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000daf64)
#12 _PyObject_VectorcallTstate pycore_call.h:169 (python.exe:arm64+0x1000e0cf8)
#13 method_vectorcall classobject.c:65 (python.exe:arm64+0x1000df080)
#14 _PyVectorcall_Call call.c:285 (python.exe:arm64+0x1000da8ac)
#15 _PyObject_Call call.c:348 (python.exe:arm64+0x1000dab40)
#16 PyObject_Call call.c:373 (python.exe:arm64+0x1000dad50)
#17 _PyEval_EvalFrameDefault generated_cases.c.h:2654 (python.exe:arm64+0x10033fb70)
#18 _PyEval_Vector ceval.c:2083 (python.exe:arm64+0x10032542c)
#19 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000daf64)
#20 _PyObject_VectorcallTstate pycore_call.h:169 (python.exe:arm64+0x1000e0cf8)
#21 method_vectorcall classobject.c:73 (python.exe:arm64+0x1000df0b8)
#22 _PyVectorcall_Call call.c:273 (python.exe:arm64+0x1000da900)
#23 _PyObject_Call call.c:348 (python.exe:arm64+0x1000dab40)
#24 PyObject_Call call.c:373 (python.exe:arm64+0x1000dad50)
#25 _PyEval_EvalFrameDefault generated_cases.c.h:2654 (python.exe:arm64+0x10033fb70)
#26 _PyEval_Vector ceval.c:2083 (python.exe:arm64+0x10032542c)
#27 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000daf64)
#28 _PyObject_VectorcallTstate pycore_call.h:169 (python.exe:arm64+0x1000e0cf8)
#29 method_vectorcall classobject.c:73 (python.exe:arm64+0x1000df0b8)
#30 context_run context.c:728 (python.exe:arm64+0x100395d64)
#31 _PyEval_EvalFrameDefault generated_cases.c.h:3744 (python.exe:arm64+0x10034c1b4)
#32 _PyEval_Vector ceval.c:2083 (python.exe:arm64+0x10032542c)
#33 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000daf64)
#34 _PyObject_VectorcallTstate pycore_call.h:169 (python.exe:arm64+0x1000e0cf8)
#35 method_vectorcall classobject.c:73 (python.exe:arm64+0x1000df0b8)
#36 _PyVectorcall_Call call.c:273 (python.exe:arm64+0x1000da900)
#37 _PyObject_Call call.c:348 (python.exe:arm64+0x1000dab40)
#38 PyObject_Call call.c:373 (python.exe:arm64+0x1000dad50)
#39 thread_run _threadmodule.c:359 (python.exe:arm64+0x100546b0c)
SUMMARY: ThreadSanitizer: data race object.c:430 in _Py_DecRefShared
==================
==================
WARNING: ThreadSanitizer: data race (pid=68622)
Read of size 8 at 0x00032a1d0a98 by thread T3206:
#0 PyFrame_New frameobject.c:2137 (python.exe:arm64+0x10012ae18)
#1 __Pyx_AddTraceback mtrand.pyx.c:35760 (mtrand.cpython-314t-darwin.so:arm64+0x73c4)
#2 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x1000daa4c)
#3 _PyEval_EvalFrameDefault generated_cases.c.h:1619 (python.exe:arm64+0x10034df40)
#4 _PyEval_Vector ceval.c:2083 (python.exe:arm64+0x10032542c)
#5 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000daf64)
#6 _PyObject_VectorcallTstate pycore_call.h:169 (python.exe:arm64+0x1000e0cf8)
#7 method_vectorcall classobject.c:73 (python.exe:arm64+0x1000df0b8)
#8 context_run context.c:728 (python.exe:arm64+0x100395d64)
#9 _PyEval_EvalFrameDefault generated_cases.c.h:3744 (python.exe:arm64+0x10034c1b4)
#10 _PyEval_Vector ceval.c:2083 (python.exe:arm64+0x10032542c)
#11 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000daf64)
#12 _PyObject_VectorcallTstate pycore_call.h:169 (python.exe:arm64+0x1000e0cf8)
#13 method_vectorcall classobject.c:73 (python.exe:arm64+0x1000df0b8)
#14 _PyVectorcall_Call call.c:273 (python.exe:arm64+0x1000da900)
#15 _PyObject_Call call.c:348 (python.exe:arm64+0x1000dab40)
#16 PyObject_Call call.c:373 (python.exe:arm64+0x1000dad50)
#17 thread_run _threadmodule.c:359 (python.exe:arm64+0x100546b0c)
#18 pythread_wrapper thread_pthread.h:242 (python.exe:arm64+0x1004759b4)
Previous write of size 8 at 0x00032a1d0a98 by thread T3208:
#0 _PyUnicode_InternMortal unicodeobject.c:16082 (python.exe:arm64+0x100278620)
#1 _PyCode_New codeobject.c:748 (python.exe:arm64+0x1000e26b8)
#2 PyCode_NewEmpty codeobject.c:1000 (python.exe:arm64+0x1000e3e20)
#3 __Pyx_AddTraceback mtrand.pyx.c:35747 (mtrand.cpython-314t-darwin.so:arm64+0x7230)
#4 PyObject_Vectorcall call.c:327 (python.exe:arm64+0x1000daa4c)
#5 _PyEval_EvalFrameDefault generated_cases.c.h:1619 (python.exe:arm64+0x10034df40)
#6 _PyEval_Vector ceval.c:2083 (python.exe:arm64+0x10032542c)
#7 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000daf64)
#8 _PyObject_VectorcallTstate pycore_call.h:169 (python.exe:arm64+0x1000e0cf8)
#9 method_vectorcall classobject.c:73 (python.exe:arm64+0x1000df0b8)
#10 context_run context.c:728 (python.exe:arm64+0x100395d64)
#11 _PyEval_EvalFrameDefault generated_cases.c.h:3744 (python.exe:arm64+0x10034c1b4)
#12 _PyEval_Vector ceval.c:2083 (python.exe:arm64+0x10032542c)
#13 _PyFunction_Vectorcall call.c (python.exe:arm64+0x1000daf64)
#14 _PyObject_VectorcallTstate pycore_call.h:169 (python.exe:arm64+0x1000e0cf8)
#15 method_vectorcall classobject.c:73 (python.exe:arm64+0x1000df0b8)
#16 _PyVectorcall_Call call.c:273 (python.exe:arm64+0x1000da900)
#17 _PyObject_Call call.c:348 (python.exe:arm64+0x1000dab40)
#18 PyObject_Call call.c:373 (python.exe:arm64+0x1000dad50)
#19 thread_run _threadmodule.c:359 (python.exe:arm64+0x100546b0c)
#20 pythread_wrapper thread_pthread.h:242 (python.exe:arm64+0x1004759b4)
SUMMARY: ThreadSanitizer: data race frameobject.c:2137 in PyFrame_New
==================Code to reproduce the behaviour:
Run the tests of numpy using pytest-run-parallel as described in numpy/numpy#30085
Expected behaviour
No data races
OS
All
Python version
3.14t
Cython version
No response
Additional context
No response
Activity
da-woods commentedon Dec 11, 2025
Based on a superficial skim-read, this doesn't feel like a Cython bug.
E.g. one of the errors is from
PyCode_NewEmptywhich really should be reentrant. I'm fairly sure theconst char*s we pass in to it are genuinely const (but I'll double-check that later). I'm struggling to see how we should be able to create a data race involving it.da-woods commentedon Dec 12, 2025
I need to investigate this a bit more on Numpy but I can reproduce this on a toy example only if I don't instrument the Cython module. For reference, toy example:
and runner:
My current view is that there are suitable load-acquire barriers between the frame being generated in
PyCode_NewEmptyand being used by other threads inPyFrame_New. However, if the Cython module is built without thread-sanitizer then they're invisible and so you get this error. I've consistently failed to reproduce it when building the Cython module with thread-sanitizer.I'll have a look at the full Numpy example at some point later to see if I can confirm my suspicions, but that's the current theory.
da-woods commentedon Dec 12, 2025
Another possibility is that it's being built with atomics that thread sanitizer can't see properly (possibly the old GCC
syncatomics). I can't make this fail on x64 Linux, but it's possible that OS X on arm is slightly different.If that was the case, it'd be an argument for having slightly more modern fallback atomics.
kumaraditya303 commentedon Dec 12, 2025
I had tested it on arm64 mac, haven't tested on linux so that is possible.
da-woods commentedon Dec 18, 2025
I didn't manage to reproduce this when running
spin test -p auto -t numpy/random/tests -v(or at least didn't see any thread sanitizer complaints in the output - it's possible they're going somewhere else...). Running the full test suite is too slow for me unfortunately.I still think it's most likely an issue with thread sanitizer not seeing the Cython atomics rather than a genuine thread-safety issue.
Would it be possible to build numpy with
CFLAGS="-D__PYX_DEBUG_ATOMICS=1" spin build ...?There's no need to run numpy though - all I'm interesting in is some warnings that get printed during the build - something like:
It should also be the same all through, so once you've seen one there's no need to continue.
ngoldbaum commentedon Jan 6, 2026
Sorry for taking so long to get to this. Only a week later than I promised on Discord 🙃.
I think you'd need to be using pytest-run-parallel to see these reports. Here you're using pytest-xdist, which is based on multiprocessing.
You should use a TSan-instrumented build of CPython and then build NumPy and run the np.random tests like so:
See https://py-free-threading.github.io/thread_sanitizer/ for more tips.
All that said, I don't see any TSan reports doing that over here on numpy
mainand a CPython compiled frommaintoday using pyenv. @kumaraditya303 is there any chance you can try to reproduce this on your end? If you're able to, can you share more detail about how you set up your environment and what you did to trigger the issue? It'd help if you could share the specific test that is triggering the race to help narrow down where to look.