[BUG]: Legacy driver callback bookkeeping races under free-threaded Python, causing native abort #3282

Activity

1. jpascucci-nv

   added

   bugSomething isn't workingSomething isn't working

   on Oct 29, 2026
2. rparolin

   assigned

   lijinf2

   on Oct 29, 2026
3. rparolin

   added

   P0High priority - Must do!High priority - Must do!

   on Oct 29, 2026
4. rparolin

   added the

   Bug

   issue type on Oct 29, 2026
5. juenglin commented on Oct 29, 2026

   

   juenglin

   on Oct 29, 2026

   Contributor

   More actions

   Another instance of the same thread-unsafety class turned up, this time on Windows and in a different (non-legacy) binding module, so I'm recording it here rather than opening a near-duplicate.

   Observation

   Failing job: Test win-64 / Python 3.14t, CUDA 13.4.2 (wheels), GPU a100 (MCDM)

   cuda_bindings_13/tests/test_cuda.py::test_private_function_pointer_inspector failed under pytest-run-parallel (the test ran in PARALLEL mode):

   PARALLEL FAILED tests/test_cuda.py::test_private_function_pointer_inspector - SystemError: ...\Python-3.14.10\Objects\dictobject.c:2468: bad argument to internal function
   

   The test is a one-liner that just calls into the private inspector:

   def test_private_function_pointer_inspector():
       from cuda.bindings._internal.driver import _inspect_function_pointer
       assert _inspect_function_pointer("__cuGetErrorString") != 0

   The failure is unrelated to the PR whose CI surfaced it; it is a pre-existing latent race, consistent with the "recent parallelism changes expose an older bug" history in the issue description.

   Diagnosis

   Same root pattern as the callback map, but in generated host-API bindings: a module-level mutable global mutated with an unsynchronized check-then-set. In cuda/bindings/_internal/driver_windows.pyx (the same shape exists in driver_linux.pyx and in every other _internal/*.pyx module: runtime, nvrtc, nvjitlink, nvvm, cufile, ...):

   cdef dict _cyb_func_ptrs = None      # module-level global
   
   cpdef dict _inspect_function_pointers():
       global _cyb_func_ptrs
       if _cyb_func_ptrs is not None:
           return _cyb_func_ptrs
       _check_or_init_driver()
       cdef dict data = {}
       # ... populate data["__cu..."] = <intptr_t>__cu... for every entry point ...
       _cyb_func_ptrs = data
       return data
   
   cpdef _inspect_function_pointer(str name):
       global _cyb_func_ptrs
       if _cyb_func_ptrs is None:
           _cyb_func_ptrs = _inspect_function_pointers()
       return _cyb_func_ptrs[name]

   Under free-threaded Python the lazy init races: multiple threads can simultaneously observe _cyb_func_ptrs is None, each build a separate data dict, and assign/read the global concurrently. _cyb_func_ptrs[name] is a cdef dict subscript that Cython lowers to the internal CPython dict C-API; when that API is entered against a dict object being mutated/replaced on another thread without the GIL, it trips the dictobject.c precondition assertion and raises SystemError: bad argument to internal function. The GIL previously made this check-then-set effectively atomic, which is why it only shows up on the free-threaded build.

   So this is the same bug class as the m_global._allocated map in this issue — shared binding state mutated without synchronization — just manifesting as a corrupted/racily-replaced lazy singleton dict in the _inspect_function_pointer* path rather than a std::map::erase abort.

   Suggested fix

   Fix it at the binding-generation source, consistent with the fix proposed here for the callback map:

   • Make the lazy init of _cyb_func_ptrs atomic. The simplest robust option is to populate it eagerly at module import (the data is just a static table of entry-point addresses after _check_or_init_driver()), which removes the racy singleton entirely. If lazy init must be retained, guard the check-then-set with a module-level lock (e.g. a threading.Lock), or build into a local and publish with a single atomic assignment plus a re-check, so readers only ever see a fully-populated dict.
   • Apply the change in the generator so it covers every _internal/*.pyx module and both _linux/_windows variants, since they all share this pattern.
   • Add concurrent regression coverage that calls _inspect_function_pointer from multiple threads.

   Interim CI mitigation (symptom-only, mirrors the "serialize the affected test" suggestion above): mark test_private_function_pointer_inspector with @pytest.mark.thread_unsafe so it is not scheduled in parallel. This unblocks CI but leaves the underlying API thread-unsafe for real callers (e.g. cuda.core._linker calls nvjitlink._inspect_function_pointer("__nvJitLinkVersion")).
6. lijinf2 commented on Oct 30, 2026

   

   lijinf2

   on Oct 30, 2026

   Last edited by lijinf2

   Contributor

   More actions

   Thanks for the detailed report. This looks like two separate problems, but I suspect they share a root cause: both involve a global variable being used by multiple threads at the same time without protection.

   I've got a chance to look at the first one closely and can now reproduce it (Thanks for the detailed info @jpascucci-nv !). The suspected cause is the global m_global._allocated map in the generated driver bindings. cuLogsRegisterCallback inserts into it and cuLogsUnregisterCallback frees and erases from it, with no locking. With the GIL disabled, concurrent threads modify the same std::map, which corrupts the heap.

   Next, I'll add a lock around the map accesses locally to confirm that this makes the crash go away. If it does, I'd like to check with the author who introduced this variable whether a lock-based fix is acceptable.

   Will look into the problem @juenglin mentioned to see whether it has the same cause.
7. lijinf2

   closed this as completedin #3308on Nov 2, 2026
8. Andy-Jost

   added a commit that references this issue on Nov 2, 2026

   Fix crash when registering/unregistering callbacks concurrently under…

   ...

   This issue will close once commit 65935f1 is merged into the 'main' branch.

   65935f1
9. lijinf2

   added this to the cuda.core next milestone on Nov 3, 2026
10. lijinf2 commented on Nov 3, 2026

    

    lijinf2

    on Nov 3, 2026

    Contributor

    More actions

    Need to backport to 12.9 and investigate test_private_function_pointer_inspector
11. lijinf2

    reopened this on Nov 3, 2026
12. rparolin

    modified the milestones: cuda.core next, cuda.bindings 13.5.0 & 12.9.10 on Nov 4, 2026
13. lijinf2 commented on Nov 4, 2026

    

    lijinf2

    on Nov 4, 2026

    Contributor

    More actions

    The Windows failure in test_private_function_pointer_inspector reported above (by @juenglin) is a separate problem (an unsynchronized lazy init, not the cuLogs callback map), so I split it out into #3373.

    This issue now only tracks the cuLogs crash fixed in #3308, with the backport to 12.9 remaining.
14. lijinf2

    added a parent issue on Nov 9, 2026

    • Support the driver error logging system #671
15. rparolin

    modified the milestones: cuda.bindings 13.5.0 & 12.9.10, cuda.bindings next on Nov 10, 2026