Skip to content

TSAN race in //tests:shard_map_test_cpu #26359

Closed

Description

@hawkinsp

Description

From https://github.com/jax-ml/jax/actions/runs/13172208815/job/36764479206

I suspect this is a CPython bug, I'll try to make a reproduction and file it upstream.

WARNING: ThreadSanitizer: data race (pid=208275)
  Write of size 8 at 0x555555d43b60 by thread T121:
    #0 grow_thread_array /__w/jax/jax/cpython/Python/qsbr.c:101:19 (python3.13+0x4a3905) (BuildId: 8f8869b5f3143bd14dda26aa2bf37336b4902370)
    #1 _Py_qsbr_reserve /__w/jax/jax/cpython/Python/qsbr.c:203:13 (python3.13+0x4a3905)
    #2 new_threadstate /__w/jax/jax/cpython/Python/pystate.c:1569:27 (python3.13+0x497df2) (BuildId: 8f8869b5f3143bd14dda26aa2bf37336b4902370)
    #3 PyGILState_Ensure /__w/jax/jax/cpython/Python/pystate.c:2766:16 (python3.13+0x49af78) (BuildId: 8f8869b5f3143bd14dda26aa2bf37336b4902370)
    #4 nanobind::gil_scoped_acquire::gil_scoped_acquire() /proc/self/cwd/external/nanobind/include/nanobind/nb_misc.h:15:43 (xla_extension.so+0xa4fe551) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #5 xla::CpuCallback::PrepareAndCall(void*, void**) /proc/self/cwd/external/xla/xla/python/callback.cc:67:26 (xla_extension.so+0xa4fe551)
    #6 xla::XlaPythonCpuCallback(void*, void**, XlaCustomCallStatus_*) /proc/self/cwd/external/xla/xla/python/callback.cc:177:22 (xla_extension.so+0xa500c9a) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #7 xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2::operator()(void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*) const /proc/self/cwd/external/xla/xla/backends/cpu/runtime/custom_call_thunk.cc:349:9 (xla_extension.so+0x18c3234) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #8 void std::__invoke_impl<void, xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2&, void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*>(std::__invoke_other, xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2&, void*&&, void const**&&, char const*&&, unsigned long&&, XlaCustomCallStatus_*&&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61:14 (xla_extension.so+0x18c3234)
    #9 std::enable_if<is_invocable_r_v<void, xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2&, void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*>, void>::type std::__invoke_r<void, xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2&, void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*>(xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2&, void*&&, void const**&&, char const*&&, unsigned long&&, XlaCustomCallStatus_*&&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:111:2 (xla_extension.so+0x18c3234)
    #10 std::_Function_handler<void (void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*), xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2>::_M_invoke(std::_Any_data const&, void*&&, void const**&&, char const*&&, unsigned long&&, XlaCustomCallStatus_*&&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:290:9 (xla_extension.so+0x18c3234)
    #11 std::function<void (void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*)>::operator()(void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*) const /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:591:9 (xla_extension.so+0x18c0409) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #12 xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&) /proc/self/cwd/external/xla/xla/backends/cpu/runtime/custom_call_thunk.cc:366:3 (xla_extension.so+0x18c0409)
    #13 xla::cpu::CustomCallThunk::Execute(xla::cpu::Thunk::ExecuteParams const&) /proc/self/cwd/external/xla/xla/backends/cpu/runtime/custom_call_thunk.cc:224:10 (xla_extension.so+0x18bd3eb) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #14 xla::cpu::ThunkExecutor::ExecuteSequential(xla::cpu::Thunk::ExecuteParams const&) /proc/self/cwd/external/xla/xla/backends/cpu/runtime/thunk_executor.cc:279:32 (xla_extension.so+0x75fe8b7) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #15 xla::cpu::ThunkExecutor::Execute(xla::cpu::Thunk::ExecuteParams const&) /proc/self/cwd/external/xla/xla/backends/cpu/runtime/thunk_executor.cc:213:12 (xla_extension.so+0x75fd97a) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #16 xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0::operator()() /proc/self/cwd/external/xla/xla/pjrt/cpu/cpu_client.cc:1798:44 (xla_extension.so+0xa01b0e) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #17 void std::__invoke_impl<void, xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>(std::__invoke_other, xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61:14 (xla_extension.so+0xa01b0e)
    #18 std::__invoke_result<xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>::type std::__invoke<xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>(xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:96:14 (xla_extension.so+0xa01b0e)
    #19 std::invoke_result<xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>::type std::invoke<xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>(xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/functional:120:14 (xla_extension.so+0xa01b0e)
    #20 void absl::lts_20230802::internal_any_invocable::InvokeR<void, xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&, void>(xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (xla_extension.so+0xa01b0e)
    #21 void absl::lts_20230802::internal_any_invocable::RemoteInvoker<false, void, xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:368:10 (xla_extension.so+0xa01b0e)
    #22 absl::lts_20230802::internal_any_invocable::Impl<void ()>::operator()() /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (xla_extension.so+0x9fb7bd) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #23 xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0::operator()() const /proc/self/cwd/external/xla/xla/pjrt/cpu/cpu_client.cc:181:5 (xla_extension.so+0x9fb7bd)
    #24 void std::__invoke_impl<void, xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0&>(std::__invoke_other, xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61:14 (xla_extension.so+0x9fb7bd)
    #25 std::enable_if<is_invocable_r_v<void, xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0&>, void>::type std::__invoke_r<void, xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0&>(xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:111:2 (xla_extension.so+0x9fb7bd)
    #26 std::_Function_handler<void (), xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0>::_M_invoke(std::_Any_data const&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:290:9 (xla_extension.so+0x9fb7bd)
    #27 std::function<void ()>::operator()() const /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:591:9 (xla_extension.so+0xe875342) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #28 tsl::thread::EigenEnvironment::ExecuteTask(tsl::thread::EigenEnvironment::Task const&) /proc/self/cwd/external/xla/xla/tsl/platform/threadpool.cc:112:5 (xla_extension.so+0xe875342)
    #29 Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) /proc/self/cwd/external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/ThreadPool/NonBlockingThreadPool.h:319:16 (xla_extension.so+0xe875342)
    #30 Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()::operator()() const /proc/self/cwd/external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/ThreadPool/NonBlockingThreadPool.h:58:68 (xla_extension.so+0xe873f71) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #31 void std::__invoke_impl<void, Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&>(std::__invoke_other, Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61:14 (xla_extension.so+0xe873f71)
    #32 std::enable_if<is_invocable_r_v<void, Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&>, void>::type std::__invoke_r<void, Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&>(Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:111:2 (xla_extension.so+0xe873f71)
    #33 std::_Function_handler<void (), Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()>::_M_invoke(std::_Any_data const&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:290:9 (xla_extension.so+0xe873f71)
    #34 std::function<void ()>::operator()() const /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:591:9 (xla_extension.so+0xe873f0f) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #35 tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()::operator()() const /proc/self/cwd/external/xla/xla/tsl/platform/threadpool.cc:95:7 (xla_extension.so+0xe873f0f)
    #36 void std::__invoke_impl<void, tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>(std::__invoke_other, tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61:14 (xla_extension.so+0xe873f0f)
    #37 std::__invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>::type std::__invoke<tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:96:14 (xla_extension.so+0xe873e11) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #38 std::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>::type std::invoke<tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/functional:120:14 (xla_extension.so+0xe873e11)
    #39 void absl::lts_20230802::internal_any_invocable::InvokeR<void, tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&, void>(tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (xla_extension.so+0xe873e11)
    #40 void absl::lts_20230802::internal_any_invocable::RemoteInvoker<false, void, tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:368:10 (xla_extension.so+0xe873e11)
    #41 absl::lts_20230802::internal_any_invocable::Impl<void ()>::operator()() /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (xla_extension.so+0xe8631b8) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #42 tsl::(anonymous namespace)::PThread::ThreadFn(void*) /proc/self/cwd/external/xla/xla/tsl/platform/default/env.cc:95:5 (xla_extension.so+0xe8631b8)

  Previous read of size 8 at 0x555555d43b60 by thread T124:
    #0 _Py_qsbr_reserve /__w/jax/jax/cpython/Python/qsbr.c:216:47 (python3.13+0x4a3ad7) (BuildId: 8f8869b5f3143bd14dda26aa2bf37336b4902370)
    #1 new_threadstate /__w/jax/jax/cpython/Python/pystate.c:1569:27 (python3.13+0x497df2) (BuildId: 8f8869b5f3143bd14dda26aa2bf37336b4902370)
    #2 PyGILState_Ensure /__w/jax/jax/cpython/Python/pystate.c:2766:16 (python3.13+0x49af78) (BuildId: 8f8869b5f3143bd14dda26aa2bf37336b4902370)
    #3 nanobind::gil_scoped_acquire::gil_scoped_acquire() /proc/self/cwd/external/nanobind/include/nanobind/nb_misc.h:15:43 (xla_extension.so+0xa4fe551) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #4 xla::CpuCallback::PrepareAndCall(void*, void**) /proc/self/cwd/external/xla/xla/python/callback.cc:67:26 (xla_extension.so+0xa4fe551)
    #5 xla::XlaPythonCpuCallback(void*, void**, XlaCustomCallStatus_*) /proc/self/cwd/external/xla/xla/python/callback.cc:177:22 (xla_extension.so+0xa500c9a) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #6 xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2::operator()(void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*) const /proc/self/cwd/external/xla/xla/backends/cpu/runtime/custom_call_thunk.cc:349:9 (xla_extension.so+0x18c3234) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #7 void std::__invoke_impl<void, xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2&, void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*>(std::__invoke_other, xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2&, void*&&, void const**&&, char const*&&, unsigned long&&, XlaCustomCallStatus_*&&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61:14 (xla_extension.so+0x18c3234)
    #8 std::enable_if<is_invocable_r_v<void, xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2&, void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*>, void>::type std::__invoke_r<void, xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2&, void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*>(xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2&, void*&&, void const**&&, char const*&&, unsigned long&&, XlaCustomCallStatus_*&&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:111:2 (xla_extension.so+0x18c3234)
    #9 std::_Function_handler<void (void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*), xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2>::_M_invoke(std::_Any_data const&, void*&&, void const**&&, char const*&&, unsigned long&&, XlaCustomCallStatus_*&&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:290:9 (xla_extension.so+0x18c3234)
    #10 std::function<void (void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*)>::operator()(void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*) const /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:591:9 (xla_extension.so+0x18c0409) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #11 xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&) /proc/self/cwd/external/xla/xla/backends/cpu/runtime/custom_call_thunk.cc:366:3 (xla_extension.so+0x18c0409)
    #12 xla::cpu::CustomCallThunk::Execute(xla::cpu::Thunk::ExecuteParams const&) /proc/self/cwd/external/xla/xla/backends/cpu/runtime/custom_call_thunk.cc:224:10 (xla_extension.so+0x18bd3eb) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #13 xla::cpu::ThunkExecutor::ExecuteSequential(xla::cpu::Thunk::ExecuteParams const&) /proc/self/cwd/external/xla/xla/backends/cpu/runtime/thunk_executor.cc:279:32 (xla_extension.so+0x75fe8b7) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #14 xla::cpu::ThunkExecutor::Execute(xla::cpu::Thunk::ExecuteParams const&) /proc/self/cwd/external/xla/xla/backends/cpu/runtime/thunk_executor.cc:213:12 (xla_extension.so+0x75fd97a) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #15 xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0::operator()() /proc/self/cwd/external/xla/xla/pjrt/cpu/cpu_client.cc:1798:44 (xla_extension.so+0xa01b0e) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #16 void std::__invoke_impl<void, xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>(std::__invoke_other, xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61:14 (xla_extension.so+0xa01b0e)
    #17 std::__invoke_result<xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>::type std::__invoke<xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>(xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:96:14 (xla_extension.so+0xa01b0e)
    #18 std::invoke_result<xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>::type std::invoke<xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>(xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/functional:120:14 (xla_extension.so+0xa01b0e)
    #19 void absl::lts_20230802::internal_any_invocable::InvokeR<void, xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&, void>(xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (xla_extension.so+0xa01b0e)
    #20 void absl::lts_20230802::internal_any_invocable::RemoteInvoker<false, void, xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:368:10 (xla_extension.so+0xa01b0e)
    #21 absl::lts_20230802::internal_any_invocable::Impl<void ()>::operator()() /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (xla_extension.so+0x9fb7bd) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #22 xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0::operator()() const /proc/self/cwd/external/xla/xla/pjrt/cpu/cpu_client.cc:181:5 (xla_extension.so+0x9fb7bd)
    #23 void std::__invoke_impl<void, xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0&>(std::__invoke_other, xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61:14 (xla_extension.so+0x9fb7bd)
    #24 std::enable_if<is_invocable_r_v<void, xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0&>, void>::type std::__invoke_r<void, xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0&>(xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:111:2 (xla_extension.so+0x9fb7bd)
    #25 std::_Function_handler<void (), xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0>::_M_invoke(std::_Any_data const&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:290:9 (xla_extension.so+0x9fb7bd)
    #26 std::function<void ()>::operator()() const /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:591:9 (xla_extension.so+0xe875342) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #27 tsl::thread::EigenEnvironment::ExecuteTask(tsl::thread::EigenEnvironment::Task const&) /proc/self/cwd/external/xla/xla/tsl/platform/threadpool.cc:112:5 (xla_extension.so+0xe875342)
    #28 Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) /proc/self/cwd/external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/ThreadPool/NonBlockingThreadPool.h:319:16 (xla_extension.so+0xe875342)
    #29 Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()::operator()() const /proc/self/cwd/external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/ThreadPool/NonBlockingThreadPool.h:58:68 (xla_extension.so+0xe873f71) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #30 void std::__invoke_impl<void, Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&>(std::__invoke_other, Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61:14 (xla_extension.so+0xe873f71)
    #31 std::enable_if<is_invocable_r_v<void, Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&>, void>::type std::__invoke_r<void, Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&>(Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:111:2 (xla_extension.so+0xe873f71)
    #32 std::_Function_handler<void (), Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()>::_M_invoke(std::_Any_data const&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:290:9 (xla_extension.so+0xe873f71)
    #33 std::function<void ()>::operator()() const /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:591:9 (xla_extension.so+0xe873f0f) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #34 tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()::operator()() const /proc/self/cwd/external/xla/xla/tsl/platform/threadpool.cc:95:7 (xla_extension.so+0xe873f0f)
    #35 void std::__invoke_impl<void, tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>(std::__invoke_other, tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61:14 (xla_extension.so+0xe873f0f)
    #36 std::__invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>::type std::__invoke<tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:96:14 (xla_extension.so+0xe873e11) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #37 std::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>::type std::invoke<tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/functional:120:14 (xla_extension.so+0xe873e11)
    #38 void absl::lts_20230802::internal_any_invocable::InvokeR<void, tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&, void>(tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (xla_extension.so+0xe873e11)
    #39 void absl::lts_20230802::internal_any_invocable::RemoteInvoker<false, void, tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:368:10 (xla_extension.so+0xe873e11)
    #40 absl::lts_20230802::internal_any_invocable::Impl<void ()>::operator()() /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (xla_extension.so+0xe8631b8) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
    #41 tsl::(anonymous namespace)::PThread::ThreadFn(void*) /proc/self/cwd/external/xla/xla/tsl/platform/default/env.cc:95:5 (xla_extension.so+0xe8631b8)

My guess is that there's not enough locking if you call PyGILState_Ensure from C threads not previously known to the Python interpreter.

System info (python version, jaxlib version, accelerator, etc.)

Python 3.13t

Activity

  1. added
    bugSomething isn't working
    free threadingIssues found in free threading builds
    on Feb 6, 2025
  2. self-assigned this
    on Feb 6, 2025
  3. hawkinsp commented on Feb 6, 2025

    @hawkinsp
    CollaboratorAuthor
  4. hawkinsp commented on Feb 6, 2025

    @hawkinsp
    CollaboratorAuthor

    Fixed in upstream CPython.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

bugSomething isn't workingfree threadingIssues found in free threading builds

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions