Repository navigation
Closed
Labels
bugSomething isn't workingSomething isn't workingfree threadingIssues found in free threading buildsIssues found in free threading builds
Description
Description
I suspect this is a CPython bug, I'll try to make a reproduction and file it upstream.
WARNING: ThreadSanitizer: data race (pid=208275)
Write of size 8 at 0x555555d43b60 by thread T121:
#0 grow_thread_array /__w/jax/jax/cpython/Python/qsbr.c:101:19 (python3.13+0x4a3905) (BuildId: 8f8869b5f3143bd14dda26aa2bf37336b4902370)
#1 _Py_qsbr_reserve /__w/jax/jax/cpython/Python/qsbr.c:203:13 (python3.13+0x4a3905)
#2 new_threadstate /__w/jax/jax/cpython/Python/pystate.c:1569:27 (python3.13+0x497df2) (BuildId: 8f8869b5f3143bd14dda26aa2bf37336b4902370)
#3 PyGILState_Ensure /__w/jax/jax/cpython/Python/pystate.c:2766:16 (python3.13+0x49af78) (BuildId: 8f8869b5f3143bd14dda26aa2bf37336b4902370)
#4 nanobind::gil_scoped_acquire::gil_scoped_acquire() /proc/self/cwd/external/nanobind/include/nanobind/nb_misc.h:15:43 (xla_extension.so+0xa4fe551) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#5 xla::CpuCallback::PrepareAndCall(void*, void**) /proc/self/cwd/external/xla/xla/python/callback.cc:67:26 (xla_extension.so+0xa4fe551)
#6 xla::XlaPythonCpuCallback(void*, void**, XlaCustomCallStatus_*) /proc/self/cwd/external/xla/xla/python/callback.cc:177:22 (xla_extension.so+0xa500c9a) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#7 xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2::operator()(void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*) const /proc/self/cwd/external/xla/xla/backends/cpu/runtime/custom_call_thunk.cc:349:9 (xla_extension.so+0x18c3234) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#8 void std::__invoke_impl<void, xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2&, void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*>(std::__invoke_other, xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2&, void*&&, void const**&&, char const*&&, unsigned long&&, XlaCustomCallStatus_*&&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61:14 (xla_extension.so+0x18c3234)
#9 std::enable_if<is_invocable_r_v<void, xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2&, void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*>, void>::type std::__invoke_r<void, xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2&, void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*>(xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2&, void*&&, void const**&&, char const*&&, unsigned long&&, XlaCustomCallStatus_*&&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:111:2 (xla_extension.so+0x18c3234)
#10 std::_Function_handler<void (void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*), xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2>::_M_invoke(std::_Any_data const&, void*&&, void const**&&, char const*&&, unsigned long&&, XlaCustomCallStatus_*&&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:290:9 (xla_extension.so+0x18c3234)
#11 std::function<void (void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*)>::operator()(void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*) const /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:591:9 (xla_extension.so+0x18c0409) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#12 xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&) /proc/self/cwd/external/xla/xla/backends/cpu/runtime/custom_call_thunk.cc:366:3 (xla_extension.so+0x18c0409)
#13 xla::cpu::CustomCallThunk::Execute(xla::cpu::Thunk::ExecuteParams const&) /proc/self/cwd/external/xla/xla/backends/cpu/runtime/custom_call_thunk.cc:224:10 (xla_extension.so+0x18bd3eb) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#14 xla::cpu::ThunkExecutor::ExecuteSequential(xla::cpu::Thunk::ExecuteParams const&) /proc/self/cwd/external/xla/xla/backends/cpu/runtime/thunk_executor.cc:279:32 (xla_extension.so+0x75fe8b7) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#15 xla::cpu::ThunkExecutor::Execute(xla::cpu::Thunk::ExecuteParams const&) /proc/self/cwd/external/xla/xla/backends/cpu/runtime/thunk_executor.cc:213:12 (xla_extension.so+0x75fd97a) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#16 xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0::operator()() /proc/self/cwd/external/xla/xla/pjrt/cpu/cpu_client.cc:1798:44 (xla_extension.so+0xa01b0e) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#17 void std::__invoke_impl<void, xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>(std::__invoke_other, xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61:14 (xla_extension.so+0xa01b0e)
#18 std::__invoke_result<xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>::type std::__invoke<xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>(xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:96:14 (xla_extension.so+0xa01b0e)
#19 std::invoke_result<xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>::type std::invoke<xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>(xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/functional:120:14 (xla_extension.so+0xa01b0e)
#20 void absl::lts_20230802::internal_any_invocable::InvokeR<void, xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&, void>(xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (xla_extension.so+0xa01b0e)
#21 void absl::lts_20230802::internal_any_invocable::RemoteInvoker<false, void, xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:368:10 (xla_extension.so+0xa01b0e)
#22 absl::lts_20230802::internal_any_invocable::Impl<void ()>::operator()() /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (xla_extension.so+0x9fb7bd) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#23 xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0::operator()() const /proc/self/cwd/external/xla/xla/pjrt/cpu/cpu_client.cc:181:5 (xla_extension.so+0x9fb7bd)
#24 void std::__invoke_impl<void, xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0&>(std::__invoke_other, xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61:14 (xla_extension.so+0x9fb7bd)
#25 std::enable_if<is_invocable_r_v<void, xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0&>, void>::type std::__invoke_r<void, xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0&>(xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:111:2 (xla_extension.so+0x9fb7bd)
#26 std::_Function_handler<void (), xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0>::_M_invoke(std::_Any_data const&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:290:9 (xla_extension.so+0x9fb7bd)
#27 std::function<void ()>::operator()() const /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:591:9 (xla_extension.so+0xe875342) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#28 tsl::thread::EigenEnvironment::ExecuteTask(tsl::thread::EigenEnvironment::Task const&) /proc/self/cwd/external/xla/xla/tsl/platform/threadpool.cc:112:5 (xla_extension.so+0xe875342)
#29 Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) /proc/self/cwd/external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/ThreadPool/NonBlockingThreadPool.h:319:16 (xla_extension.so+0xe875342)
#30 Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()::operator()() const /proc/self/cwd/external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/ThreadPool/NonBlockingThreadPool.h:58:68 (xla_extension.so+0xe873f71) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#31 void std::__invoke_impl<void, Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&>(std::__invoke_other, Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61:14 (xla_extension.so+0xe873f71)
#32 std::enable_if<is_invocable_r_v<void, Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&>, void>::type std::__invoke_r<void, Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&>(Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:111:2 (xla_extension.so+0xe873f71)
#33 std::_Function_handler<void (), Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()>::_M_invoke(std::_Any_data const&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:290:9 (xla_extension.so+0xe873f71)
#34 std::function<void ()>::operator()() const /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:591:9 (xla_extension.so+0xe873f0f) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#35 tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()::operator()() const /proc/self/cwd/external/xla/xla/tsl/platform/threadpool.cc:95:7 (xla_extension.so+0xe873f0f)
#36 void std::__invoke_impl<void, tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>(std::__invoke_other, tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61:14 (xla_extension.so+0xe873f0f)
#37 std::__invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>::type std::__invoke<tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:96:14 (xla_extension.so+0xe873e11) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#38 std::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>::type std::invoke<tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/functional:120:14 (xla_extension.so+0xe873e11)
#39 void absl::lts_20230802::internal_any_invocable::InvokeR<void, tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&, void>(tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (xla_extension.so+0xe873e11)
#40 void absl::lts_20230802::internal_any_invocable::RemoteInvoker<false, void, tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:368:10 (xla_extension.so+0xe873e11)
#41 absl::lts_20230802::internal_any_invocable::Impl<void ()>::operator()() /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (xla_extension.so+0xe8631b8) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#42 tsl::(anonymous namespace)::PThread::ThreadFn(void*) /proc/self/cwd/external/xla/xla/tsl/platform/default/env.cc:95:5 (xla_extension.so+0xe8631b8)
Previous read of size 8 at 0x555555d43b60 by thread T124:
#0 _Py_qsbr_reserve /__w/jax/jax/cpython/Python/qsbr.c:216:47 (python3.13+0x4a3ad7) (BuildId: 8f8869b5f3143bd14dda26aa2bf37336b4902370)
#1 new_threadstate /__w/jax/jax/cpython/Python/pystate.c:1569:27 (python3.13+0x497df2) (BuildId: 8f8869b5f3143bd14dda26aa2bf37336b4902370)
#2 PyGILState_Ensure /__w/jax/jax/cpython/Python/pystate.c:2766:16 (python3.13+0x49af78) (BuildId: 8f8869b5f3143bd14dda26aa2bf37336b4902370)
#3 nanobind::gil_scoped_acquire::gil_scoped_acquire() /proc/self/cwd/external/nanobind/include/nanobind/nb_misc.h:15:43 (xla_extension.so+0xa4fe551) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#4 xla::CpuCallback::PrepareAndCall(void*, void**) /proc/self/cwd/external/xla/xla/python/callback.cc:67:26 (xla_extension.so+0xa4fe551)
#5 xla::XlaPythonCpuCallback(void*, void**, XlaCustomCallStatus_*) /proc/self/cwd/external/xla/xla/python/callback.cc:177:22 (xla_extension.so+0xa500c9a) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#6 xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2::operator()(void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*) const /proc/self/cwd/external/xla/xla/backends/cpu/runtime/custom_call_thunk.cc:349:9 (xla_extension.so+0x18c3234) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#7 void std::__invoke_impl<void, xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2&, void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*>(std::__invoke_other, xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2&, void*&&, void const**&&, char const*&&, unsigned long&&, XlaCustomCallStatus_*&&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61:14 (xla_extension.so+0x18c3234)
#8 std::enable_if<is_invocable_r_v<void, xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2&, void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*>, void>::type std::__invoke_r<void, xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2&, void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*>(xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2&, void*&&, void const**&&, char const*&&, unsigned long&&, XlaCustomCallStatus_*&&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:111:2 (xla_extension.so+0x18c3234)
#9 std::_Function_handler<void (void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*), xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&)::$_2>::_M_invoke(std::_Any_data const&, void*&&, void const**&&, char const*&&, unsigned long&&, XlaCustomCallStatus_*&&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:290:9 (xla_extension.so+0x18c3234)
#10 std::function<void (void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*)>::operator()(void*, void const**, char const*, unsigned long, XlaCustomCallStatus_*) const /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:591:9 (xla_extension.so+0x18c0409) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#11 xla::cpu::CustomCallThunk::CallUntypedAPI(xla::cpu::Thunk::ExecuteParams const&) /proc/self/cwd/external/xla/xla/backends/cpu/runtime/custom_call_thunk.cc:366:3 (xla_extension.so+0x18c0409)
#12 xla::cpu::CustomCallThunk::Execute(xla::cpu::Thunk::ExecuteParams const&) /proc/self/cwd/external/xla/xla/backends/cpu/runtime/custom_call_thunk.cc:224:10 (xla_extension.so+0x18bd3eb) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#13 xla::cpu::ThunkExecutor::ExecuteSequential(xla::cpu::Thunk::ExecuteParams const&) /proc/self/cwd/external/xla/xla/backends/cpu/runtime/thunk_executor.cc:279:32 (xla_extension.so+0x75fe8b7) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#14 xla::cpu::ThunkExecutor::Execute(xla::cpu::Thunk::ExecuteParams const&) /proc/self/cwd/external/xla/xla/backends/cpu/runtime/thunk_executor.cc:213:12 (xla_extension.so+0x75fd97a) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#15 xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0::operator()() /proc/self/cwd/external/xla/xla/pjrt/cpu/cpu_client.cc:1798:44 (xla_extension.so+0xa01b0e) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#16 void std::__invoke_impl<void, xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>(std::__invoke_other, xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61:14 (xla_extension.so+0xa01b0e)
#17 std::__invoke_result<xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>::type std::__invoke<xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>(xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:96:14 (xla_extension.so+0xa01b0e)
#18 std::invoke_result<xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>::type std::invoke<xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>(xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/functional:120:14 (xla_extension.so+0xa01b0e)
#19 void absl::lts_20230802::internal_any_invocable::InvokeR<void, xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&, void>(xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (xla_extension.so+0xa01b0e)
#20 void absl::lts_20230802::internal_any_invocable::RemoteInvoker<false, void, xla::TfrtCpuExecutable::ExecuteHelper(absl::lts_20230802::Span<xla::PjRtBuffer* const>, int, int, xla::RunId const&, xla::ExecuteOptions const&, tsl::AsyncValueRef<xla::CpuEvent>, bool, xla::TfrtCpuDevice*)::$_0&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:368:10 (xla_extension.so+0xa01b0e)
#21 absl::lts_20230802::internal_any_invocable::Impl<void ()>::operator()() /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (xla_extension.so+0x9fb7bd) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#22 xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0::operator()() const /proc/self/cwd/external/xla/xla/pjrt/cpu/cpu_client.cc:181:5 (xla_extension.so+0x9fb7bd)
#23 void std::__invoke_impl<void, xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0&>(std::__invoke_other, xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61:14 (xla_extension.so+0x9fb7bd)
#24 std::enable_if<is_invocable_r_v<void, xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0&>, void>::type std::__invoke_r<void, xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0&>(xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:111:2 (xla_extension.so+0x9fb7bd)
#25 std::_Function_handler<void (), xla::(anonymous namespace)::EnqueueWork(tsl::thread::ThreadPool*, absl::lts_20230802::AnyInvocable<void ()>)::$_0>::_M_invoke(std::_Any_data const&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:290:9 (xla_extension.so+0x9fb7bd)
#26 std::function<void ()>::operator()() const /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:591:9 (xla_extension.so+0xe875342) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#27 tsl::thread::EigenEnvironment::ExecuteTask(tsl::thread::EigenEnvironment::Task const&) /proc/self/cwd/external/xla/xla/tsl/platform/threadpool.cc:112:5 (xla_extension.so+0xe875342)
#28 Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::WorkerLoop(int) /proc/self/cwd/external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/ThreadPool/NonBlockingThreadPool.h:319:16 (xla_extension.so+0xe875342)
#29 Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()::operator()() const /proc/self/cwd/external/eigen_archive/unsupported/Eigen/CXX11/../../../Eigen/src/ThreadPool/NonBlockingThreadPool.h:58:68 (xla_extension.so+0xe873f71) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#30 void std::__invoke_impl<void, Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&>(std::__invoke_other, Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61:14 (xla_extension.so+0xe873f71)
#31 std::enable_if<is_invocable_r_v<void, Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&>, void>::type std::__invoke_r<void, Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&>(Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:111:2 (xla_extension.so+0xe873f71)
#32 std::_Function_handler<void (), Eigen::ThreadPoolTempl<tsl::thread::EigenEnvironment>::ThreadPoolTempl(int, bool, tsl::thread::EigenEnvironment)::'lambda'()>::_M_invoke(std::_Any_data const&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:290:9 (xla_extension.so+0xe873f71)
#33 std::function<void ()>::operator()() const /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/std_function.h:591:9 (xla_extension.so+0xe873f0f) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#34 tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()::operator()() const /proc/self/cwd/external/xla/xla/tsl/platform/threadpool.cc:95:7 (xla_extension.so+0xe873f0f)
#35 void std::__invoke_impl<void, tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>(std::__invoke_other, tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:61:14 (xla_extension.so+0xe873f0f)
#36 std::__invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>::type std::__invoke<tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/bits/invoke.h:96:14 (xla_extension.so+0xe873e11) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#37 std::invoke_result<tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>::type std::invoke<tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>(tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&) /usr/lib/gcc/x86_64-linux-gnu/14/../../../../include/c++/14/functional:120:14 (xla_extension.so+0xe873e11)
#38 void absl::lts_20230802::internal_any_invocable::InvokeR<void, tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&, void>(tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:132:3 (xla_extension.so+0xe873e11)
#39 void absl::lts_20230802::internal_any_invocable::RemoteInvoker<false, void, tsl::thread::EigenEnvironment::CreateThread(std::function<void ()>)::'lambda'()&>(absl::lts_20230802::internal_any_invocable::TypeErasedState*) /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:368:10 (xla_extension.so+0xe873e11)
#40 absl::lts_20230802::internal_any_invocable::Impl<void ()>::operator()() /proc/self/cwd/external/com_google_absl/absl/functional/internal/any_invocable.h:868:1 (xla_extension.so+0xe8631b8) (BuildId: 32eac14928efa68545d22a6013f16aa63a686fef)
#41 tsl::(anonymous namespace)::PThread::ThreadFn(void*) /proc/self/cwd/external/xla/xla/tsl/platform/default/env.cc:95:5 (xla_extension.so+0xe8631b8)
My guess is that there's not enough locking if you call PyGILState_Ensure from C threads not previously known to the Python interpreter.
System info (python version, jaxlib version, accelerator, etc.)
Python 3.13t
Metadata
Metadata
Assignees
Labels
bugSomething isn't workingSomething isn't workingfree threadingIssues found in free threading buildsIssues found in free threading builds
Activity
hawkinsp commentedon Feb 6, 2025
Filed python/cpython#129732
@vfdev-5 FYI
hawkinsp commentedon Feb 6, 2025
Fixed in upstream CPython.