Skip to content

Improve OpenTelemetry spans creation after Processors were implemented#37186

Closed
qoega wants to merge 1 commit intoClickHouse:masterfrom
qoega:improve-opentelemetry
Closed

Improve OpenTelemetry spans creation after Processors were implemented#37186
qoega wants to merge 1 commit intoClickHouse:masterfrom
qoega:improve-opentelemetry

Conversation

@qoega
Copy link
Copy Markdown
Member

@qoega qoega commented May 13, 2022

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

OpenTelemetry spans now show active time in Processors.

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

@robot-ch-test-poll2 robot-ch-test-poll2 added the pr-improvement Pull request with some product improvements label May 13, 2022
@qoega
Copy link
Copy Markdown
Member Author

qoega commented May 13, 2022

Example
Screenshot 2022-05-13 at 13 57 02

vdimir
vdimir reviewed May 13, 2022
View reviewed changes
src/Processors/Executors/ExecutionThreadContext.cpp
{
UInt64 elapsed_microseconds = execution_time_watch->elapsedMicroseconds();
node->processor->elapsed_us += elapsed_microseconds;
span.addAttribute("execution_time_ns", elapsed_microseconds);
Copy link
Copy Markdown
Member

@vdimir vdimir May 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why execution_time_ns, but elapsed_microseconds ?
Also if bool(profile_processors) == true and NDEBUG attributes would be set twice, it it ok?

Copy link
Copy Markdown
Member Author

@qoega qoega May 13, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will fix.

@qoega qoega force-pushed the improve-opentelemetry branch from f20b5d5 to 1f5ffd7 Compare May 13, 2022 17:30
@vdimir vdimir self-assigned this May 16, 2022
@qoega
Copy link
Copy Markdown
Member Author

qoega commented May 16, 2022

I guess I will rework spans order. It is almost unusable with longer queries. I think

  1. Processors should be under query(or Executor)
  2. Current executor thread span is harmful.
  3. It is nice to see when work in single processor was done or filter it out trivially when work() is child of processor span.

Failure in FunctionalStatefulTestTsan is related:
It should see Sanitizer report in logs, but only timeout showed that CH crashed

==================
WARNING: ThreadSanitizer: data race (pid=662)
  Write of size 8 at 0x7b4c01df48c8 by thread T351 (mutexes: write M31311763):
    #0 std::__1::enable_if<(is_move_constructible<DB::Context const*>::value) && (is_move_assignable<DB::Context const*>::value), void>::type std::__1::swap<DB::Context const*>(DB::Context const*&, DB::Context const*&) build_docker/../contrib/libcxx/include/__utility/swap.h:37:7 (clickhouse+0x19d33f23)
    #1 std::__1::weak_ptr<DB::Context const>::swap(std::__1::weak_ptr<DB::Context const>&) build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:1407:5 (clickhouse+0x19d33f23)
    #2 std::__1::weak_ptr<DB::Context const>::operator=(std::__1::weak_ptr<DB::Context const> const&) build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:1347:19 (clickhouse+0x19d33f23)
    #3 DB::ThreadStatus::attachQueryContext(std::__1::shared_ptr<DB::Context const>) build_docker/../src/Interpreters/ThreadStatusExt.cpp:82:37 (clickhouse+0x19d33f23)
    #4 DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0::operator()() const build_docker/../src/Disks/IO/ThreadPoolRemoteFSReader.cpp:63:27 (clickhouse+0x19163039)
    #5 decltype(static_cast<DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&>(fp)()) std::__1::__invoke<DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&) build_docker/../contrib/libcxx/include/type_traits:3640:23 (clickhouse+0x19163039)
    #6 std::__1::__packaged_task_func<DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0, std::__1::allocator<DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>, DB::IAsynchronousReader::Result ()>::operator()() build_docker/../contrib/libcxx/include/future:1688:12 (clickhouse+0x19163039)
    #7 std::__1::__packaged_task_function<DB::IAsynchronousReader::Result ()>::operator()() const build_docker/../contrib/libcxx/include/future:1870:12 (clickhouse+0x1896c8ae)
    #8 std::__1::packaged_task<DB::IAsynchronousReader::Result ()>::operator()() build_docker/../contrib/libcxx/include/future:1961:24 (clickhouse+0x1896c8ae)
    #9 DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_1::operator()() const build_docker/../src/Disks/IO/ThreadPoolRemoteFSReader.cpp:97:34 (clickhouse+0x1916324c)
    #10 decltype(static_cast<DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_1&>(fp)()) std::__1::__invoke<DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_1&>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_1&) build_docker/../contrib/libcxx/include/type_traits:3640:23 (clickhouse+0x1916324c)
    #11 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_1&>(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_1&) build_docker/../contrib/libcxx/include/__functional/invoke.h:61:9 (clickhouse+0x1916324c)
    #12 std::__1::__function::__default_alloc_func<DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_1, void ()>::operator()() build_docker/../contrib/libcxx/include/__functional/function.h:230:12 (clickhouse+0x1916324c)
    #13 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_1, void ()> >(std::__1::__function::__policy_storage const*) build_docker/../contrib/libcxx/include/__functional/function.h:711:16 (clickhouse+0x1916324c)
    #14 std::__1::__function::__policy_func<void ()>::operator()() const build_docker/../contrib/libcxx/include/__functional/function.h:843:16 (clickhouse+0xaff79b7)
    #15 std::__1::function<void ()>::operator()() const build_docker/../contrib/libcxx/include/__functional/function.h:1184:12 (clickhouse+0xaff79b7)
    #16 ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) build_docker/../src/Common/ThreadPool.cpp:281:17 (clickhouse+0xaff79b7)
    #17 void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()::operator()() const build_docker/../src/Common/ThreadPool.cpp:143:73 (clickhouse+0xaffa218)
    #18 decltype(static_cast<void>(fp)()) std::__1::__invoke_constexpr<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()&>(void&&) build_docker/../contrib/libcxx/include/type_traits:3648:23 (clickhouse+0xaffa218)
    #19 decltype(auto) std::__1::__apply_tuple_impl<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()&, std::__1::tuple<>&>(void&&, std::__1::tuple<>&, std::__1::__tuple_indices<>) build_docker/../contrib/libcxx/include/tuple:1595:1 (clickhouse+0xaffa218)
    #20 decltype(auto) std::__1::apply<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()&, std::__1::tuple<>&>(void&&, std::__1::tuple<>&) build_docker/../contrib/libcxx/include/tuple:1604:1 (clickhouse+0xaffa218)
    #21 ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&)::'lambda'()::operator()() build_docker/../src/Common/ThreadPool.h:188:13 (clickhouse+0xaffa218)
    #22 decltype(static_cast<void>(fp)()) std::__1::__invoke<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&)::'lambda'()&>(void&&) build_docker/../contrib/libcxx/include/type_traits:3640:23 (clickhouse+0xaffa0c1)
    #23 void std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&)::'lambda'()&>(ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&)::'lambda'()&) build_docker/../contrib/libcxx/include/__functional/invoke.h:61:9 (clickhouse+0xaffa0c1)
    #24 std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&)::'lambda'(), void ()>::operator()() build_docker/../contrib/libcxx/include/__functional/function.h:230:12 (clickhouse+0xaffa0c1)
    #25 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&)::'lambda'(), void ()> >(std::__1::__function::__policy_storage const*) build_docker/../contrib/libcxx/include/__functional/function.h:711:16 (clickhouse+0xaffa0c1)
    #26 std::__1::__function::__policy_func<void ()>::operator()() const build_docker/../contrib/libcxx/include/__functional/function.h:843:16 (clickhouse+0xaff4fed)
    #27 std::__1::function<void ()>::operator()() const build_docker/../contrib/libcxx/include/__functional/function.h:1184:12 (clickhouse+0xaff4fed)
    #28 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) build_docker/../src/Common/ThreadPool.cpp:281:17 (clickhouse+0xaff4fed)
    #29 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()::operator()() const build_docker/../src/Common/ThreadPool.cpp:143:73 (clickhouse+0xaff8830)
    #30 decltype(static_cast<void>(fp)()) std::__1::__invoke<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&) build_docker/../contrib/libcxx/include/type_traits:3640:23 (clickhouse+0xaff8830)
    #31 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>&, std::__1::__tuple_indices<>) build_docker/../contrib/libcxx/include/thread:282:5 (clickhouse+0xaff8830)
    #32 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()> >(void*) build_docker/../contrib/libcxx/include/thread:293:5 (clickhouse+0xaff8830)

  Previous read of size 8 at 0x7b4c01df48c8 by thread T548:
    #0 std::__1::weak_ptr<DB::Context const>::lock() const build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:1434:22 (clickhouse+0x19b53e9f)
    #1 DB::OpenTelemetrySpanHolder::~OpenTelemetrySpanHolder() build_docker/../src/Interpreters/OpenTelemetrySpanLog.cpp:125:52 (clickhouse+0x19b53e9f)
    #2 DB::executeJob(DB::IProcessor*) build_docker/../src/Processors/Executors/ExecutionThreadContext.cpp:49:5 (clickhouse+0x1accaa10)
    #3 DB::ExecutionThreadContext::executeTask() build_docker/../src/Processors/Executors/ExecutionThreadContext.cpp:72:9 (clickhouse+0x1accaa10)
    #4 DB::PipelineExecutor::executeStepImpl(unsigned long, std::__1::atomic<bool>*) build_docker/../src/Processors/Executors/PipelineExecutor.cpp:216:26 (clickhouse+0x1acbcac0)
    #5 DB::PipelineExecutor::executeSingleThread(unsigned long) build_docker/../src/Processors/Executors/PipelineExecutor.cpp:181:5 (clickhouse+0x1acbdcb5)
    #6 DB::PipelineExecutor::executeImpl(unsigned long)::$_1::operator()() const build_docker/../src/Processors/Executors/PipelineExecutor.cpp:307:21 (clickhouse+0x1acbdcb5)
    #7 decltype(static_cast<DB::PipelineExecutor::executeImpl(unsigned long)::$_1&>(fp)()) std::__1::__invoke_constexpr<DB::PipelineExecutor::executeImpl(unsigned long)::$_1&>(DB::PipelineExecutor::executeImpl(unsigned long)::$_1&) build_docker/../contrib/libcxx/include/type_traits:3648:23 (clickhouse+0x1acbdcb5)
    #8 decltype(auto) std::__1::__apply_tuple_impl<DB::PipelineExecutor::executeImpl(unsigned long)::$_1&, std::__1::tuple<>&>(DB::PipelineExecutor::executeImpl(unsigned long)::$_1&, std::__1::tuple<>&, std::__1::__tuple_indices<>) build_docker/../contrib/libcxx/include/tuple:1595:1 (clickhouse+0x1acbdcb5)
    #9 decltype(auto) std::__1::apply<DB::PipelineExecutor::executeImpl(unsigned long)::$_1&, std::__1::tuple<>&>(DB::PipelineExecutor::executeImpl(unsigned long)::$_1&, std::__1::tuple<>&) build_docker/../contrib/libcxx/include/tuple:1604:1 (clickhouse+0x1acbdcb5)
    #10 ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long)::$_1>(DB::PipelineExecutor::executeImpl(unsigned long)::$_1&&)::'lambda'()::operator()() build_docker/../src/Common/ThreadPool.h:188:13 (clickhouse+0x1acbdcb5)
    #11 decltype(static_cast<DB::PipelineExecutor::executeImpl(unsigned long)::$_1>(fp)()) std::__1::__invoke<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long)::$_1>(DB::PipelineExecutor::executeImpl(unsigned long)::$_1&&)::'lambda'()&>(DB::PipelineExecutor::executeImpl(unsigned long)::$_1&&) build_docker/../contrib/libcxx/include/type_traits:3640:23 (clickhouse+0x1acbdcb5)
    #12 void std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long)::$_1>(DB::PipelineExecutor::executeImpl(unsigned long)::$_1&&)::'lambda'()&>(ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long)::$_1>(DB::PipelineExecutor::executeImpl(unsigned long)::$_1&&)::'lambda'()&) build_docker/../contrib/libcxx/include/__functional/invoke.h:61:9 (clickhouse+0x1acbdcb5)
    #13 std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long)::$_1>(DB::PipelineExecutor::executeImpl(unsigned long)::$_1&&)::'lambda'(), void ()>::operator()() build_docker/../contrib/libcxx/include/__functional/function.h:230:12 (clickhouse+0x1acbdcb5)
    #14 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long)::$_1>(DB::PipelineExecutor::executeImpl(unsigned long)::$_1&&)::'lambda'(), void ()> >(std::__1::__function::__policy_storage const*) build_docker/../contrib/libcxx/include/__functional/function.h:711:16 (clickhouse+0x1acbdcb5)
    #15 std::__1::__function::__policy_func<void ()>::operator()() const build_docker/../contrib/libcxx/include/__functional/function.h:843:16 (clickhouse+0xaff4fed)
    #16 std::__1::function<void ()>::operator()() const build_docker/../contrib/libcxx/include/__functional/function.h:1184:12 (clickhouse+0xaff4fed)
    #17 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) build_docker/../src/Common/ThreadPool.cpp:281:17 (clickhouse+0xaff4fed)
    #18 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()::operator()() const build_docker/../src/Common/ThreadPool.cpp:143:73 (clickhouse+0xaff8830)
    #19 decltype(static_cast<void>(fp)()) std::__1::__invoke<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&) build_docker/../contrib/libcxx/include/type_traits:3640:23 (clickhouse+0xaff8830)
    #20 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>&, std::__1::__tuple_indices<>) build_docker/../contrib/libcxx/include/thread:282:5 (clickhouse+0xaff8830)
    #21 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()> >(void*) build_docker/../contrib/libcxx/include/thread:293:5 (clickhouse+0xaff8830)

  Location is heap block of size 432 at 0x7b4c01df4800 allocated by thread T411:
    #0 operator new(unsigned long) <null> (clickhouse+0xaf1c458)
    #1 void* std::__1::__libcpp_operator_new<unsigned long>(unsigned long) build_docker/../contrib/libcxx/include/new:245:10 (clickhouse+0x19d39260)
    #2 std::__1::__libcpp_allocate(unsigned long, unsigned long) build_docker/../contrib/libcxx/include/new:271:10 (clickhouse+0x19d39260)
    #3 std::__1::allocator<std::__1::__shared_ptr_emplace<DB::ThreadGroupStatus, std::__1::allocator<DB::ThreadGroupStatus> > >::allocate(unsigned long) build_docker/../contrib/libcxx/include/__memory/allocator.h:105:38 (clickhouse+0x19d39260)
    #4 std::__1::allocator_traits<std::__1::allocator<std::__1::__shared_ptr_emplace<DB::ThreadGroupStatus, std::__1::allocator<DB::ThreadGroupStatus> > > >::allocate(std::__1::allocator<std::__1::__shared_ptr_emplace<DB::ThreadGroupStatus, std::__1::allocator<DB::ThreadGroupStatus> > >&, unsigned long) build_docker/../contrib/libcxx/include/__memory/allocator_traits.h:262:20 (clickhouse+0x19d39260)
    #5 std::__1::__allocation_guard<std::__1::allocator<std::__1::__shared_ptr_emplace<DB::ThreadGroupStatus, std::__1::allocator<DB::ThreadGroupStatus> > > >::__allocation_guard<std::__1::allocator<DB::ThreadGroupStatus> >(std::__1::allocator<DB::ThreadGroupStatus>, unsigned long) build_docker/../contrib/libcxx/include/__memory/allocation_guard.h:53:18 (clickhouse+0x19d39260)
    #6 std::__1::shared_ptr<DB::ThreadGroupStatus> std::__1::allocate_shared<DB::ThreadGroupStatus, std::__1::allocator<DB::ThreadGroupStatus>, void>(std::__1::allocator<DB::ThreadGroupStatus> const&) build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:953:48 (clickhouse+0x19d39260)
    #7 std::__1::shared_ptr<DB::ThreadGroupStatus> std::__1::make_shared<DB::ThreadGroupStatus, void>() build_docker/../contrib/libcxx/include/__memory/shared_ptr.h:963:12 (clickhouse+0x19d3776d)
    #8 DB::ThreadStatus::initializeQuery() build_docker/../src/Interpreters/ThreadStatusExt.cpp:156:16 (clickhouse+0x19d3776d)
    #9 DB::CurrentThread::initializeQuery() build_docker/../src/Interpreters/ThreadStatusExt.cpp:542:21 (clickhouse+0x19d3776d)
    #10 DB::CurrentThread::QueryScope::QueryScope(std::__1::shared_ptr<DB::Context>) build_docker/../src/Interpreters/ThreadStatusExt.cpp:593:5 (clickhouse+0x19d37d7a)
    #11 DB::CurrentThread::QueryScope* std::__1::construct_at<DB::CurrentThread::QueryScope, std::__1::shared_ptr<DB::Context>&, DB::CurrentThread::QueryScope*>(DB::CurrentThread::QueryScope*, std::__1::shared_ptr<DB::Context>&) build_docker/../contrib/libcxx/include/__memory/construct_at.h:38:50 (clickhouse+0x1ac500bd)
    #12 void std::__1::__optional_storage_base<DB::CurrentThread::QueryScope, false>::__construct<std::__1::shared_ptr<DB::Context>&>(std::__1::shared_ptr<DB::Context>&) build_docker/../contrib/libcxx/include/optional:351:9 (clickhouse+0x1ac500bd)
    #13 DB::CurrentThread::QueryScope& std::__1::optional<DB::CurrentThread::QueryScope>::emplace<std::__1::shared_ptr<DB::Context>&, void>(std::__1::shared_ptr<DB::Context>&) build_docker/../contrib/libcxx/include/optional:875:15 (clickhouse+0x1ac500bd)
    #14 DB::TCPHandler::runImpl() build_docker/../src/Server/TCPHandler.cpp:225:25 (clickhouse+0x1ac500bd)
    #15 DB::TCPHandler::run() build_docker/../src/Server/TCPHandler.cpp:1781:9 (clickhouse+0x1ac61387)
    #16 Poco::Net::TCPServerConnection::start() build_docker/../contrib/poco/Net/src/TCPServerConnection.cpp:43:3 (clickhouse+0x1f9170c2)
    #17 Poco::Net::TCPServerDispatcher::run() build_docker/../contrib/poco/Net/src/TCPServerDispatcher.cpp:115:20 (clickhouse+0x1f917932)
    #18 Poco::PooledThread::run() build_docker/../contrib/poco/Foundation/src/ThreadPool.cpp:199:14 (clickhouse+0x1fb7c1d5)
    #19 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/../contrib/poco/Foundation/src/Thread.cpp:55:11 (clickhouse+0x1fb7a36f)
    #20 Poco::ThreadImpl::runnableEntry(void*) build_docker/../contrib/poco/Foundation/src/Thread_POSIX.cpp:345:27 (clickhouse+0x1fb789e7)

  Mutex M31311763 (0x7b4c01df4818) created at:
    #0 pthread_mutex_lock <null> (clickhouse+0xaec49f8)
    #1 std::__1::__libcpp_mutex_lock(pthread_mutex_t*) build_docker/../contrib/libcxx/include/__threading_support:303:10 (clickhouse+0x216f8499)
    #2 std::__1::mutex::lock() build_docker/../contrib/libcxx/src/mutex.cpp:33:14 (clickhouse+0x216f8499)
    #3 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&) build_docker/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse+0x19d35108)
    #4 DB::ThreadStatus::setupState(std::__1::shared_ptr<DB::ThreadGroupStatus> const&) build_docker/../src/Interpreters/ThreadStatusExt.cpp:117:25 (clickhouse+0x19d35108)
    #5 DB::ThreadStatus::initializeQuery() build_docker/../src/Interpreters/ThreadStatusExt.cpp:156:5 (clickhouse+0x19d37784)
    #6 DB::CurrentThread::initializeQuery() build_docker/../src/Interpreters/ThreadStatusExt.cpp:542:21 (clickhouse+0x19d37784)
    #7 DB::CurrentThread::QueryScope::QueryScope(std::__1::shared_ptr<DB::Context>) build_docker/../src/Interpreters/ThreadStatusExt.cpp:593:5 (clickhouse+0x19d37d7a)
    #8 DB::CurrentThread::QueryScope* std::__1::construct_at<DB::CurrentThread::QueryScope, std::__1::shared_ptr<DB::Context>&, DB::CurrentThread::QueryScope*>(DB::CurrentThread::QueryScope*, std::__1::shared_ptr<DB::Context>&) build_docker/../contrib/libcxx/include/__memory/construct_at.h:38:50 (clickhouse+0x1ac500bd)
    #9 void std::__1::__optional_storage_base<DB::CurrentThread::QueryScope, false>::__construct<std::__1::shared_ptr<DB::Context>&>(std::__1::shared_ptr<DB::Context>&) build_docker/../contrib/libcxx/include/optional:351:9 (clickhouse+0x1ac500bd)
    #10 DB::CurrentThread::QueryScope& std::__1::optional<DB::CurrentThread::QueryScope>::emplace<std::__1::shared_ptr<DB::Context>&, void>(std::__1::shared_ptr<DB::Context>&) build_docker/../contrib/libcxx/include/optional:875:15 (clickhouse+0x1ac500bd)
    #11 DB::TCPHandler::runImpl() build_docker/../src/Server/TCPHandler.cpp:225:25 (clickhouse+0x1ac500bd)
    #12 DB::TCPHandler::run() build_docker/../src/Server/TCPHandler.cpp:1781:9 (clickhouse+0x1ac61387)
    #13 Poco::Net::TCPServerConnection::start() build_docker/../contrib/poco/Net/src/TCPServerConnection.cpp:43:3 (clickhouse+0x1f9170c2)
    #14 Poco::Net::TCPServerDispatcher::run() build_docker/../contrib/poco/Net/src/TCPServerDispatcher.cpp:115:20 (clickhouse+0x1f917932)
    #15 Poco::PooledThread::run() build_docker/../contrib/poco/Foundation/src/ThreadPool.cpp:199:14 (clickhouse+0x1fb7c1d5)
    #16 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/../contrib/poco/Foundation/src/Thread.cpp:55:11 (clickhouse+0x1fb7a36f)
    #17 Poco::ThreadImpl::runnableEntry(void*) build_docker/../contrib/poco/Foundation/src/Thread_POSIX.cpp:345:27 (clickhouse+0x1fb789e7)

  Thread T351 'ThreadPool' (tid=1098, running) created by thread T286 at:
    #0 pthread_create <null> (clickhouse+0xaea771d)
    #1 std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) build_docker/../contrib/libcxx/include/__threading_support:375:10 (clickhouse+0xaff82e0)
    #2 std::__1::thread::thread<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'(), void>(void&&) build_docker/../contrib/libcxx/include/thread:309:16 (clickhouse+0xaff82e0)
    #3 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>) build_docker/../src/Common/ThreadPool.cpp:143:35 (clickhouse+0xaff3614)
    #4 ThreadPoolImpl<std::__1::thread>::scheduleOrThrow(std::__1::function<void ()>, int, unsigned long) build_docker/../src/Common/ThreadPool.cpp:175:5 (clickhouse+0xaff3f47)
    #5 ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long)::$_1>(DB::PipelineExecutor::executeImpl(unsigned long)::$_1&&) build_docker/../src/Common/ThreadPool.h:169:38 (clickhouse+0x1acbd927)
    #6 ThreadFromGlobalPool* std::__1::construct_at<ThreadFromGlobalPool, DB::PipelineExecutor::executeImpl(unsigned long)::$_1, ThreadFromGlobalPool*>(ThreadFromGlobalPool*, DB::PipelineExecutor::executeImpl(unsigned long)::$_1&&) build_docker/../contrib/libcxx/include/__memory/construct_at.h:38:50 (clickhouse+0x1acbd927)
    #7 void std::__1::allocator_traits<std::__1::allocator<ThreadFromGlobalPool> >::construct<ThreadFromGlobalPool, DB::PipelineExecutor::executeImpl(unsigned long)::$_1, void, void>(std::__1::allocator<ThreadFromGlobalPool>&, ThreadFromGlobalPool*, DB::PipelineExecutor::executeImpl(unsigned long)::$_1&&) build_docker/../contrib/libcxx/include/__memory/allocator_traits.h:298:9 (clickhouse+0x1acbd927)
    #8 void std::__1::vector<ThreadFromGlobalPool, std::__1::allocator<ThreadFromGlobalPool> >::__construct_one_at_end<DB::PipelineExecutor::executeImpl(unsigned long)::$_1>(DB::PipelineExecutor::executeImpl(unsigned long)::$_1&&) build_docker/../contrib/libcxx/include/vector:778:5 (clickhouse+0x1acbbb13)
    #9 ThreadFromGlobalPool& std::__1::vector<ThreadFromGlobalPool, std::__1::allocator<ThreadFromGlobalPool> >::emplace_back<DB::PipelineExecutor::executeImpl(unsigned long)::$_1>(DB::PipelineExecutor::executeImpl(unsigned long)::$_1&&) build_docker/../contrib/libcxx/include/vector:1577:9 (clickhouse+0x1acbbb13)
    #10 DB::PipelineExecutor::executeImpl(unsigned long) build_docker/../src/Processors/Executors/PipelineExecutor.cpp:296:21 (clickhouse+0x1acbbb13)
    #11 DB::PipelineExecutor::execute(unsigned long) build_docker/../src/Processors/Executors/PipelineExecutor.cpp:87:9 (clickhouse+0x1acbb878)
    #12 DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long) build_docker/../src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:78:24 (clickhouse+0x1accedc3)
    #13 DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const build_docker/../src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:105:13 (clickhouse+0x1accedc3)
    #14 decltype(static_cast<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&>(fp)()) std::__1::__invoke_constexpr<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) build_docker/../contrib/libcxx/include/type_traits:3648:23 (clickhouse+0x1accedc3)
    #15 decltype(auto) std::__1::__apply_tuple_impl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::__1::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::__1::tuple<>&, std::__1::__tuple_indices<>) build_docker/../contrib/libcxx/include/tuple:1595:1 (clickhouse+0x1accedc3)
    #16 decltype(auto) std::__1::apply<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::__1::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::__1::tuple<>&) build_docker/../contrib/libcxx/include/tuple:1604:1 (clickhouse+0x1accedc3)
    #17 ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() build_docker/../src/Common/ThreadPool.h:188:13 (clickhouse+0x1accedc3)
    #18 decltype(static_cast<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(fp)()) std::__1::__invoke<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) build_docker/../contrib/libcxx/include/type_traits:3640:23 (clickhouse+0x1accedc3)
    #19 void std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) build_docker/../contrib/libcxx/include/__functional/invoke.h:61:9 (clickhouse+0x1accedc3)
    #20 std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()() build_docker/../contrib/libcxx/include/__functional/function.h:230:12 (clickhouse+0x1accedc3)
    #21 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()> >(std::__1::__function::__policy_storage const*) build_docker/../contrib/libcxx/include/__functional/function.h:711:16 (clickhouse+0x1accedc3)
    #22 std::__1::__function::__policy_func<void ()>::operator()() const build_docker/../contrib/libcxx/include/__functional/function.h:843:16 (clickhouse+0xaff4fed)
    #23 std::__1::function<void ()>::operator()() const build_docker/../contrib/libcxx/include/__functional/function.h:1184:12 (clickhouse+0xaff4fed)
    #24 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) build_docker/../src/Common/ThreadPool.cpp:281:17 (clickhouse+0xaff4fed)
    #25 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()::operator()() const build_docker/../src/Common/ThreadPool.cpp:143:73 (clickhouse+0xaff8830)
    #26 decltype(static_cast<void>(fp)()) std::__1::__invoke<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&) build_docker/../contrib/libcxx/include/type_traits:3640:23 (clickhouse+0xaff8830)
    #27 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>&, std::__1::__tuple_indices<>) build_docker/../contrib/libcxx/include/thread:282:5 (clickhouse+0xaff8830)
    #28 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()> >(void*) build_docker/../contrib/libcxx/include/thread:293:5 (clickhouse+0xaff8830)

  Thread T548 'QueryPipelineEx' (tid=1276, running) created by thread T481 at:
    #0 pthread_create <null> (clickhouse+0xaea771d)
    #1 std::__1::__libcpp_thread_create(unsigned long*, void* (*)(void*), void*) build_docker/../contrib/libcxx/include/__threading_support:375:10 (clickhouse+0xaff82e0)
    #2 std::__1::thread::thread<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'(), void>(void&&) build_docker/../contrib/libcxx/include/thread:309:16 (clickhouse+0xaff82e0)
    #3 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>) build_docker/../src/Common/ThreadPool.cpp:143:35 (clickhouse+0xaff3614)
    #4 ThreadPoolImpl<std::__1::thread>::scheduleOrThrow(std::__1::function<void ()>, int, unsigned long) build_docker/../src/Common/ThreadPool.cpp:175:5 (clickhouse+0xaff3f47)
    #5 ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PipelineExecutor::executeImpl(unsigned long)::$_1>(DB::PipelineExecutor::executeImpl(unsigned long)::$_1&&) build_docker/../src/Common/ThreadPool.h:169:38 (clickhouse+0x1acbd927)
    #6 ThreadFromGlobalPool* std::__1::construct_at<ThreadFromGlobalPool, DB::PipelineExecutor::executeImpl(unsigned long)::$_1, ThreadFromGlobalPool*>(ThreadFromGlobalPool*, DB::PipelineExecutor::executeImpl(unsigned long)::$_1&&) build_docker/../contrib/libcxx/include/__memory/construct_at.h:38:50 (clickhouse+0x1acbd927)
    #7 void std::__1::allocator_traits<std::__1::allocator<ThreadFromGlobalPool> >::construct<ThreadFromGlobalPool, DB::PipelineExecutor::executeImpl(unsigned long)::$_1, void, void>(std::__1::allocator<ThreadFromGlobalPool>&, ThreadFromGlobalPool*, DB::PipelineExecutor::executeImpl(unsigned long)::$_1&&) build_docker/../contrib/libcxx/include/__memory/allocator_traits.h:298:9 (clickhouse+0x1acbd927)
    #8 void std::__1::vector<ThreadFromGlobalPool, std::__1::allocator<ThreadFromGlobalPool> >::__construct_one_at_end<DB::PipelineExecutor::executeImpl(unsigned long)::$_1>(DB::PipelineExecutor::executeImpl(unsigned long)::$_1&&) build_docker/../contrib/libcxx/include/vector:778:5 (clickhouse+0x1acbbb13)
    #9 ThreadFromGlobalPool& std::__1::vector<ThreadFromGlobalPool, std::__1::allocator<ThreadFromGlobalPool> >::emplace_back<DB::PipelineExecutor::executeImpl(unsigned long)::$_1>(DB::PipelineExecutor::executeImpl(unsigned long)::$_1&&) build_docker/../contrib/libcxx/include/vector:1577:9 (clickhouse+0x1acbbb13)
    #10 DB::PipelineExecutor::executeImpl(unsigned long) build_docker/../src/Processors/Executors/PipelineExecutor.cpp:296:21 (clickhouse+0x1acbbb13)
    #11 DB::PipelineExecutor::execute(unsigned long) build_docker/../src/Processors/Executors/PipelineExecutor.cpp:87:9 (clickhouse+0x1acbb878)
    #12 DB::threadFunction(DB::PullingAsyncPipelineExecutor::Data&, std::__1::shared_ptr<DB::ThreadGroupStatus>, unsigned long) build_docker/../src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:78:24 (clickhouse+0x1accedc3)
    #13 DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0::operator()() const build_docker/../src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:105:13 (clickhouse+0x1accedc3)
    #14 decltype(static_cast<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&>(fp)()) std::__1::__invoke_constexpr<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&) build_docker/../contrib/libcxx/include/type_traits:3648:23 (clickhouse+0x1accedc3)
    #15 decltype(auto) std::__1::__apply_tuple_impl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::__1::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::__1::tuple<>&, std::__1::__tuple_indices<>) build_docker/../contrib/libcxx/include/tuple:1595:1 (clickhouse+0x1accedc3)
    #16 decltype(auto) std::__1::apply<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::__1::tuple<>&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&, std::__1::tuple<>&) build_docker/../contrib/libcxx/include/tuple:1604:1 (clickhouse+0x1accedc3)
    #17 ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()::operator()() build_docker/../src/Common/ThreadPool.h:188:13 (clickhouse+0x1accedc3)
    #18 decltype(static_cast<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(fp)()) std::__1::__invoke<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&) build_docker/../contrib/libcxx/include/type_traits:3640:23 (clickhouse+0x1accedc3)
    #19 void std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&>(ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'()&) build_docker/../contrib/libcxx/include/__functional/invoke.h:61:9 (clickhouse+0x1accedc3)
    #20 std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>::operator()() build_docker/../contrib/libcxx/include/__functional/function.h:230:12 (clickhouse+0x1accedc3)
    #21 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()> >(std::__1::__function::__policy_storage const*) build_docker/../contrib/libcxx/include/__functional/function.h:711:16 (clickhouse+0x1accedc3)
    #22 std::__1::__function::__policy_func<void ()>::operator()() const build_docker/../contrib/libcxx/include/__functional/function.h:843:16 (clickhouse+0xaff4fed)
    #23 std::__1::function<void ()>::operator()() const build_docker/../contrib/libcxx/include/__functional/function.h:1184:12 (clickhouse+0xaff4fed)
    #24 ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) build_docker/../src/Common/ThreadPool.cpp:281:17 (clickhouse+0xaff4fed)
    #25 void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()::operator()() const build_docker/../src/Common/ThreadPool.cpp:143:73 (clickhouse+0xaff8830)
    #26 decltype(static_cast<void>(fp)()) std::__1::__invoke<void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&) build_docker/../contrib/libcxx/include/type_traits:3640:23 (clickhouse+0xaff8830)
    #27 void std::__1::__thread_execute<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(std::__1::tuple<void, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>&, std::__1::__tuple_indices<>) build_docker/../contrib/libcxx/include/thread:282:5 (clickhouse+0xaff8830)
    #28 void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()> >(void*) build_docker/../contrib/libcxx/include/thread:293:5 (clickhouse+0xaff8830)

  Thread T411 'TCPHandler' (tid=2253, running) created by thread T249 at:
    #0 pthread_create <null> (clickhouse+0xaea771d)
    #1 Poco::ThreadImpl::startImpl(Poco::SharedPtr<Poco::Runnable, Poco::ReferenceCounter, Poco::ReleasePolicy<Poco::Runnable> >) build_docker/../contrib/poco/Foundation/src/Thread_POSIX.cpp:202:6 (clickhouse+0x1fb78435)
    #2 Poco::Thread::start(Poco::Runnable&) build_docker/../contrib/poco/Foundation/src/Thread.cpp:128:2 (clickhouse+0x1fb79cac)
    #3 Poco::PooledThread::start() build_docker/../contrib/poco/Foundation/src/ThreadPool.cpp:85:10 (clickhouse+0x1fb7dfc6)
    #4 Poco::ThreadPool::getThread() build_docker/../contrib/poco/Foundation/src/ThreadPool.cpp:461:14 (clickhouse+0x1fb7dfc6)
    #5 Poco::ThreadPool::startWithPriority(Poco::Thread::Priority, Poco::Runnable&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) build_docker/../contrib/poco/Foundation/src/ThreadPool.cpp:365:2 (clickhouse+0x1fb7e3c7)
    #6 Poco::Net::TCPServerDispatcher::enqueue(Poco::Net::StreamSocket const&) build_docker/../contrib/poco/Net/src/TCPServerDispatcher.cpp:152:17 (clickhouse+0x1f917e2e)
    #7 Poco::Net::TCPServer::run() build_docker/../contrib/poco/Net/src/TCPServer.cpp:148:21 (clickhouse+0x1f916947)
    #8 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/../contrib/poco/Foundation/src/Thread.cpp:55:11 (clickhouse+0x1fb7a36f)
    #9 Poco::ThreadImpl::runnableEntry(void*) build_docker/../contrib/poco/Foundation/src/Thread_POSIX.cpp:345:27 (clickhouse+0x1fb789e7)

SUMMARY: ThreadSanitizer: data race build_docker/../contrib/libcxx/include/__utility/swap.h:37:7 in std::__1::enable_if<(is_move_constructible<DB::Context const*>::value) && (is_move_assignable<DB::Context const*>::value), void>::type std::__1::swap<DB::Context const*>(DB::Context const*&, DB::Context const*&)
==================

@qoega qoega marked this pull request as draft May 16, 2022 15:17
@qoega qoega force-pushed the improve-opentelemetry branch 2 times, most recently from 4f2028f to f20b5d5 Compare June 3, 2022 12:07
@qoega qoega mentioned this pull request Jun 3, 2022
Merged
@qoega
Copy link
Copy Markdown
Member Author

qoega commented Jun 3, 2022

This PR instead #37837

@qoega qoega closed this Jun 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vdimir vdimir vdimir approved these changes

Assignees

@vdimir vdimir

Labels

pr-improvement Pull request with some product improvements

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@qoega @vdimir @robot-ch-test-poll2