Skip to content

Data race in KeeperTCPHandler #42528

Closed
#42532
Closed
Data race in KeeperTCPHandler#42528
#42532
Assignees
antonio2368
Labels
testingSpecial issue with list of bugs found by CI
@tavplubix

Description

@tavplubix
tavplubix
opened on Oct 20, 2022

https://s3.amazonaws.com/clickhouse-test-reports/42134/e4f93149ff4c671b6e6aec3ce78e550ef5e27edf/stress_test__tsan_.html

WARNING: ThreadSanitizer: data race (pid=1444)
  Write of size 8 at 0x7b4800e9bfd0 by thread T384:
    #0 DB::KeeperTCPHandler::runImpl() build_docker/../src/Server/KeeperTCPHandler.cpp:353:24 (clickhouse+0x1d24c977) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #1 DB::KeeperTCPHandler::run() build_docker/../src/Server/KeeperTCPHandler.cpp:259:5 (clickhouse+0x1d24bf39) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #2 Poco::Net::TCPServerConnection::start() build_docker/../contrib/poco/Net/src/TCPServerConnection.cpp:43:3 (clickhouse+0x20af0142) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #3 Poco::Net::TCPServerDispatcher::run() build_docker/../contrib/poco/Net/src/TCPServerDispatcher.cpp:115:20 (clickhouse+0x20af09b2) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #4 Poco::PooledThread::run() build_docker/../contrib/poco/Foundation/src/ThreadPool.cpp:199:14 (clickhouse+0x20d48759) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #5 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/../contrib/poco/Foundation/src/Thread.cpp:55:11 (clickhouse+0x20d46b6f) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #6 Poco::ThreadImpl::runnableEntry(void*) build_docker/../contrib/poco/Foundation/src/Thread_POSIX.cpp:345:27 (clickhouse+0x20d451a7) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)

  Previous read of size 8 at 0x7b4800e9bfd0 by thread T704 (mutexes: write M0):
    #0 DB::KeeperTCPHandler::dumpStats(DB::WriteBufferFromOwnString&, bool) build_docker/../src/Server/KeeperTCPHandler.cpp:597:13 (clickhouse+0x1d251d82) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #1 DB::KeeperTCPHandler::dumpConnections(DB::WriteBufferFromOwnString&, bool) build_docker/../src/Server/KeeperTCPHandler.cpp:665:15 (clickhouse+0x1d2530bc) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #2 DB::ConsCommand::run() build_docker/../src/Coordination/FourLetterCommand.cpp:287:5 (clickhouse+0x1d895a83) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #3 DB::KeeperTCPHandler::tryExecuteFourLetterWordCmd(int) build_docker/../src/Server/KeeperTCPHandler.cpp:515:39 (clickhouse+0x1d2509af) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #4 DB::KeeperTCPHandler::runImpl() build_docker/../src/Server/KeeperTCPHandler.cpp:328:9 (clickhouse+0x1d24c41b) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #5 DB::KeeperTCPHandler::run() build_docker/../src/Server/KeeperTCPHandler.cpp:259:5 (clickhouse+0x1d24bf39) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #6 Poco::Net::TCPServerConnection::start() build_docker/../contrib/poco/Net/src/TCPServerConnection.cpp:43:3 (clickhouse+0x20af0142) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #7 Poco::Net::TCPServerDispatcher::run() build_docker/../contrib/poco/Net/src/TCPServerDispatcher.cpp:115:20 (clickhouse+0x20af09b2) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #8 Poco::PooledThread::run() build_docker/../contrib/poco/Foundation/src/ThreadPool.cpp:199:14 (clickhouse+0x20d48759) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #9 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/../contrib/poco/Foundation/src/Thread.cpp:55:11 (clickhouse+0x20d46b6f) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #10 Poco::ThreadImpl::runnableEntry(void*) build_docker/../contrib/poco/Foundation/src/Thread_POSIX.cpp:345:27 (clickhouse+0x20d451a7) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)

  Location is heap block of size 344 at 0x7b4800e9bf80 allocated by thread T384:
    #0 operator new(unsigned long) <null> (clickhouse+0xb93a706) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #1 DB::KeeperTCPHandlerFactory::createConnection(Poco::Net::StreamSocket const&, DB::TCPServer&) build_docker/../src/Server/KeeperTCPHandlerFactory.h:51:20 (clickhouse+0x132985c6) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #2 DB::TCPServerConnectionFactoryImpl::createConnection(Poco::Net::StreamSocket const&) build_docker/../src/Server/TCPServer.cpp:17:25 (clickhouse+0x1d2a838e) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #3 Poco::Net::TCPServerDispatcher::run() build_docker/../contrib/poco/Net/src/TCPServerDispatcher.cpp:113:77 (clickhouse+0x20af098e) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #4 Poco::PooledThread::run() build_docker/../contrib/poco/Foundation/src/ThreadPool.cpp:199:14 (clickhouse+0x20d48759) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #5 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/../contrib/poco/Foundation/src/Thread.cpp:55:11 (clickhouse+0x20d46b6f) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #6 Poco::ThreadImpl::runnableEntry(void*) build_docker/../contrib/poco/Foundation/src/Thread_POSIX.cpp:345:27 (clickhouse+0x20d451a7) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)

  Mutex M0 (0x000025cfa018) created at:
    #0 pthread_mutex_lock <null> (clickhouse+0xb8d8a9a) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #1 std::__1::__libcpp_mutex_lock(pthread_mutex_t*) build_docker/../contrib/libcxx/include/__threading_support:303:10 (clickhouse+0x2450bff9) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #2 std::__1::mutex::lock() build_docker/../contrib/libcxx/src/mutex.cpp:33:14 (clickhouse+0x2450bff9)
    #3 std::__1::lock_guard<std::__1::mutex>::lock_guard(std::__1::mutex&) build_docker/../contrib/libcxx/include/__mutex_base:91:27 (clickhouse+0x1d24bac6) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #4 DB::KeeperTCPHandler::registerConnection(DB::KeeperTCPHandler*) build_docker/../src/Server/KeeperTCPHandler.cpp:650:21 (clickhouse+0x1d24bac6)
    #5 DB::KeeperTCPHandler::KeeperTCPHandler(Poco::Util::AbstractConfiguration const&, std::__1::shared_ptr<DB::KeeperDispatcher>, Poco::Timespan, Poco::Timespan, Poco::Net::StreamSocket const&) build_docker/../src/Server/KeeperTCPHandler.cpp:232:5 (clickhouse+0x1d24bac6)
    #6 DB::KeeperTCPHandlerFactory::createConnection(Poco::Net::StreamSocket const&, DB::TCPServer&) build_docker/../src/Server/KeeperTCPHandlerFactory.h:51:24 (clickhouse+0x1329867a) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #7 DB::TCPServerConnectionFactoryImpl::createConnection(Poco::Net::StreamSocket const&) build_docker/../src/Server/TCPServer.cpp:17:25 (clickhouse+0x1d2a838e) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #8 Poco::Net::TCPServerDispatcher::run() build_docker/../contrib/poco/Net/src/TCPServerDispatcher.cpp:113:77 (clickhouse+0x20af098e) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #9 Poco::PooledThread::run() build_docker/../contrib/poco/Foundation/src/ThreadPool.cpp:199:14 (clickhouse+0x20d48759) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #10 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/../contrib/poco/Foundation/src/Thread.cpp:55:11 (clickhouse+0x20d46b6f) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #11 Poco::ThreadImpl::runnableEntry(void*) build_docker/../contrib/poco/Foundation/src/Thread_POSIX.cpp:345:27 (clickhouse+0x20d451a7) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)

  Thread T384 'KeeperHandler' (tid=2093, running) created by thread T270 at:
    #0 pthread_create <null> (clickhouse+0xb8badbd) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #1 Poco::ThreadImpl::startImpl(Poco::SharedPtr<Poco::Runnable, Poco::ReferenceCounter, Poco::ReleasePolicy<Poco::Runnable>>) build_docker/../contrib/poco/Foundation/src/Thread_POSIX.cpp:202:6 (clickhouse+0x20d44bb3) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #2 Poco::Thread::start(Poco::Runnable&) build_docker/../contrib/poco/Foundation/src/Thread.cpp:128:2 (clickhouse+0x20d4652c) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #3 Poco::PooledThread::start() build_docker/../contrib/poco/Foundation/src/ThreadPool.cpp:85:10 (clickhouse+0x20d4a60b) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #4 Poco::ThreadPool::getThread() build_docker/../contrib/poco/Foundation/src/ThreadPool.cpp:461:14 (clickhouse+0x20d4a60b)
    #5 Poco::ThreadPool::startWithPriority(Poco::Thread::Priority, Poco::Runnable&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_docker/../contrib/poco/Foundation/src/ThreadPool.cpp:365:2 (clickhouse+0x20d4aa07) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #6 Poco::Net::TCPServerDispatcher::enqueue(Poco::Net::StreamSocket const&) build_docker/../contrib/poco/Net/src/TCPServerDispatcher.cpp:152:17 (clickhouse+0x20af0ec6) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #7 Poco::Net::TCPServer::run() build_docker/../contrib/poco/Net/src/TCPServer.cpp:148:21 (clickhouse+0x20aefa27) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #8 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/../contrib/poco/Foundation/src/Thread.cpp:55:11 (clickhouse+0x20d46b6f) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #9 Poco::ThreadImpl::runnableEntry(void*) build_docker/../contrib/poco/Foundation/src/Thread_POSIX.cpp:345:27 (clickhouse+0x20d451a7) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)

  Thread T704 'KeeperHandler' (tid=3432, running) created by thread T49 at:
    #0 pthread_create <null> (clickhouse+0xb8badbd) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #1 Poco::ThreadImpl::startImpl(Poco::SharedPtr<Poco::Runnable, Poco::ReferenceCounter, Poco::ReleasePolicy<Poco::Runnable>>) build_docker/../contrib/poco/Foundation/src/Thread_POSIX.cpp:202:6 (clickhouse+0x20d44bb3) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #2 Poco::Thread::start(Poco::Runnable&) build_docker/../contrib/poco/Foundation/src/Thread.cpp:128:2 (clickhouse+0x20d4652c) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #3 Poco::PooledThread::start() build_docker/../contrib/poco/Foundation/src/ThreadPool.cpp:85:10 (clickhouse+0x20d4a60b) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #4 Poco::ThreadPool::getThread() build_docker/../contrib/poco/Foundation/src/ThreadPool.cpp:461:14 (clickhouse+0x20d4a60b)
    #5 Poco::ThreadPool::startWithPriority(Poco::Thread::Priority, Poco::Runnable&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) build_docker/../contrib/poco/Foundation/src/ThreadPool.cpp:365:2 (clickhouse+0x20d4aa07) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #6 Poco::Net::TCPServerDispatcher::enqueue(Poco::Net::StreamSocket const&) build_docker/../contrib/poco/Net/src/TCPServerDispatcher.cpp:152:17 (clickhouse+0x20af0ec6) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #7 Poco::Net::TCPServer::run() build_docker/../contrib/poco/Net/src/TCPServer.cpp:148:21 (clickhouse+0x20aefa27) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #8 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/../contrib/poco/Foundation/src/Thread.cpp:55:11 (clickhouse+0x20d46b6f) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)
    #9 Poco::ThreadImpl::runnableEntry(void*) build_docker/../contrib/poco/Foundation/src/Thread_POSIX.cpp:345:27 (clickhouse+0x20d451a7) (BuildId: 19955c6bf2300c8134e1b40d3ce2a635e24c0299)

SUMMARY: ThreadSanitizer: data race build_docker/../src/Server/KeeperTCPHandler.cpp:353:24 in DB::KeeperTCPHandler::runImpl()
==================

cc: @antonio2368

Metadata

Metadata

Assignees

Labels

testingSpecial issue with list of bugs found by CI

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions