Closed
Conversation
Enmk
pushed a commit
that referenced
this pull request
Sep 21, 2022
…s and parallel KILL
Right now it is possible to call QueryStatus::addPipelineExecutor() when
the executors_mutex already acquired, it is possible when the query was
cancelled via KILL QUERY.
Here I will show some traces from debugger from a real example, where
tons of ProcessList::insert() got deadlocked.
Let's look at the lock owner for one of the threads that was deadlocked
in ProcessList::insert():
(gdb) p *mutex
$2 = {
__data = {
__owner = 46899,
},
}
And now let's see the stack trace of the 46899:
#0 __lll_lock_wait () at ../sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:103
#1 0x00007fb65569b714 in __GI___pthread_mutex_lock (mutex=0x7fb4a9d15298) at ../nptl/pthread_mutex_lock.c:80
#2 0x000000001b6edd91 in pthread_mutex_lock (arg=0x7fb4a9d15298) at ../src/Common/ThreadFuzzer.cpp:317
#3 std::__1::__libcpp_mutex_lock (__m=0x7fb4a9d15298) at ../contrib/libcxx/include/__threading_support:303
#4 std::__1::mutex::lock (this=0x7fb4a9d15298) at ../contrib/libcxx/src/mutex.cpp:33
#5 0x0000000014c7ae63 in std::__1::lock_guard<std::__1::mutex>::lock_guard (__m=..., this=<optimized out>) at ../contrib/libcxx/include/__mutex_base:91
#6 DB::QueryStatus::addPipelineExecutor (this=0x7fb4a9d14f90, e=0x80) at ../src/Interpreters/ProcessList.cpp:372
#7 0x0000000015bee4a7 in DB::PipelineExecutor::PipelineExecutor (this=0x7fb4b1e53618, processors=..., elem=<optimized out>) at ../src/Processors/Executors/PipelineExecutor.cpp:54
#12 std::__1::make_shared<DB::PipelineExecutor, std::__1::vector<std::__1::shared_ptr<DB::IProcessor>, std::__1::allocator<std::__1::shared_ptr<DB::IProcessor> > >&, DB::QueryStatus*&, void> (__args=@0x7fb63095b9b0: 0x7fb4a9d14f90, __args=@0x7fb63095b9b0: 0x7fb4a9d14f90) at ../contrib/libcxx/include/__memory/shared_ptr.h:963
#13 DB::QueryPipelineBuilder::execute (this=0x7fb63095b8b0) at ../src/QueryPipeline/QueryPipelineBuilder.cpp:552
#14 0x00000000158c6c27 in DB::Connection::sendExternalTablesData (this=0x7fb6545e9d98, data=...) at ../src/Client/Connection.cpp:797
#27 0x0000000014043a81 in DB::RemoteQueryExecutorRoutine::operator() (this=0x7fb63095bf20, sink=...) at ../src/QueryPipeline/RemoteQueryExecutorReadContext.cpp:46
#32 0x000000000a16dd4f in make_fcontext () at ../contrib/boost/libs/context/src/asm/make_x86_64_sysv_elf_gas.S:71
And also in the logs you can see very strange things for this thread:
2022.09.13 14:14:51.228979 [ 51145 ] {1712D4E914EC7C99} <Debug> Connection (localhost:9000): Sent data for 1 external tables, total 11 rows in 0.00046389 sec., 23688 rows/sec., 3.84 KiB (8.07 MiB/sec.), compressed 1.1070121092649958 times to 3.47 KiB (7.29 MiB/sec.)
...
2022.09.13 14:14:51.719402 [ 46899 ] {7c90ffa4-1dc8-42fd-938c-4e307c244394} <Debug> executeQuery: (from 10.101.15.181:42478) KILL QUERY WHERE query_id = '1712D4E914EC7C99' (stage: Complete)
2022.09.13 14:14:51.719488 [ 46899 ] {7c90ffa4-1dc8-42fd-938c-4e307c244394} <Debug> executeQuery: (internal) SELECT query_id, user, query FROM system.processes WHERE query_id = '1712D4E914EC7C99' (stage: Complete)
2022.09.13 14:14:51.719754 [ 46899 ] {7c90ffa4-1dc8-42fd-938c-4e307c244394} <Trace> ContextAccess (default): Access granted: SELECT(user, query_id, query) ON system.processes
2022.09.13 14:14:51.720544 [ 46899 ] {7c90ffa4-1dc8-42fd-938c-4e307c244394} <Trace> InterpreterSelectQuery: FetchColumns -> Complete
2022.09.13 14:14:53.228964 [ 46899 ] {7c90ffa4-1dc8-42fd-938c-4e307c244394} <Debug> Connection (localhost:9000): Sent data for 2 scalars, total 2 rows in 2.6838e-05 sec., 73461 rows/sec., 68.00 B (2.38 MiB/sec.), compressed 0.4594594594594595 times to 148.00 B (5.16 MiB/sec.)
How is this possible? The answer is fibers and query cancellation
routine. During cancellation of async queries it going into fibers again
and try to do this gracefully. However because of this during canceling
query it may call QueryStatus::addPipelineExecutor() from
QueryStatus::cancelQuery().
Signed-off-by: Azat Khuzhin <[email protected]>
Enmk
pushed a commit
that referenced
this pull request
Oct 26, 2022
…verflow
UBSAN report:
$ UBSAN_SYMBOLIZER_PATH=/usr/bin/llvm-symbolizer-14 UBSAN_OPTIONS=print_stacktrace=1 ./unit_tests_dbms --gtest_filter=*Gorilla*
../src/Compression/tests/gtest_compressionCodec.cpp:1216:47: runtime error: signed integer overflow: 23 * 100000000 cannot be represented in type 'int'
#0 0x14f67fd1 in auto (anonymous namespace)::$_6::operator()(int) const::'lambda'(auto)::operator()<int>(auto) const build_docker/../src/Compression/tests/gtest_compressionCodec.cpp:1216:47
#1 0x14f67fd1 in (anonymous namespace)::CodecTestSequence (anonymous namespace)::generateSeq<long, (anonymous namespace)::$_6::operator()(int) const::'lambda'(auto), int, int>((anonymous namespace)::$_6::operator()(int) const::'lambda'(auto), char const*, int, int) build_docker/../src/Compression/tests/gtest_compressionCodec.cpp:394:36
#2 0x14f67fd1 in auto (anonymous namespace)::GCompatibilityTestSequence<long>() build_docker/../src/Compression/tests/gtest_compressionCodec.cpp:1224:12
#3 0x14f3c7f3 in (anonymous namespace)::gtest_GorillaCodecTestCompatibility_EvalGenerator_() build_docker/../src/Compression/tests/gtest_compressionCodec.cpp:1227:1
#4 0x14f6bdb5 in testing::internal::ParameterizedTestSuiteInfo<(anonymous namespace)::CodecTestCompatibility>::RegisterTests() build_docker/../contrib/googletest/googletest/include/gtest/internal/gtest-param-util.h:553:45
#5 0x27d87988 in testing::internal::ParameterizedTestSuiteRegistry::RegisterTests() build_docker/../contrib/googletest/googletest/include/gtest/internal/gtest-param-util.h:726:24
#6 0x27d87988 in testing::internal::UnitTestImpl::RegisterParameterizedTests() build_docker/../contrib/googletest/googletest/src/gtest.cc:2805:34
#7 0x27d87988 in testing::internal::UnitTestImpl::PostFlagParsingInit() build_docker/../contrib/googletest/googletest/src/gtest.cc:5492:5
#8 0x27d9d002 in void testing::internal::InitGoogleTestImpl<char>(int*, char**) build_docker/../contrib/googletest/googletest/src/gtest.cc:6499:22
#9 0x14fd5495 in main build_docker/../src/Coordination/tests/gtest_coordination.cpp:2189:5
#10 0x7f8c29005209 (/lib/x86_64-linux-gnu/libc.so.6+0x29209) (BuildId: 71a7c7b97bc0b3e349a3d8640252655552082bf5)
#11 0x7f8c290052bb in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x292bb) (BuildId: 71a7c7b97bc0b3e349a3d8640252655552082bf5)
#12 0x14ce356d in _start (/work1/azat/tmp/42190/unit_tests_dbms+0x14ce356d) (BuildId: 482550e3f8d45f06e8c7f8147f427ee798c1f645)
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior ../src/Compression/tests/gtest_compressionCodec.cpp:1216:47 in
Signed-off-by: Azat Khuzhin <[email protected]>
Enmk
pushed a commit
that referenced
this pull request
Apr 29, 2024
Fixes integration test_reload_certificate/test.py::test_first_than_second_cert --- E Exception: Sanitizer assert found for instance ================== E WARNING: ThreadSanitizer: data race (pid=1) E Write of size 8 at 0x7b2800025d30 by thread T2 (mutexes: write M0, write M1): E #0 free <null> (clickhouse+0x709a3e5) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #1 CRYPTO_free build_docker/./contrib/openssl/crypto/mem.c:282:5 (clickhouse+0x2015f8ea) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #2 EVP_PKEY_free build_docker/./contrib/openssl/crypto/evp/p_lib.c:1809:5 (clickhouse+0x2012a751) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #3 Poco::Crypto::EVPPKey::~EVPPKey() build_docker/./base/poco/Crypto/src/EVPPKey.cpp:121:17 (clickhouse+0x1d00ffa9) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #4 DB::CertificateReloader::Data::~Data() build_docker/./src/Server/CertificateReloader.h:71:12 (clickhouse+0x194fb42d) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #5 std::__1::default_delete<DB::CertificateReloader::Data const>::operator()[abi:v15000](DB::CertificateReloader::Data const*) const build_docker/./contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:48:5 (clickhouse+0x194fb42d) E #6 std::__1::__shared_ptr_pointer<DB::CertificateReloader::Data const*, std::__1::default_delete<DB::CertificateReloader::Data const>, std::__1::allocator<DB::CertificateReloader::Data const>>::__on_zero_shared() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:263:5 (clickhouse+0x194fb42d) E #7 std::__1::__shared_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:174:9 (clickhouse+0x194fade0) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #8 std::__1::__shared_weak_count::__release_shared[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:215:27 (clickhouse+0x194fade0) E #9 std::__1::shared_ptr<DB::CertificateReloader::Data const>::~shared_ptr[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:702:23 (clickhouse+0x194fade0) E #10 std::__1::shared_ptr<DB::CertificateReloader::Data const>::operator=[abi:v15000](std::__1::shared_ptr<DB::CertificateReloader::Data const>&&) build_docker/./contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:723:9 (clickhouse+0x194fade0) E #11 MultiVersion<DB::CertificateReloader::Data>::set(std::__1::unique_ptr<DB::CertificateReloader::Data const, std::__1::default_delete<DB::CertificateReloader::Data const>>&&) build_docker/./src/Common/MultiVersion.h:76:25 (clickhouse+0x194fade0) E #12 DB::CertificateReloader::tryLoad(Poco::Util::AbstractConfiguration const&) build_docker/./src/Server/CertificateReloader.cpp:83:18 (clickhouse+0x194f94ca) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #13 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6::operator()(Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool) const build_docker/./programs/server/Server.cpp:1546:45 (clickhouse+0xf384df7) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #14 decltype(std::declval<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&>()(std::declval<Poco::AutoPtr<Poco::Util::AbstractConfiguration>>(), std::declval<bool>())) std::__1::__invoke[abi:v15000]<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf3827a9) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #15 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6&, Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf3827a9) E #16 std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6, void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::operator()[abi:v15000](Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf3827a9) E #17 void std::__1::__function::__policy_invoker<void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::__call_impl<std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_6, void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>>(std::__1::__function::__policy_storage const*, Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf3827a9) E #18 std::__1::__function::__policy_func<void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::operator()[abi:v15000](Poco::AutoPtr<Poco::Util::AbstractConfiguration>&&, bool&&) const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0x19fd2cbe) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #19 std::__1::function<void (Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool)>::operator()(Poco::AutoPtr<Poco::Util::AbstractConfiguration>, bool) const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0x19fd2cbe) E #20 DB::ConfigReloader::reloadIfNewer(bool, bool, bool, bool) build_docker/./src/Common/Config/ConfigReloader.cpp:150:13 (clickhouse+0x19fd2cbe) E #21 DB::ConfigReloader::reload() build_docker/./src/Common/Config/ConfigReloader.h:51:21 (clickhouse+0xf38767c) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #22 DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13::operator()() const build_docker/./programs/server/Server.cpp:1731:31 (clickhouse+0xf38767c) E #23 decltype(std::declval<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&>()()) std::__1::__invoke[abi:v15000]<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:394:23 (clickhouse+0xf38767c) E #24 void std::__1::__invoke_void_return_wrapper<void, true>::__call<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&>(DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13&) build_docker/./contrib/llvm-project/libcxx/include/__functional/invoke.h:479:9 (clickhouse+0xf38767c) E #25 std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13, void ()>::operator()[abi:v15000]() build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:235:12 (clickhouse+0xf38767c) E #26 void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<DB::Server::main(std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>> const&)::$_13, void ()>>(std::__1::__function::__policy_storage const*) build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:716:16 (clickhouse+0xf38767c) E #27 std::__1::__function::__policy_func<void ()>::operator()[abi:v15000]() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:848:16 (clickhouse+0x16907aa0) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #28 std::__1::function<void ()>::operator()() const build_docker/./contrib/llvm-project/libcxx/include/__functional/function.h:1187:12 (clickhouse+0x16907aa0) E #29 DB::Context::reloadConfig() const build_docker/./src/Interpreters/Context.cpp:4357:5 (clickhouse+0x16907aa0) E #30 DB::InterpreterSystemQuery::execute() build_docker/./src/Interpreters/InterpreterSystemQuery.cpp:577:29 (clickhouse+0x17e78c19) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #31 DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) build_docker/./src/Interpreters/executeQuery.cpp:1195:40 (clickhouse+0x17e3e462) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #32 DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::shared_ptr<DB::Context>, DB::QueryFlags, DB::QueryProcessingStage::Enum) build_docker/./src/Interpreters/executeQuery.cpp:1374:26 (clickhouse+0x17e39837) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #33 DB::TCPHandler::runImpl() build_docker/./src/Server/TCPHandler.cpp:518:54 (clickhouse+0x195cc651) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #34 DB::TCPHandler::run() build_docker/./src/Server/TCPHandler.cpp:2329:9 (clickhouse+0x195e8707) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #35 Poco::Net::TCPServerConnection::start() build_docker/./base/poco/Net/src/TCPServerConnection.cpp:43:3 (clickhouse+0x1d00d942) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #36 Poco::Net::TCPServerDispatcher::run() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:115:20 (clickhouse+0x1d00e1b1) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #37 Poco::PooledThread::run() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:188:14 (clickhouse+0x1d20f2e6) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #38 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/./base/poco/Foundation/src/Thread.cpp:45:11 (clickhouse+0x1d20d5af) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #39 Poco::ThreadImpl::runnableEntry(void*) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:335:27 (clickhouse+0x1d20ba69) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E E Previous atomic write of size 4 at 0x7b2800025d30 by thread T3 (mutexes: write M2): E #0 CRYPTO_DOWN_REF build_docker/./contrib/openssl/include/internal/refcount.h:51:12 (clickhouse+0x2012a6e6) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #1 EVP_PKEY_free build_docker/./contrib/openssl/crypto/evp/p_lib.c:1795:5 (clickhouse+0x2012a6e6) E #2 ssl_cert_clear_certs build_docker/./contrib/openssl/ssl/ssl_cert.c:246:9 (clickhouse+0x1ffafd37) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #3 ssl_cert_free build_docker/./contrib/openssl/ssl/ssl_cert.c:277:5 (clickhouse+0x1ffafd37) E #4 ossl_ssl_connection_free build_docker/./contrib/openssl/ssl/ssl_lib.c:1458:5 (clickhouse+0x1ffba6af) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #5 SSL_free build_docker/./contrib/openssl/ssl/ssl_lib.c:1417:9 (clickhouse+0x1ffb920e) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #6 Poco::Net::SecureSocketImpl::reset() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:583:3 (clickhouse+0x1cfaac60) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #7 Poco::Net::SecureSocketImpl::~SecureSocketImpl() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureSocketImpl.cpp:80:3 (clickhouse+0x1cfaac60) E #8 Poco::Net::SecureStreamSocketImpl::~SecureStreamSocketImpl() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureStreamSocketImpl.cpp:52:1 (clickhouse+0x1cfb15dd) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #9 Poco::Net::SecureStreamSocketImpl::~SecureStreamSocketImpl() build_docker/./base/poco/NetSSL_OpenSSL/src/SecureStreamSocketImpl.cpp:43:1 (clickhouse+0x1cfb15dd) E #10 Poco::RefCountedObject::release() const build_docker/./base/poco/Foundation/include/Poco/RefCountedObject.h:86:13 (clickhouse+0x1cffc81e) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #11 Poco::Net::Socket::~Socket() build_docker/./base/poco/Net/src/Socket.cpp:68:10 (clickhouse+0x1cffc81e) E #12 Poco::Net::StreamSocket::~StreamSocket() build_docker/./base/poco/Net/src/StreamSocket.cpp:63:1 (clickhouse+0x1d009c39) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #13 Poco::Net::TCPConnectionNotification::~TCPConnectionNotification() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:43:2 (clickhouse+0x1d00ef50) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #14 Poco::Net::TCPConnectionNotification::~TCPConnectionNotification() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:42:2 (clickhouse+0x1d00ef50) E #15 Poco::RefCountedObject::release() const build_docker/./base/poco/Foundation/include/Poco/RefCountedObject.h:86:13 (clickhouse+0x1d00e203) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #16 Poco::AutoPtr<Poco::Notification>::~AutoPtr() build_docker/./base/poco/Foundation/include/Poco/AutoPtr.h:91:19 (clickhouse+0x1d00e203) E #17 Poco::Net::TCPServerDispatcher::run() build_docker/./base/poco/Net/src/TCPServerDispatcher.cpp:122:3 (clickhouse+0x1d00e203) E #18 Poco::PooledThread::run() build_docker/./base/poco/Foundation/src/ThreadPool.cpp:188:14 (clickhouse+0x1d20f2e6) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #19 Poco::(anonymous namespace)::RunnableHolder::run() build_docker/./base/poco/Foundation/src/Thread.cpp:45:11 (clickhouse+0x1d20d5af) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E #20 Poco::ThreadImpl::runnableEntry(void*) build_docker/./base/poco/Foundation/src/Thread_POSIX.cpp:335:27 (clickhouse+0x1d20ba69) (BuildId: 706d92b17db171493f293d517643f726ee1b7b1e) E
Enmk
pushed a commit
that referenced
this pull request
Apr 29, 2024
This commit adresses below leaksan find. I tried to reproduce this locally but 02802_clickhouse_disks_s3_copy.sh but couldn't. clickhouse-disks did not do anything useful (it would not even print logging), neither with a standard build nor with a leaksan build. Could not find further documentation of it or even what this tool is supposed to do, perhaps it is just for internal use. Also, - line numbers in the leaksan report were partially missing, - I am not really sure how Sha256HMACOpenSSLImpl::Calculate is calling into hmac_init (there must be some sort of static initialization somewhere but I did not find it), and - my fix is in a weird place due to other restrictions (see the commit in the aws-sdk-cpp contrib repo). The chance that this fix fixes the leak are low. If it doesn't work, add "# Tag no-asan" + a comment to 02802_clickhouse_disks_s3_copy.sh and don't worry further. EDIT: The commit fixes the issue, everything is good. https://s3.amazonaws.com/clickhouse-test-reports/59870/b452e3d1ab87b8cc5810693aeea28f69ad28d671/stateless_tests__asan__[3_4].html 2024-03-19 08:34:03 ================================================================= 2024-03-19 08:34:03 ==13149==ERROR: LeakSanitizer: detected memory leaks 2024-03-19 08:34:03 2024-03-19 08:34:03 Direct leak of 904 byte(s) in 1 object(s) allocated from: 2024-03-19 08:34:03 #0 0x55f9cb5a18ee in malloc (/usr/bin/clickhouse+0xa9a48ee) (BuildId: b3766b865d6580f6dcba75acf37673d4aeedc2b6) 2024-03-19 08:34:03 #1 0x55fa01e34070 in CRYPTO_malloc build_docker/./contrib/openssl/crypto/mem.c:202:11 2024-03-19 08:34:03 #2 0x55fa01e34070 in CRYPTO_zalloc build_docker/./contrib/openssl/crypto/mem.c:222:11 2024-03-19 08:34:03 #3 0x55fa01d6dcca in ossl_err_get_state_int build_docker/./contrib/openssl/crypto/err/err.c:691:17 2024-03-19 08:34:03 #4 0x55fa01d71748 in ERR_set_mark build_docker/./contrib/openssl/crypto/err/err_mark.c:19:10 2024-03-19 08:34:03 #5 0x55fa01f4735b in ossl_prov_digest_load_from_params build_docker/./contrib/openssl/providers/common/provider_util.c:194:5 2024-03-19 08:34:03 #6 0x55fa01ff467a in hmac_set_ctx_params build_docker/./contrib/openssl/providers/implementations/macs/hmac_prov.c:307:10 2024-03-19 08:34:03 #7 0x55fa01ff3ef2 in hmac_init build_docker/./contrib/openssl/providers/implementations/macs/hmac_prov.c:169:37 2024-03-19 08:34:03 #8 0x55f9fb83c75b in Aws::Utils::Crypto::Sha256HMACOpenSSLImpl::Calculate(Aws::Utils::Array<unsigned char> const&, Aws::Utils::Array<unsigned char> const&) (/usr/bin/clickhouse+0x3ac3f75b) (BuildId: b3766b865d6580f6dcba75acf37673d4aeedc2b6) 2024-03-19 08:34:03 #9 0x55f9fb82ebeb in Aws::Utils::Crypto::Sha256HMAC::Calculate(Aws::Utils::Array<unsigned char> const&, Aws::Utils::Array<unsigned char> const&) (/usr/bin/clickhouse+0x3ac31beb) (BuildId: b3766b865d6580f6dcba75acf37673d4aeedc2b6) 2024-03-19 08:34:03 #10 0x55f9fb6d5afd in Aws::Client::AWSAuthV4Signer::ComputeHash(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) const (/usr/bin/clickhouse+0x3aad8afd) (BuildId: b3766b865d6580f6dcba75acf37673d4aeedc2b6) 2024-03-19 08:34:03 #11 0x55f9fb6e0bad in Aws::Client::AWSAuthV4Signer::SignRequest(Aws::Http::HttpRequest&, char const*, char const*, bool) const (/usr/bin/clickhouse+0x3aae3bad) (BuildId: b3766b865d6580f6dcba75acf37673d4aeedc2b6) 2024-03-19 08:34:03 #12 0x55f9fb72651d in bool smithy::components::tracing::TracingUtils::MakeCallWithTiming<bool>(std::__1::function<bool ()>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, smithy::components::tracing::Meter const&, std::__1::map<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, std::__1::less<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>, std::__1::allocator<std::__1::pair<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>>>>&&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&) (/usr/bin/clickhouse+0x3ab2951d) (BuildId: b3766b865d6580f6dcba75acf37673d4aeedc2b6) 2024-03-19 08:34:03 #13 0x55f9fb70adcb in Aws::Client::AWSClient::AttemptOneRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const*, char const*, char const*) const (/usr/bin/clickhouse+0x3ab0ddcb) (BuildId: b3766b865d6580f6dcba75acf37673d4aeedc2b6) 2024-03-19 08:34:03 #14 0x55f9fb6fdb17 in Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char cons
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
...
Detailed description / Documentation draft:
...
By adding documentation, you'll allow users to try your new feature immediately, not when someone else will have time to document it later. Documentation is necessary for all features that affect user experience in any way. You can add brief documentation draft above, or add documentation right into your patch as Markdown files in docs folder.
If you are doing this for the first time, it's recommended to read the lightweight Contributing to ClickHouse Documentation guide first.