Skip to content

Very rare segfault in OpenSSL #17502

@alexey-milovidov

Description

@alexey-milovidov

Just once in two months on a cluster with 360 servers:

2020.11.11 17:39:32.916226 [ 726138 ] {} <Fatal> BaseDaemon: ########################################
2020.11.11 17:39:32.916259 [ 726138 ] {} <Fatal> BaseDaemon: (version 20.6.9.1 (official build), no build id) (from thread 725260) (query_id: 67c594db-2c20-49f1-9339-4f867d39f3bb) Received signal Segmentation fault (11)
2020.11.11 17:39:32.916280 [ 726138 ] {} <Fatal> BaseDaemon: Address: NULL pointer. Access: read. Address not mapped to object.
2020.11.11 17:39:32.916300 [ 726138 ] {} <Fatal> BaseDaemon: Stack trace: 0x131f7b4b 0x13206ad7 0x131ff41c 0x131f672b 0x13113734 0x13135bcb 0x1312a9a4 0x130fef68 0x122b644f 0x122b67e5 0x122b98c1 0xfac35c4 0xfac3bf5 0xfadb4ce 0xfadc34a 0xfae2670 0xfadc74b 0xfadce53 0xedff4b4 0xee029d8 0xfdec7e5 0xfba98db 0xfdf2ed7 0xfbd9c61 0xfbdf586 0xfbe0846 0xa272157 0xa270493 0x7fd80b358184
2020.11.11 17:39:32.916388 [ 726138 ] {} <Fatal> BaseDaemon: 3. EVP_MAC_CTX_new @ 0x131f7b4b in /usr/bin/clickhouse
2020.11.11 17:39:32.916415 [ 726138 ] {} <Fatal> BaseDaemon: 4. ? @ 0x13206ad7 in /usr/bin/clickhouse
2020.11.11 17:39:32.916423 [ 726138 ] {} <Fatal> BaseDaemon: 5. EVP_PKEY_CTX_new @ 0x131ff41c in /usr/bin/clickhouse
2020.11.11 17:39:32.916431 [ 726138 ] {} <Fatal> BaseDaemon: 6. EVP_DigestSignInit @ 0x131f672b in /usr/bin/clickhouse
2020.11.11 17:39:32.916440 [ 726138 ] {} <Fatal> BaseDaemon: 7. tls13_final_finish_mac @ 0x13113734 in /usr/bin/clickhouse
2020.11.11 17:39:32.916448 [ 726138 ] {} <Fatal> BaseDaemon: 8. tls_get_message_body @ 0x13135bcb in /usr/bin/clickhouse
2020.11.11 17:39:32.916471 [ 726138 ] {} <Fatal> BaseDaemon: 9. ? @ 0x1312a9a4 in /usr/bin/clickhouse
2020.11.11 17:39:32.916479 [ 726138 ] {} <Fatal> BaseDaemon: 10. SSL_do_handshake @ 0x130fef68 in /usr/bin/clickhouse
2020.11.11 17:39:32.916494 [ 726138 ] {} <Fatal> BaseDaemon: 11. Poco::Net::SecureSocketImpl::connectSSL(bool) @ 0x122b644f in /usr/bin/clickhouse
2020.11.11 17:39:32.916506 [ 726138 ] {} <Fatal> BaseDaemon: 12. Poco::Net::SecureSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&, bool) @ 0x122b67e5 in /usr/bin/clickhouse
2020.11.11 17:39:32.916521 [ 726138 ] {} <Fatal> BaseDaemon: 13. Poco::Net::SecureStreamSocketImpl::connect(Poco::Net::SocketAddress const&, Poco::Timespan const&) @ 0x122b98c1 in /usr/bin/clickhouse
2020.11.11 17:39:32.916530 [ 726138 ] {} <Fatal> BaseDaemon: 14. DB::Connection::connect(DB::ConnectionTimeouts const&) @ 0xfac35c4 in /usr/bin/clickhouse
2020.11.11 17:39:32.916538 [ 726138 ] {} <Fatal> BaseDaemon: 15. DB::Connection::getServerRevision(DB::ConnectionTimeouts const&) @ 0xfac3bf5 in /usr/bin/clickhouse
2020.11.11 17:39:32.916551 [ 726138 ] {} <Fatal> BaseDaemon: 16. DB::ConnectionPoolWithFailover::tryGetEntry(DB::IConnectionPool&, DB::ConnectionTimeouts const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&, DB::Settings const*, DB::QualifiedTableName const*) @ 0xfadb4ce in /usr/bin/clickhouse
2020.11.11 17:39:32.916559 [ 726138 ] {} <Fatal> BaseDaemon: 17. ? @ 0xfadc34a in /usr/bin/clickhouse
2020.11.11 17:39:32.916576 [ 726138 ] {} <Fatal> BaseDaemon: 18. PoolWithFailoverBase<DB::IConnectionPool>::getMany(unsigned long, unsigned long, unsigned long, unsigned long, bool, std::__1::function<PoolWithFailoverBase<DB::IConnectionPool>::TryResult (DB::IConnectionPool&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&)> const&, std::__1::function<unsigned long (unsigned long)> const&) @ 0xfae2670 in /usr/bin/clickhouse
2020.11.11 17:39:32.916608 [ 726138 ] {} <Fatal> BaseDaemon: 19. DB::ConnectionPoolWithFailover::getManyImpl(DB::Settings const*, DB::PoolMode, std::__1::function<PoolWithFailoverBase<DB::IConnectionPool>::TryResult (DB::IConnectionPool&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >&)> const&) @ 0xfadc74b in /usr/bin/clickhouse
2020.11.11 17:39:32.916618 [ 726138 ] {} <Fatal> BaseDaemon: 20. DB::ConnectionPoolWithFailover::getManyChecked(DB::ConnectionTimeouts const&, DB::Settings const*, DB::PoolMode, DB::QualifiedTableName const&) @ 0xfadce53 in /usr/bin/clickhouse
2020.11.11 17:39:32.916627 [ 726138 ] {} <Fatal> BaseDaemon: 21. ? @ 0xedff4b4 in /usr/bin/clickhouse
2020.11.11 17:39:32.916635 [ 726138 ] {} <Fatal> BaseDaemon: 22. DB::RemoteQueryExecutor::sendQuery() @ 0xee029d8 in /usr/bin/clickhouse
2020.11.11 17:39:32.916678 [ 726138 ] {} <Fatal> BaseDaemon: 23. DB::RemoteSource::generate() @ 0xfdec7e5 in /usr/bin/clickhouse
2020.11.11 17:39:32.916726 [ 726138 ] {} <Fatal> BaseDaemon: 24. DB::ISource::work() @ 0xfba98db in /usr/bin/clickhouse
2020.11.11 17:39:32.916735 [ 726138 ] {} <Fatal> BaseDaemon: 25. DB::SourceWithProgress::work() @ 0xfdf2ed7 in /usr/bin/clickhouse
2020.11.11 17:39:32.916751 [ 726138 ] {} <Fatal> BaseDaemon: 26. ? @ 0xfbd9c61 in /usr/bin/clickhouse
2020.11.11 17:39:32.916756 [ 726138 ] {} <Fatal> BaseDaemon: 27. ? @ 0xfbdf586 in /usr/bin/clickhouse
2020.11.11 17:39:32.916762 [ 726138 ] {} <Fatal> BaseDaemon: 28. ? @ 0xfbe0846 in /usr/bin/clickhouse
2020.11.11 17:39:32.916771 [ 726138 ] {} <Fatal> BaseDaemon: 29. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xa272157 in /usr/bin/clickhouse
2020.11.11 17:39:32.916777 [ 726138 ] {} <Fatal> BaseDaemon: 30. ? @ 0xa270493 in /usr/bin/clickhouse
2020.11.11 17:39:32.916787 [ 726138 ] {} <Fatal> BaseDaemon: 31. start_thread @ 0x8184 in /lib/x86_64-linux-gnu/libpthread-2.19.so

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugConfirmed user-visible misbehaviour in official releaseobsolete-version

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions