Skip to content

clickhouse-test: Retry DROP DATABASE in case of concurrent DROP/CREATE TABLE#29949

Closed
azat wants to merge 1 commit intoClickHouse:masterfrom
azat:tests-retry-DROP-DATABASE
Closed

clickhouse-test: Retry DROP DATABASE in case of concurrent DROP/CREATE TABLE#29949
azat wants to merge 1 commit intoClickHouse:masterfrom
azat:tests-retry-DROP-DATABASE

Conversation

@azat
Copy link
Copy Markdown
Member

@azat azat commented Oct 10, 2021 •
edited
Loading

CI 1:

2021-10-10 09:28:55 01103_optimize_drop_race_zookeeper:                                     [ UNKNOWN ] 0.00 sec. - Test internal error: ServerException
2021-10-10 09:28:55 Code: 219.
2021-10-10 09:28:55 DB::Exception: New table appeared in database being dropped or detached. Try again.. Stack trace:
Details 
2021-10-10 09:28:55
2021-10-10 09:28:55 0. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/exception:133: std::exception::capture() @ 0x14e44ba2 in /usr/bin/clickhouse
2021-10-10 09:28:55 1. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/exception:111: std::exception::exception() @ 0x14e44b70 in /usr/bin/clickhouse
2021-10-10 09:28:55 2. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x26e107c0 in /usr/bin/clickhouse
2021-10-10 09:28:55 3. ./obj-x86_64-linux-gnu/../src/Common/Exception.cpp:57: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x14e2662e in /usr/bin/clickhouse
2021-10-10 09:28:55 4. ./obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.cpp:351: DB::DatabaseCatalog::detachDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, bool) @ 0x211590f4 in /usr/bin/clickhouse
2021-10-10 09:28:55 5. ./obj-x86_64-linux-gnu/../src/Interpreters/InterpreterDropQuery.cpp:372: DB::InterpreterDropQuery::executeToDatabaseImpl(DB::ASTDropQuery const&, std::__1::shared_ptr<DB::IDatabase>&, std::__1::vector<StrongTypedef<wide::integer<128ul, unsigned int>, DB::UUIDTag>, std::__1::allocator<StrongTypedef<wide::integer<128ul, unsigned int>, DB::UUIDTag> > >&) @ 0x216348ee in /usr/bin/clickhouse
2021-10-10 09:28:55 6. ./obj-x86_64-linux-gnu/../src/Interpreters/InterpreterDropQuery.cpp:280: DB::InterpreterDropQuery::executeToDatabase(DB::ASTDropQuery const&) @ 0x21630cb9 in /usr/bin/clickhouse
2021-10-10 09:28:55 7. ./obj-x86_64-linux-gnu/../src/Interpreters/InterpreterDropQuery.cpp:64: DB::InterpreterDropQuery::execute() @ 0x216306fa in /usr/bin/clickhouse
2021-10-10 09:28:55 8. ./obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:635: DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x21cf057b in /usr/bin/clickhouse
2021-10-10 09:28:55 9. ./obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:950: DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x21cee144 in /usr/bin/clickhouse
2021-10-10 09:28:55 10. ./obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:302: DB::TCPHandler::runImpl() @ 0x228f8627 in /usr/bin/clickhouse
2021-10-10 09:28:55 11. ./obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1653: DB::TCPHandler::run() @ 0x22905f65 in /usr/bin/clickhouse
2021-10-10 09:28:55 12. ./obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x26d4a0f9 in /usr/bin/clickhouse
2021-10-10 09:28:55 13. ./obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x26d4a908 in /usr/bin/clickhouse
2021-10-10 09:28:55 14. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/ThreadPool.cpp:199: Poco::PooledThread::run() @ 0x26e98834 in /usr/bin/clickhouse
2021-10-10 09:28:55 15. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread.cpp:56: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x26e9531a in /usr/bin/clickhouse
2021-10-10 09:28:55 16. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread_POSIX.cpp:345: Poco::ThreadImpl::runnableEntry(void*) @ 0x26e940fc in /usr/bin/clickhouse
2021-10-10 09:28:55 17. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
2021-10-10 09:28:55 18. __clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
2021-10-10 09:28:55
2021-10-10 09:28:55   File "/usr/bin/clickhouse-test", line 638, in run
2021-10-10 09:28:55     proc, stdout, stderr, total_time = self.run_single_test(server_logs_level, client_options)
2021-10-10 09:28:55
2021-10-10 09:28:55   File "/usr/bin/clickhouse-test", line 595, in run_single_test
2021-10-10 09:28:55     client.execute("DROP DATABASE " + database)
2021-10-10 09:28:55
2021-10-10 09:28:55   File "/usr/local/lib/python3.8/dist-packages/clickhouse_driver/client.py", line 286, in execute
2021-10-10 09:28:55     rv = self.process_ordinary_query(
2021-10-10 09:28:55
2021-10-10 09:28:55   File "/usr/local/lib/python3.8/dist-packages/clickhouse_driver/client.py", line 479, in process_ordinary_query
2021-10-10 09:28:55     return self.receive_result(with_column_types=with_column_types,
2021-10-10 09:28:55
2021-10-10 09:28:55   File "/usr/local/lib/python3.8/dist-packages/clickhouse_driver/client.py", line 136, in receive_result
2021-10-10 09:28:55     return result.get_result()
2021-10-10 09:28:55
2021-10-10 09:28:55   File "/usr/local/lib/python3.8/dist-packages/clickhouse_driver/result.py", line 50, in get_result
2021-10-10 09:28:55     for packet in self.packet_generator:
2021-10-10 09:28:55
2021-10-10 09:28:55   File "/usr/local/lib/python3.8/dist-packages/clickhouse_driver/client.py", line 152, in packet_generator
2021-10-10 09:28:55     packet = self.receive_packet()
2021-10-10 09:28:55
2021-10-10 09:28:55   File "/usr/local/lib/python3.8/dist-packages/clickhouse_driver/client.py", line 169, in receive_packet
2021-10-10 09:28:55     raise packet.exception

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

Follow-up for: #29856

@azat
clickhouse-test: Retry DROP DATABASE in case of concurrent DROP/CREAT…
6b5dd53 
…E TABLE

CI [1]:

    2021-10-10 09:28:55 01103_optimize_drop_race_zookeeper:                                     [ UNKNOWN ] 0.00 sec. - Test internal error: ServerException
    2021-10-10 09:28:55 Code: 219.
    2021-10-10 09:28:55 DB::Exception: New table appeared in database being dropped or detached. Try again.. Stack trace:

<details>

    2021-10-10 09:28:55
    2021-10-10 09:28:55 0. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/exception:133: std::exception::capture() @ 0x14e44ba2 in /usr/bin/clickhouse
    2021-10-10 09:28:55 1. ./obj-x86_64-linux-gnu/../contrib/libcxx/include/exception:111: std::exception::exception() @ 0x14e44b70 in /usr/bin/clickhouse
    2021-10-10 09:28:55 2. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Exception.cpp:27: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x26e107c0 in /usr/bin/clickhouse
    2021-10-10 09:28:55 3. ./obj-x86_64-linux-gnu/../src/Common/Exception.cpp:57: DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0x14e2662e in /usr/bin/clickhouse
    2021-10-10 09:28:55 4. ./obj-x86_64-linux-gnu/../src/Interpreters/DatabaseCatalog.cpp:351: DB::DatabaseCatalog::detachDatabase(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, bool) @ 0x211590f4 in /usr/bin/clickhouse
    2021-10-10 09:28:55 5. ./obj-x86_64-linux-gnu/../src/Interpreters/InterpreterDropQuery.cpp:372: DB::InterpreterDropQuery::executeToDatabaseImpl(DB::ASTDropQuery const&, std::__1::shared_ptr<DB::IDatabase>&, std::__1::vector<StrongTypedef<wide::integer<128ul, unsigned int>, DB::UUIDTag>, std::__1::allocator<StrongTypedef<wide::integer<128ul, unsigned int>, DB::UUIDTag> > >&) @ 0x216348ee in /usr/bin/clickhouse
    2021-10-10 09:28:55 6. ./obj-x86_64-linux-gnu/../src/Interpreters/InterpreterDropQuery.cpp:280: DB::InterpreterDropQuery::executeToDatabase(DB::ASTDropQuery const&) @ 0x21630cb9 in /usr/bin/clickhouse
    2021-10-10 09:28:55 7. ./obj-x86_64-linux-gnu/../src/Interpreters/InterpreterDropQuery.cpp:64: DB::InterpreterDropQuery::execute() @ 0x216306fa in /usr/bin/clickhouse
    2021-10-10 09:28:55 8. ./obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:635: DB::executeQueryImpl(char const*, char const*, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum, DB::ReadBuffer*) @ 0x21cf057b in /usr/bin/clickhouse
    2021-10-10 09:28:55 9. ./obj-x86_64-linux-gnu/../src/Interpreters/executeQuery.cpp:950: DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::shared_ptr<DB::Context>, bool, DB::QueryProcessingStage::Enum) @ 0x21cee144 in /usr/bin/clickhouse
    2021-10-10 09:28:55 10. ./obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:302: DB::TCPHandler::runImpl() @ 0x228f8627 in /usr/bin/clickhouse
    2021-10-10 09:28:55 11. ./obj-x86_64-linux-gnu/../src/Server/TCPHandler.cpp:1653: DB::TCPHandler::run() @ 0x22905f65 in /usr/bin/clickhouse
    2021-10-10 09:28:55 12. ./obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerConnection.cpp:43: Poco::Net::TCPServerConnection::start() @ 0x26d4a0f9 in /usr/bin/clickhouse
    2021-10-10 09:28:55 13. ./obj-x86_64-linux-gnu/../contrib/poco/Net/src/TCPServerDispatcher.cpp:115: Poco::Net::TCPServerDispatcher::run() @ 0x26d4a908 in /usr/bin/clickhouse
    2021-10-10 09:28:55 14. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/ThreadPool.cpp:199: Poco::PooledThread::run() @ 0x26e98834 in /usr/bin/clickhouse
    2021-10-10 09:28:55 15. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread.cpp:56: Poco::(anonymous namespace)::RunnableHolder::run() @ 0x26e9531a in /usr/bin/clickhouse
    2021-10-10 09:28:55 16. ./obj-x86_64-linux-gnu/../contrib/poco/Foundation/src/Thread_POSIX.cpp:345: Poco::ThreadImpl::runnableEntry(void*) @ 0x26e940fc in /usr/bin/clickhouse
    2021-10-10 09:28:55 17. start_thread @ 0x9609 in /usr/lib/x86_64-linux-gnu/libpthread-2.31.so
    2021-10-10 09:28:55 18. __clone @ 0x122293 in /usr/lib/x86_64-linux-gnu/libc-2.31.so
    2021-10-10 09:28:55
    2021-10-10 09:28:55   File "/usr/bin/clickhouse-test", line 638, in run
    2021-10-10 09:28:55     proc, stdout, stderr, total_time = self.run_single_test(server_logs_level, client_options)
    2021-10-10 09:28:55
    2021-10-10 09:28:55   File "/usr/bin/clickhouse-test", line 595, in run_single_test
    2021-10-10 09:28:55     client.execute("DROP DATABASE " + database)
    2021-10-10 09:28:55
    2021-10-10 09:28:55   File "/usr/local/lib/python3.8/dist-packages/clickhouse_driver/client.py", line 286, in execute
    2021-10-10 09:28:55     rv = self.process_ordinary_query(
    2021-10-10 09:28:55
    2021-10-10 09:28:55   File "/usr/local/lib/python3.8/dist-packages/clickhouse_driver/client.py", line 479, in process_ordinary_query
    2021-10-10 09:28:55     return self.receive_result(with_column_types=with_column_types,
    2021-10-10 09:28:55
    2021-10-10 09:28:55   File "/usr/local/lib/python3.8/dist-packages/clickhouse_driver/client.py", line 136, in receive_result
    2021-10-10 09:28:55     return result.get_result()
    2021-10-10 09:28:55
    2021-10-10 09:28:55   File "/usr/local/lib/python3.8/dist-packages/clickhouse_driver/result.py", line 50, in get_result
    2021-10-10 09:28:55     for packet in self.packet_generator:
    2021-10-10 09:28:55
    2021-10-10 09:28:55   File "/usr/local/lib/python3.8/dist-packages/clickhouse_driver/client.py", line 152, in packet_generator
    2021-10-10 09:28:55     packet = self.receive_packet()
    2021-10-10 09:28:55
    2021-10-10 09:28:55   File "/usr/local/lib/python3.8/dist-packages/clickhouse_driver/client.py", line 169, in receive_packet
    2021-10-10 09:28:55     raise packet.exception

</details>

  [1]: https://clickhouse-test-reports.s3.yandex.net/0/2c3b866ab16b9b6ab315928aba1b2b3b19c60789/functional_stateless_tests_(debug).html#fail1
@robot-clickhouse robot-clickhouse added the pr-not-for-changelog This PR should not be mentioned in the changelog label Oct 10, 2021
@azat
Copy link
Copy Markdown
Member Author

azat commented Oct 10, 2021 •
edited
Loading

Functional stateless tests (thread) — Timeout, fail: 0, passed: 918, skipped: 3, unknown: 8

2021.10.10 05:12:06.256927 [ 381 ] {} <Fatal> Application: Child process was terminated by signal 9 (KILL). If it is not done by 'forcestop' command or manually, the possible cause is OOM Killer (see 'dmesg' and look at the '/var/log/kern.log' for the details).

Integration tests (asan) — fail: 3, passed: 1586, flaky: 3
Integration tests (release) — fail: 3, passed: 1587, flaky: 4
Integration tests (thread) — fail: 5, passed: 1584, flaky: 4

@tavplubix tavplubix self-assigned this Oct 10, 2021
@azat azat mentioned this pull request Oct 10, 2021
Merged
@tavplubix
Copy link
Copy Markdown
Member

tavplubix commented Oct 11, 2021

What about other cases when test should be retried?

@azat azat marked this pull request as draft October 11, 2021 21:29
@azat
Copy link
Copy Markdown
Member Author

azat commented Oct 11, 2021

Will be rewritten to python.requests and HTTP interface.

@azat azat closed this Oct 11, 2021
@azat azat deleted the tests-retry-DROP-DATABASE branch July 9, 2022 05:47
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alexey-milovidov alexey-milovidov alexey-milovidov approved these changes

Assignees

@tavplubix tavplubix

Labels

pr-not-for-changelog This PR should not be mentioned in the changelog

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@azat @tavplubix @alexey-milovidov @robot-clickhouse