-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Failed test test_backup_restore_on_cluster/test_concurrency.py #45437
Copy link
Copy link
Closed
Labels
testingSpecial issue with list of bugs found by CISpecial issue with list of bugs found by CI
Description
Drop took too long(or was blocked by something)
> node0.query("DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster' NO DELAY")
test_backup_restore_on_cluster/test_concurrency.py:65:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
helpers/cluster.py:3231: in query
return self.client.query(
helpers/client.py:36: in wrap
return func(self, *args, **kwargs)
helpers/client.py:63: in query
return self.get_query_request(
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <helpers.client.CommandRequest object at 0x7ff84c42ce50>
def get_answer(self):
self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT)
self.stdout_file.seek(0)
self.stderr_file.seek(0)
stdout = self.stdout_file.read().decode("utf-8", errors="replace")
stderr = self.stderr_file.read().decode("utf-8", errors="replace")
if (
self.timer is not None
and not self.process_finished_before_timeout
and not self.ignore_error
):
logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}")
raise QueryTimeoutExceedException("Client timed out!")
if (self.process.returncode != 0 or stderr) and not self.ignore_error:
> raise QueryRuntimeException(
"Client failed! Return code: {}, stderr: {}".format(
self.process.returncode, stderr
),
self.process.returncode,
stderr,
)
E helpers.client.QueryRuntimeException: Client failed! Return code: 159, stderr: Received exception from server (version 22.13.1):
E Code: 159. DB::Exception: Received from 172.16.6.11:9000. DB::Exception: Watching task /clickhouse/task_queue/ddl/query-0000000212 is executing longer than distributed_ddl_task_timeout (=180) seconds. There are 3 unfinished hosts (3 of them are currently active), they are going to execute the query in background. Stack trace:
E
E 0. ./build_docker/../contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>> const&, int) @ 0x3d9b6bec in /usr/bin/clickhouse
E 1. ./build_docker/../src/Common/Exception.cpp:77: DB::Exception::Exception(DB::Exception::MessageMasked const&, int, bool) @ 0x202232e3 in /usr/bin/clickhouse
E 2. ./build_docker/../contrib/llvm-project/libcxx/include/string:1499: std::__1::__unique_if<DB::Exception>::__unique_single std::__1::make_unique[abi:v15000]<DB::Exception, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>, int const&>(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char>>&&, int const&) @ 0x3360f299 in /usr/bin/clickhouse
E 3. ./build_docker/../contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: DB::DDLQueryStatusSource::generate() @ 0x33605558 in /usr/bin/clickhouse
E 4. ./build_docker/../src/Processors/Chunk.h:90: DB::ISource::tryGenerate() @ 0x35f39bb9 in /usr/bin/clickhouse
E 5. ./build_docker/../src/Processors/ISource.cpp:0: DB::ISource::work() @ 0x35f38fc5 in /usr/bin/clickhouse
E 6. ./build_docker/../contrib/llvm-project/libcxx/include/list:588: DB::ExecutionThreadContext::executeTask() @ 0x35f855c1 in /usr/bin/clickhouse
E 7. ./build_docker/../src/Processors/Executors/PipelineExecutor.cpp:229: DB::PipelineExecutor::executeStepImpl(unsigned long, std::__1::atomic<bool>*) @ 0x35f645fd in /usr/bin/clickhouse
E 8. ./build_docker/../contrib/llvm-project/libcxx/include/__memory/shared_ptr.h:701: DB::PipelineExecutor::executeImpl(unsigned long) @ 0x35f62f66 in /usr/bin/clickhouse
E 9. ./build_docker/../contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:274: DB::PipelineExecutor::execute(unsigned long) @ 0x35f62b55 in /usr/bin/clickhouse
E 10. ./build_docker/../src/Processors/Executors/PullingAsyncPipelineExecutor.cpp:95: void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0>(DB::PullingAsyncPipelineExecutor::pull(DB::Chunk&, unsigned long)::$_0&&)::'lambda'(), void ()>>(std::__1::__function::__policy_storage const*) @ 0x35f8f184 in /usr/bin/clickhouse
E 11. ./build_docker/../base/base/wide_integer_impl.h:786: ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0x2047dcad in /usr/bin/clickhouse
E 12. ./build_docker/../contrib/llvm-project/libcxx/include/__memory/unique_ptr.h:302: void* std::__1::__thread_proxy[abi:v15000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct>>, void ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::'lambda0'()>>(void*) @ 0x20487ec0 in /usr/bin/clickhouse
E 13. ? @ 0x7f0fd9e32609 in ?
E 14. clone @ 0x7f0fd9d57133 in ?
E . (TIMEOUT_EXCEEDED)
E (query: DROP DATABASE IF EXISTS mydb ON CLUSTER 'cluster' NO DELAY)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
testingSpecial issue with list of bugs found by CISpecial issue with list of bugs found by CI