Skip to content

test_distributed_queries_stress is flaky #41776

@tavplubix

Description

@tavplubix

https://s3.amazonaws.com/clickhouse-test-reports/41728/5852df7f6504fbae4a5df379d2d792f45d4067d2/integration_tests__asan__[3/3].html

>       run_benchmark(payload, settings)

test_distributed_queries_stress/test.py:121: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
test_distributed_queries_stress/test.py:18: in run_benchmark
    node1_r1.exec_in_container(
helpers/cluster.py:3408: in exec_in_container
    return self.cluster.exec_in_container(
helpers/cluster.py:1752: in exec_in_container
    result = subprocess_check_call(
helpers/cluster.py:151: in subprocess_check_call
    return run_and_check(args, detach=detach, nothrow=nothrow)
helpers/cluster.py:96: in run_and_check
    res = subprocess.run(
/usr/lib/python3.8/subprocess.py:495: in run
    stdout, stderr = process.communicate(input, timeout=timeout)
/usr/lib/python3.8/subprocess.py:1028: in communicate
    stdout, stderr = self._communicate(input, endtime, timeout)
/usr/lib/python3.8/subprocess.py:1869: in _communicate
    self._check_timeout(endtime, orig_timeout, stdout, stderr)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <subprocess.Popen object at 0x7f9298feb280>, endtime = 3483.629056787
orig_timeout = 300, stdout_seq = [], stderr_seq = [b'Loaded 5 queries.\n']
skip_check_and_raise = False

    def _check_timeout(self, endtime, orig_timeout, stdout_seq, stderr_seq,
                       skip_check_and_raise=False):
        """Convenience for checking if a timeout has expired."""
        if endtime is None:
            return
        if skip_check_and_raise or _time() > endtime:
>           raise TimeoutExpired(
                    self.args, orig_timeout,
                    output=b''.join(stdout_seq) if stdout_seq else None,
                    stderr=b''.join(stderr_seq) if stderr_seq else None)
E           subprocess.TimeoutExpired: Command '['docker', 'exec', 'roottestdistributedqueriesstress_node1_r1_1', 'bash', '-c', "echo 'select * from dist_two_over_dist where key = 0;\n    select * from dist_two_over_dist where key = 1;\n    select * from dist_two_over_dist where key = 2;\n    select * from dist_two_over_dist where key = 3;\n    select * from dist_two_over_dist;' | clickhouse benchmark --concurrency=100 --cumulative --delay=0 --timelimit=3 --hedged_connection_timeout_ms=200 --connect_timeout_with_failover_ms=200 --connections_with_failover_max_tries=5 --optimize_skip_unused_shards=1 --distributed_group_by_no_merge=2"]' timed out after 300 seconds

But I have no ideas why clickhouse-benchmark did not finish in 300 seconds when timelimit was set to 3 seconds

Metadata

Metadata

Assignees

No one assigned

    Labels

    testingSpecial issue with list of bugs found by CI

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions