Skip to content

test_s3_disk_reads_on_unstable_connection is flaky #88503

@Ergus

Description

@Ergus

Failure example from #85920 here

File: test_merge_tree_s3/test.py:781 - in test_s3_disk_reads_on_unstable_connection
    assert node.query("SELECT sum(id) FROM s3_test").splitlines() == [
File: helpers/cluster.py:4255 - in query
    return self.client.query(
File: helpers/client.py:39 - in wrap
    return func(self, *args, **kwargs)
File: helpers/client.py:79 - in query
    ).get_answer()
File: helpers/client.py:247 - in get_answer
    raise QueryRuntimeException(
E   helpers.client.QueryRuntimeException: Client failed! Return code: 232, stderr: Received exception from server (version 25.10.1):
E   Code: 1000. DB::Exception: Received from 172.16.1.6:9000. DB::Exception: Malformed message: Unexpected EOF. Stack trace:
E   
E   0. ./contrib/llvm-project/libcxx/include/__exception/exception.h:113: Poco::IOException::IOException(String const&, int) @ 0x000000001e2639f2
E   1. ./ci/tmp/build/./base/poco/Net/src/NetException.cpp:26: Poco::Net::MessageException::MessageException(String const&, int) @ 0x000000001e2fdec9
E   2. ./ci/tmp/build/./base/poco/Net/src/HTTPFixedLengthStream.cpp:63: Poco::Net::HTTPFixedLengthStreamBuf::readFromDevice(char*, long) @ 0x000000001e2e1e77
E   3. ./ci/tmp/build/./src/IO/ReadBufferFromIStream.cpp:27: DB::ReadBufferFromIStream::nextImpl() @ 0x00000000123c3410
E   4. ./ci/tmp/build/./src/IO/ReadBuffer.cpp:96: DB::ReadBuffer::next() @ 0x00000000101184ed
E   5. ./ci/tmp/build/./src/IO/ReadBufferFromS3.cpp:171: DB::ReadBufferFromS3::nextImpl() @ 0x0000000013ff9b55
E   6. ./ci/tmp/build/./src/IO/ReadBuffer.cpp:96: DB::ReadBuffer::next() @ 0x00000000101184ed
E   7. ./ci/tmp/build/./src/Disks/IO/ReadBufferFromRemoteFSGather.cpp:122: DB::ReadBufferFromRemoteFSGather::nextImpl() @ 0x0000000014e8d14e
E   8. ./ci/tmp/build/./src/IO/ReadBuffer.cpp:96: DB::ReadBuffer::next() @ 0x00000000101184ed
E   9. ./ci/tmp/build/./src/Disks/IO/ThreadPoolRemoteFSReader.cpp:157: DB::ThreadPoolRemoteFSReader::execute(DB::IAsynchronousReader::Request, bool) @ 0x0000000013e5c9a3
E   10. ./ci/tmp/build/./src/Disks/IO/ThreadPoolRemoteFSReader.cpp:114: std::__packaged_task_func<std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::'lambda'(), std::allocator<std::function<std::future<DB::IAsynchronousReader::Result> (DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)> DB::threadPoolCallbackRunnerUnsafe<DB::IAsynchronousReader::Result, DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0>(ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>&, String const&)::'lambda'(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::operator()(DB::ThreadPoolRemoteFSReader::submit(DB::IAsynchronousReader::Request)::$_0&&, Priority)::'lambda'()>, DB::IAsynchronousReader::Result ()>::operator()() @ 0x0000000013e5d873
E   11. ./contrib/llvm-project/libcxx/include/future:1589: std::packaged_task<DB::IAsynchronousReader::Result ()>::operator()() @ 0x0000000013e5b108
E   12. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001017d595
E   13. ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: ThreadFromGlobalPoolImpl<false, true>::ThreadFromGlobalPoolImpl<void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*>(void (ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool::*&&)(), ThreadPoolImpl<ThreadFromGlobalPoolImpl<false, true>>::ThreadFromThreadPool*&&)::'lambda'()::operator()() @ 0x0000000010182ce6
E   14. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x000000001017afe0
E   15. ./contrib/llvm-project/libcxx/include/__type_traits/invoke.h:117: void* std::__thread_proxy[abi:ne190107]<std::tuple<std::unique_ptr<std::__thread_struct, std::default_delete<std::__thread_struct>>, void (ThreadPoolImpl<std::thread>::ThreadFromThreadPool::*)(), ThreadPoolImpl<std::thread>::ThreadFromThreadPool*>>(void*) @ 0x0000000010180dfb
E   16. ? @ 0x0000000000094ac3
E   17. ? @ 0x0000000000126850
E   . (POCO_EXCEPTION)
E   (query: SELECT sum(id) FROM s3_test)
===== Captured log setup =====
2025-10-09 02:08:43.602000 [ 609792 ] INFO : list_objects (0): [] (test.py:142, list_objects)
===== Captured stdout call =====
Read sequence 0
Read sequence 1
Read sequence 2
Read sequence 3
Read sequence 4
Read sequence 5
Read sequence 6
Read sequence 7
Read sequence 8
Read sequence 9
Read sequence 10
Read sequence 11
Read sequence 12
Read sequence 13
Read sequence 14
Read sequence 15
Read sequence 16
Read sequence 17
Read sequence 18
Read sequence 19
Read sequence 20
Read sequence 21
Read sequence 22
Read sequence 23
Read sequence 24

Metadata

Metadata

Assignees

No one assigned

    Labels

    flaky testflaky test found by CI

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions