Skip to content

Using INSERT INTO FUNCTION s3 fails with a timeout #40128

Closed
Closed
Using INSERT INTO FUNCTION s3 fails with a timeout#40128
Labels
backward compatibilitycomp-object-storageObject storage connectivity (S3/GCS/Azure) including credentials, retries, multipart, etc.
@lesandie

Description

@lesandie
lesandie
opened on Aug 11, 2022

Describe the issue
Inserting into s3 compatible storage providers like min.io or backblaze fails with a timeout. AWS s3 provider always works.
Used different combinations of s3_min_upload_part_size, s3_upload_part_size_multiply_factor,
s3_upload_part_size_multiply_parts_count_threshold, s3_max_connections and max_insert_threads
without sucess

How to reproduce
v21.8.15 -> working all (backblaze, aws, min.io)
v22.3.8 -> working only AWS
v22.6.5 -> working only AWS

DML

INSERT INTO FUNCTION s3
(
    'https://s3.eu-central-003.backblazeb2.com/test-clickhouse/test_file.parquet.gz',
    'XXXXXXXXXXXXXXXXXXXXX','XXXXXXXXXXXXXXXXXXXXXXXXx','Parquet',
    'number UInt64',
    'gz'
)
SELECT * FROM numbers(1000000000) 
SETTINGS
    s3_max_connections=1,
    max_insert_threads=1,
    s3_truncate_on_insert=1,
    s3_min_upload_part_size=33554432;

Error message and/or stacktrace
Failing due to timeout v21.11:

2022.08.11 12:44:50.321368 [ 229 ] {ec42374f-c8a4-465a-bbcc-50b1deebf269} <Error> AWSClient: Failed to make request to: https://s3.eu-central-003.backblazeb2.com/pepper-clickhouse/test_file.parquet.gz?partNumber=1&uploadId=4_zd04725d6388f6c4b8b25051d_f2009e3cd50e9fce7_d20220811_m124239_c003_v0312017_t0017_u01660221759830: Poco::Exception. Code: 1000, e.code() = 0, Timeout, Stack trace (when copying this message, always include the lines below):

0. Poco::Net::SecureSocketImpl::mustRetry(int, Poco::Timespan&) @ 0x15d7bc36 in /usr/bin/clickhouse
1. Poco::Net::SecureSocketImpl::receiveBytes(void*, int, int) @ 0x15d7d612 in /usr/bin/clickhouse
2. Poco::Net::HTTPSession::refill() @ 0x15da581d in /usr/bin/clickhouse
3. Poco::Net::HTTPHeaderStreamBuf::readFromDevice(char*, long) @ 0x15d9ed0e in /usr/bin/clickhouse
4. Poco::BasicBufferedStreamBuf<char, std::__1::char_traits<char>, Poco::Net::HTTPBufferAllocator>::underflow() @ 0x15d94a05 in /usr/bin/clickhouse
5. std::__1::basic_streambuf<char, std::__1::char_traits<char> >::uflow() @ 0x17fa11aa in ?
6. std::__1::basic_istream<char, std::__1::char_traits<char> >::get() @ 0x17fa390f in ?
7. Poco::Net::HTTPResponse::read(std::__1::basic_istream<char, std::__1::char_traits<char> >&) @ 0x15da36af in /usr/bin/clickhouse
8. Poco::Net::HTTPClientSession::receiveResponse(Poco::Net::HTTPResponse&) @ 0x15d9863a in /usr/bin/clickhouse
9. DB::S3::PocoHTTPClient::makeRequestInternal(Aws::Http::HttpRequest&, std::__1::shared_ptr<DB::S3::PocoHTTPResponse>&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const @ 0x11705d96 in /usr/bin/clickhouse
10. DB::S3::PocoHTTPClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const @ 0x117046c2 in /usr/bin/clickhouse
11. Aws::Client::AWSClient::AttemptOneRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const*, char const*, char const*) const @ 0x1584faaa in /usr/bin/clickhouse
12. Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const @ 0x1584cd11 in /usr/bin/clickhouse
13. Aws::Client::AWSXMLClient::MakeRequest(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const @ 0x15859e6c in /usr/bin/clickhouse
14. Aws::S3::S3Client::UploadPart(Aws::S3::Model::UploadPartRequest const&) const @ 0x15969cad in /usr/bin/clickhouse
15. DB::WriteBufferFromS3::writePart() @ 0x11e60b39 in /usr/bin/clickhouse
16. DB::WriteBufferFromS3::nextImpl() @ 0x11e5f57e in /usr/bin/clickhouse
17. DB::ZlibDeflatingWriteBuffer::nextImpl() @ 0x100e5a38 in /usr/bin/clickhouse
18. DB::ArrowBufferedOutputStream::Write(void const*, long) @ 0x131f460d in /usr/bin/clickhouse
19. parquet::SerializedPageWriter::WriteDataPage(parquet::DataPage const&) @ 0x161d9d89 in /usr/bin/clickhouse
20. parquet::ColumnWriterImpl::BuildDataPageV1(long, long, long, std::__1::shared_ptr<arrow::Buffer> const&) @ 0x161cda29 in /usr/bin/clickhouse
21. parquet::ColumnWriterImpl::AddDataPage() @ 0x161cd573 in /usr/bin/clickhouse
22. parquet::TypedColumnWriterImpl<parquet::PhysicalType<(parquet::Type::type)2> >::WriteBatch(long, short const*, short const*, long const*)::'lambda'(long, long)::operator()(long, long) const @ 0x161ebe5d in /usr/bin/clickhouse
23. non-virtual thunk to parquet::TypedColumnWriterImpl<parquet::PhysicalType<(parquet::Type::type)2> >::WriteBatch(long, short const*, short const*, long const*) @ 0x161eba5e in /usr/bin/clickhouse
24. arrow::Status parquet::WriteArrowSerialize<parquet::PhysicalType<(parquet::Type::type)2>, arrow::UInt64Type>(arrow::Array const&, long, short const*, short const*, parquet::ArrowWriteContext*, parquet::TypedColumnWriter<parquet::PhysicalType<(parquet::Type::type)2> >*, bool) @ 0x161d4e88 in /usr/bin/clickhouse
25. parquet::TypedColumnWriterImpl<parquet::PhysicalType<(parquet::Type::type)2> >::WriteArrowDense(short const*, short const*, long, arrow::Array const&, parquet::ArrowWriteContext*, bool) @ 0x161d452e in /usr/bin/clickhouse
26. parquet::TypedColumnWriterImpl<parquet::PhysicalType<(parquet::Type::type)2> >::WriteArrow(short const*, short const*, long, arrow::Array const&, parquet::ArrowWriteContext*, bool) @ 0x161eb45d in /usr/bin/clickhouse
27. non-virtual thunk to parquet::TypedColumnWriterImpl<parquet::PhysicalType<(parquet::Type::type)2> >::WriteArrow(short const*, short const*, long, arrow::Array const&, parquet::ArrowWriteContext*, bool) @ 0x161eb97a in /usr/bin/clickhouse
28. ? @ 0x1611be13 in /usr/bin/clickhouse
29. parquet::arrow::MultipathLevelBuilderImpl::Write(int, parquet::ArrowWriteContext*, std::__1::function<arrow::Status (parquet::arrow::MultipathLevelBuilderResult const&)>) @ 0x16124568 in /usr/bin/clickhouse
30. parquet::arrow::FileWriterImpl::WriteColumnChunk(std::__1::shared_ptr<arrow::ChunkedArray> const&, long, long) @ 0x1611a3db in /usr/bin/clickhouse
31. parquet::arrow::FileWriterImpl::WriteTable(arrow::Table const&, long) @ 0x16118f02 in /usr/bin/clickhouse
 (version 21.11.11.1 (official build))
2022.08.11 12:44:50.321520 [ 229 ] {ec42374f-c8a4-465a-bbcc-50b1deebf269} <Error> AWSClient: HTTP response code: -1
Resolved remote host IP address: 
Request ID: 
Exception name: 
Error message: Poco::Exception. Code: 1000, e.code() = 0, Timeout (version 21.11.11.1 (official build))
0 response headers:

Failing due to timeout v22+ error:

2022.08.10 14:19:39.641248 [ 593151 ] {f056dc67-a022-4b0e-b55e-abc1526fb1a9} <Error> AWSClient: HTTP response code: -1
Resolved remote host IP address: 
Request ID: 
Exception name: 
Error message: Poco::Exception. Code: 1000, e.code() = 0, Timeout (version 22.7.2.15 (official build))
0 response headers:

2022.08.10 14:19:44.658242 [ 593151 ] {f056dc67-a022-4b0e-b55e-abc1526fb1a9} <Error> AWSClient: Failed to make request to: https://s3.eu-central-003.backblazeb2.com/pepper-clickhouse/test_file.parquet.gz?partNumber=2&uploadId=4_zd04725d6388f6c4b8b25051d_f2054ddde695f5f99_d20220810_m121933_c003_v0312017_t0002_u01660133973425: Poco::Exception. Code: 1000, e.code() = 0, Timeout, Stack trace (when copying this message, always include the lines below):
0. Poco::Net::SecureSocketImpl::mustRetry(int, Poco::Timespan&) @ 0x1a93887f in /usr/bin/clickhouse
1. Poco::Net::SecureSocketImpl::receiveBytes(void*, int, int) @ 0x1a9399e2 in /usr/bin/clickhouse
2. Poco::Net::HTTPSession::refill() @ 0x1a96163d in /usr/bin/clickhouse
3. Poco::Net::HTTPHeaderStreamBuf::readFromDevice(char*, long) @ 0x1a959f9e in /usr/bin/clickhouse
4. Poco::BasicBufferedStreamBuf<char, std::__1::char_traits<char>, Poco::Net::HTTPBufferAllocator>::underflow() @ 0x1a95022f in /usr/bin/clickhouse
5. std::__1::basic_streambuf<char, std::__1::char_traits<char> >::uflow() @ 0xba235aa in /usr/bin/clickhouse
6. std::__1::basic_istream<char, std::__1::char_traits<char> >::get() @ 0xba24bb9 in /usr/bin/clickhouse
7. Poco::Net::HTTPResponse::read(std::__1::basic_istream<char, std::__1::char_traits<char> >&) @ 0x1a95ee0f in /usr/bin/clickhouse
8. Poco::Net::HTTPClientSession::receiveResponse(Poco::Net::HTTPResponse&) @ 0x1a95412a in /usr/bin/clickhouse
9. DB::S3::PocoHTTPClient::makeRequestInternal(Aws::Http::HttpRequest&, std::__1::shared_ptr<DB::S3::PocoHTTPResponse>&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const @ 0x15928806 in /usr/bin/clickhouse
10. DB::S3::PocoHTTPClient::MakeRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::Utils::RateLimits::RateLimiterInterface*, Aws::Utils::RateLimits::RateLimiterInterface*) const @ 0x159265ef in /usr/bin/clickhouse
11. Aws::Client::AWSClient::AttemptOneRequest(std::__1::shared_ptr<Aws::Http::HttpRequest> const&, Aws::AmazonWebServiceRequest const&, char const*, char const*, char const*) const @ 0x1ab9a177 in /usr/bin/clickhouse
12. Aws::Client::AWSClient::AttemptExhaustively(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const @ 0x1ab9733e in /usr/bin/clickhouse
13. Aws::Client::AWSXMLClient::MakeRequest(Aws::Http::URI const&, Aws::AmazonWebServiceRequest const&, Aws::Http::HttpMethod, char const*, char const*, char const*) const @ 0x1aba4a2d in /usr/bin/clickhouse
14. Aws::S3::S3Client::UploadPart(Aws::S3::Model::UploadPartRequest const&) const @ 0x1acca21a in /usr/bin/clickhouse
15. DB::WriteBufferFromS3::processUploadRequest(DB::WriteBufferFromS3::UploadPartTask&) @ 0x16349df2 in /usr/bin/clickhouse
16. ? @ 0x1634ba40 in /usr/bin/clickhouse
17. ? @ 0x16eff400 in /usr/bin/clickhouse
18. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0xbb06e46 in /usr/bin/clickhouse
19. void std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&)::'lambda'(), void ()> >(std::__1::__function::__policy_storage const*) @ 0xbb08a55 in /usr/bin/clickhouse
20. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xbb046e8 in /usr/bin/clickhouse
21. ? @ 0xbb07a7d in /usr/bin/clickhouse
22. ? @ 0x7f3125bd7b43 in ?
23. ? @ 0x7f3125c69a00 in ?
 (version 22.7.2.15 (official build))

Additional context
Probably related with:

Metadata

Metadata

Assignees

No one assigned

    Labels

    backward compatibilitycomp-object-storageObject storage connectivity (S3/GCS/Azure) including credentials, retries, multipart, etc.

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions