Skip to content

The specified key does not exist (S3_ERROR) on server startup in functional tests with Replicated database #37791

@tavplubix

Description

@tavplubix

Sometimes server fails to start because of the following error:

2022.06.01 20:57:07.656220 [ 480 ] {} <Information> DiskS3: Starting up disk s3_cache
2022.06.01 20:57:07.656230 [ 480 ] {} <Information> DiskS3: Disk s3_cache started up
2022.06.01 20:57:07.656412 [ 480 ] {} <Information> Disk(s3_cache_2): Disk registered with cache path: /var/lib/clickhouse1/disks/s3_cache_2/data_cache/. Cache size: 22548578304, max cache elements size: 1048576, max_file_segment_size: 104857600
2022.06.01 20:57:07.656483 [ 480 ] {} <Information> AWSClient: Aws::Config::AWSConfigFileProfileConfigLoader: Initializing config loader against fileName /root/.aws/credentials and using profilePrefix = 0
2022.06.01 20:57:07.656490 [ 480 ] {} <Information> AWSClient: ProfileConfigFileAWSCredentialsProvider: Setting provider to read credentials from /root/.aws/credentials for credentials file and /root/.aws/config for the config file , for use with profile default
2022.06.01 20:57:07.656555 [ 480 ] {} <Trace> WriteBufferFromS3: Making single part upload. Bucket: test, Key: 00170_test/ceruvazksouixzeherhiuqazepedihzw, Size: 4, WithPool: true
2022.06.01 20:57:07.657724 [ 1148 ] {} <Warning> AWSClient: AWSErrorMarshaller: Encountered AWSError 'NoSuchKey': The specified key does not exist.
2022.06.01 20:57:07.657770 [ 1148 ] {} <Error> AWSClient: HTTP response code: 404
Resolved remote host IP address: 
Request ID: 16F4A47ECC949983
Exception name: NoSuchKey
Error message: The specified key does not exist.
12 response headers:
accept-ranges : bytes
connection : close
content-length : 373
content-security-policy : block-all-mixed-content
content-type : application/xml
date : Wed, 01 Jun 2022 23:57:07 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-request-id : 16F4A47ECC949983
x-content-type-options : nosniff
x-xss-protection : 1; mode=block
2022.06.01 20:57:07.657856 [ 1148 ] {} <Warning> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2022.06.01 20:57:07.658223 [ 480 ] {} <Error> virtual DB::WriteIndirectBufferFromRemoteFS::~WriteIndirectBufferFromRemoteFS(): Code: 499. DB::Exception: The specified key does not exist. (S3_ERROR), Stack trace (when copying this message, always include the lines below):

0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb6f337a in /usr/bin/clickhouse
1. DB::WriteBufferFromS3::processPutRequest(DB::WriteBufferFromS3::PutObjectTask&) @ 0x15c984df in /usr/bin/clickhouse
2. ? @ 0x15c9a121 in /usr/bin/clickhouse
3. ? @ 0x168cb080 in /usr/bin/clickhouse
4. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0xb7a72ba in /usr/bin/clickhouse
5. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0xb7a92a4 in /usr/bin/clickhouse
6. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xb7a46e7 in /usr/bin/clickhouse
7. ? @ 0xb7a811d in /usr/bin/clickhouse
8. ? @ 0x7fc8d7345609 in ?
9. __clone @ 0x7fc8d726a163 in ?
 (version 22.6.1.1015)
2022.06.01 20:57:07.658682 [ 480 ] {} <Information> Application: Shutting down storages.

https://s3.amazonaws.com/clickhouse-test-reports/37740/002f95741a1363373ee91e3dca28aff3c83c8794/stateless_tests__release__databasereplicated__actions__[1/2].html

cc: @kssenii

Metadata

Metadata

Assignees

Labels

comp-object-storageObject storage connectivity (S3/GCS/Azure) including credentials, retries, multipart, etc.testingSpecial issue with list of bugs found by CI

Type

No type
No fields configured for issues without a type.

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions