2022.06.01 20:57:07.656220 [ 480 ] {} <Information> DiskS3: Starting up disk s3_cache
2022.06.01 20:57:07.656230 [ 480 ] {} <Information> DiskS3: Disk s3_cache started up
2022.06.01 20:57:07.656412 [ 480 ] {} <Information> Disk(s3_cache_2): Disk registered with cache path: /var/lib/clickhouse1/disks/s3_cache_2/data_cache/. Cache size: 22548578304, max cache elements size: 1048576, max_file_segment_size: 104857600
2022.06.01 20:57:07.656483 [ 480 ] {} <Information> AWSClient: Aws::Config::AWSConfigFileProfileConfigLoader: Initializing config loader against fileName /root/.aws/credentials and using profilePrefix = 0
2022.06.01 20:57:07.656490 [ 480 ] {} <Information> AWSClient: ProfileConfigFileAWSCredentialsProvider: Setting provider to read credentials from /root/.aws/credentials for credentials file and /root/.aws/config for the config file , for use with profile default
2022.06.01 20:57:07.656555 [ 480 ] {} <Trace> WriteBufferFromS3: Making single part upload. Bucket: test, Key: 00170_test/ceruvazksouixzeherhiuqazepedihzw, Size: 4, WithPool: true
2022.06.01 20:57:07.657724 [ 1148 ] {} <Warning> AWSClient: AWSErrorMarshaller: Encountered AWSError 'NoSuchKey': The specified key does not exist.
2022.06.01 20:57:07.657770 [ 1148 ] {} <Error> AWSClient: HTTP response code: 404
Resolved remote host IP address:
Request ID: 16F4A47ECC949983
Exception name: NoSuchKey
Error message: The specified key does not exist.
12 response headers:
accept-ranges : bytes
connection : close
content-length : 373
content-security-policy : block-all-mixed-content
content-type : application/xml
date : Wed, 01 Jun 2022 23:57:07 GMT
server : MinIO
strict-transport-security : max-age=31536000; includeSubDomains
vary : Accept-Encoding
x-amz-request-id : 16F4A47ECC949983
x-content-type-options : nosniff
x-xss-protection : 1; mode=block
2022.06.01 20:57:07.657856 [ 1148 ] {} <Warning> AWSClient: If the signature check failed. This could be because of a time skew. Attempting to adjust the signer.
2022.06.01 20:57:07.658223 [ 480 ] {} <Error> virtual DB::WriteIndirectBufferFromRemoteFS::~WriteIndirectBufferFromRemoteFS(): Code: 499. DB::Exception: The specified key does not exist. (S3_ERROR), Stack trace (when copying this message, always include the lines below):
0. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int, bool) @ 0xb6f337a in /usr/bin/clickhouse
1. DB::WriteBufferFromS3::processPutRequest(DB::WriteBufferFromS3::PutObjectTask&) @ 0x15c984df in /usr/bin/clickhouse
2. ? @ 0x15c9a121 in /usr/bin/clickhouse
3. ? @ 0x168cb080 in /usr/bin/clickhouse
4. ThreadPoolImpl<ThreadFromGlobalPool>::worker(std::__1::__list_iterator<ThreadFromGlobalPool, void*>) @ 0xb7a72ba in /usr/bin/clickhouse
5. ThreadFromGlobalPool::ThreadFromGlobalPool<void ThreadPoolImpl<ThreadFromGlobalPool>::scheduleImpl<void>(std::__1::function<void ()>, int, std::__1::optional<unsigned long>)::'lambda0'()>(void&&)::'lambda'()::operator()() @ 0xb7a92a4 in /usr/bin/clickhouse
6. ThreadPoolImpl<std::__1::thread>::worker(std::__1::__list_iterator<std::__1::thread, void*>) @ 0xb7a46e7 in /usr/bin/clickhouse
7. ? @ 0xb7a811d in /usr/bin/clickhouse
8. ? @ 0x7fc8d7345609 in ?
9. __clone @ 0x7fc8d726a163 in ?
(version 22.6.1.1015)
2022.06.01 20:57:07.658682 [ 480 ] {} <Information> Application: Shutting down storages.
Sometimes server fails to start because of the following error:
https://s3.amazonaws.com/clickhouse-test-reports/37740/002f95741a1363373ee91e3dca28aff3c83c8794/stateless_tests__release__databasereplicated__actions__[1/2].html
cc: @kssenii