Ignore only not found errors for s3_plain_rewritable (and some other S3 code)#87426

Merged

azat merged 2 commits intoClickHouse:masterfrom

Sep 23, 2025

Member

azat commented Sep 22, 2025 •

edited

Loading

MergeTree code does call readFileIfExists() in some places. For s3_plain_rewritable, however, existsFile() looks unsafe because it ignores all errors from S3.

For example, you might get an error such as "Response status: 503, Slow Down" from S3. In that case, with s3_slow_all_threads_after_retryable_error, the number of retries is reduced to 1 (#85918), which makes it possible to get false from existsFile()

Example stacktrace:

DB::S3::(anonymous namespace)::tryGetObjectInfo()
DB::S3::getObjectInfo()
DB::S3ObjectStorage::tryGetObjectMetadata()
DB::MetadataStorageFromPlainObjectStorage::getObjectMetadataEntryWithCache()
DB::MetadataStorageFromPlainObjectStorage::getObjectMetadataEntryWithCache()
DB::MetadataStorageFromPlainRewritableObjectStorage::existsFile()

Changelog category (leave one):

Critical Bug Fix (crash, data loss, RBAC) or LOGICAL_ERROR

Changelog entry (a user-readable short description of the changes that goes into CHANGELOG.md):

Ignore only not found errors for s3_plain_rewritable (which may lead to all sort of troubles)

azat requested a review from jkartseva

September 22, 2025 19:34

Contributor

clickhouse-gh bot commented Sep 22, 2025 •

edited

Loading

Workflow [PR], commit [fe78530]

Summary: ❌

job_name	test_name	status	info	comment
Stateless tests (amd_binary, ParallelReplicas, s3 storage, parallel)		failure
	03252_check_number_of_arguments_for_dynamic	FAIL
	02122_parallel_formatting_PrettyNoEscapes	FAIL
	Exception in test runner	FAIL
	Killed by signal (in clickhouse-server.log or clickhouse-server.err.log)	FAIL
	Fatal messages (in clickhouse-server.log or clickhouse-server.err.log)	FAIL
Finish Workflow		failure
	python3 ./ci/jobs/scripts/workflow_hooks/new_tests_check.py	failure

clickhouse-gh bot added pr-critical-bugfix pr-must-backport pr-must-backport-cloud labels

azat force-pushed the s3-exists branch 2 times, most recently from 78d063e to 7bc0cce Compare

September 22, 2025 20:33


          Ignore only not found errors for s3_plain_rewritable (and some other …

cb58921

…S3 code)

MergeTree code does call readFileIfExists() in some places. For
s3_plain_rewritable, however, existsFile() looks unsafe because it
ignores **all** errors from S3.

For example, you might get an error such as "Response status: 503, Slow Down"
from S3. In that case, with s3_slow_all_threads_after_retryable_error,
the number of retries is reduced to 1, which makes it possible to get
false from existsFile()

Example stacktrace:

    DB::S3::(anonymous namespace)::tryGetObjectInfo()
    DB::S3::getObjectInfo()
    DB::S3ObjectStorage::tryGetObjectMetadata()
    DB::MetadataStorageFromPlainObjectStorage::getObjectMetadataEntryWithCache()
    DB::MetadataStorageFromPlainObjectStorage::getObjectMetadataEntryWithCache()
    DB::MetadataStorageFromPlainRewritableObjectStorage::existsFile()

azat force-pushed the s3-exists branch from 7bc0cce to cb58921 Compare

September 22, 2025 20:34

nikitamikhaylov approved these changes

View reviewed changes

nikitamikhaylov self-assigned this


          Remove unsafe S3 API (that may ignore errors)

fe78530

jkartseva approved these changes

View reviewed changes

azat enabled auto-merge

September 23, 2025 06:01

Member Author

azat commented Sep 23, 2025

CI failures are unrelated

azat added this pull request to the merge queue

Merged via the queue into ClickHouse:master with commit f51963d

120 of 123 checks passed

azat deleted the s3-exists branch

September 23, 2025 06:31

robot-ch-test-poll2 added pr-synced-to-cloud pr-backports-created-cloud pr-must-backport-synced labels

This was referenced Sep 23, 2025

Cherry pick #87426 to 25.3: Ignore only not found errors for s3_plain_rewritable (and some other S3 code) #87450

Merged

Cherry pick #87426 to 25.6: Ignore only not found errors for s3_plain_rewritable (and some other S3 code) #87451

Merged

robot-clickhouse added a commit that referenced this pull request


          Merge pull request #87451 from ClickHouse/cherrypick/25.6/87426

9d1c52d

Cherry pick #87426 to 25.6: Ignore only not found errors for s3_plain_rewritable (and some other S3 code)

robot-clickhouse added a commit that referenced this pull request


          Backport #87426 to 25.6: Ignore only not found errors for s3_plain_re…

c96725a

…writable (and some other S3 code)

This was referenced Sep 23, 2025

Backport #87426 to 25.6: Ignore only not found errors for s3_plain_rewritable (and some other S3 code) #87452

Merged

Cherry pick #87426 to 25.7: Ignore only not found errors for s3_plain_rewritable (and some other S3 code) #87453

Merged

robot-clickhouse added a commit that referenced this pull request


          Merge pull request #87453 from ClickHouse/cherrypick/25.7/87426

df8bad4

Cherry pick #87426 to 25.7: Ignore only not found errors for s3_plain_rewritable (and some other S3 code)

robot-clickhouse added a commit that referenced this pull request


          Backport #87426 to 25.7: Ignore only not found errors for s3_plain_re…

59049a0

…writable (and some other S3 code)

This was referenced Sep 23, 2025

Backport #87426 to 25.7: Ignore only not found errors for s3_plain_rewritable (and some other S3 code) #87454

Merged

Cherry pick #87426 to 25.8: Ignore only not found errors for s3_plain_rewritable (and some other S3 code) #87455

Merged

robot-clickhouse added a commit that referenced this pull request


          Merge pull request #87455 from ClickHouse/cherrypick/25.8/87426

a5c76d8

Cherry pick #87426 to 25.8: Ignore only not found errors for s3_plain_rewritable (and some other S3 code)

robot-clickhouse added a commit that referenced this pull request


          Backport #87426 to 25.8: Ignore only not found errors for s3_plain_re…

…writable (and some other S3 code)

This was referenced Sep 23, 2025

Backport #87426 to 25.8: Ignore only not found errors for s3_plain_rewritable (and some other S3 code) #87456

Merged

Cherry pick #87426 to 25.9: Ignore only not found errors for s3_plain_rewritable (and some other S3 code) #87457

Merged

robot-clickhouse added a commit that referenced this pull request


          Merge pull request #87457 from ClickHouse/cherrypick/25.9/87426

da85718

Cherry pick #87426 to 25.9: Ignore only not found errors for s3_plain_rewritable (and some other S3 code)

robot-clickhouse added a commit that referenced this pull request


          Backport #87426 to 25.9: Ignore only not found errors for s3_plain_re…

f1dc7d9

…writable (and some other S3 code)

robot-clickhouse mentioned this pull request

Backport #87426 to 25.9: Ignore only not found errors for s3_plain_rewritable (and some other S3 code) #87458

Merged

clickhouse-gh bot added a commit that referenced this pull request


          Merge pull request #87454 from ClickHouse/backport/25.7/87426

cc325d4

Backport #87426 to 25.7: Ignore only not found errors for s3_plain_rewritable (and some other S3 code)

clickhouse-gh bot added a commit that referenced this pull request


          Merge pull request #87452 from ClickHouse/backport/25.6/87426

a02f637

Backport #87426 to 25.6: Ignore only not found errors for s3_plain_rewritable (and some other S3 code)

azat added a commit that referenced this pull request


          Merge pull request #87456 from ClickHouse/backport/25.8/87426

08860a5

Backport #87426 to 25.8: Ignore only not found errors for s3_plain_rewritable (and some other S3 code)

azat added a commit that referenced this pull request


          Merge pull request #87458 from ClickHouse/backport/25.9/87426

ff18e1d

Backport #87426 to 25.9: Ignore only not found errors for s3_plain_rewritable (and some other S3 code)

robot-ch-test-poll added a commit that referenced this pull request


          Merge pull request #87450 from ClickHouse/cherrypick/25.3/87426

0cab7fa

Cherry pick #87426 to 25.3: Ignore only not found errors for s3_plain_rewritable (and some other S3 code)

robot-clickhouse added a commit that referenced this pull request


          Backport #87426 to 25.3: Ignore only not found errors for s3_plain_re…

f177ceb

…writable (and some other S3 code)

robot-ch-test-poll mentioned this pull request

Backport #87426 to 25.3: Ignore only not found errors for s3_plain_rewritable (and some other S3 code) #87471

Merged

azat added a commit that referenced this pull request


          Merge remote-tracking branch 'u/backport/25.9/87449' into cherrypick/…

cda044d

…25.9/87449

* u/backport/25.9/87449:
  Backport #87426 to 25.9: Ignore only not found errors for s3_plain_rewritable (and some other S3 code)
  Backport #87231 to 25.9: Fix "Too large size passed to allocator" UB in JOIN due to mixed const and non-const blocks
  Backport #87198 to 25.9: Disable the setting by default
  Backport #87392 to 25.9: Fix EmbeddedRocksDB upgrade
  Backport #87140 to 25.9: fix: fix server level max temporary size limit
  Backport #87178 to 25.9: PR: fix LEFT/INNER ... RIGHT ... JOINS chain
  Update autogenerated version to 25.10.1.1 and contributors

robot-ch-test-poll4 added the pr-backports-created label

azat mentioned this pull request

Safer IObjectStorage::tryGetObjectMetadata() (ignore only object existence error) #87485

Merged

Member Author

azat commented Sep 23, 2025

And a follow-up to fix the interface - #87485

azat added a commit that referenced this pull request


          Merge pull request #87471 from ClickHouse/backport/25.3/87426

0b17090

Backport #87426 to 25.3: Ignore only not found errors for s3_plain_rewritable (and some other S3 code)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-backports-created pr-backports-created-cloud pr-critical-bugfix pr-must-backport pr-must-backport-synced pr-synced-to-cloud