Skip to content

Prevent a bug in AWS S3#48816

Merged
robot-ch-test-poll1 merged 2 commits intomasterfrom
fix-s3
Apr 16, 2023
Merged

Prevent a bug in AWS S3#48816
robot-ch-test-poll1 merged 2 commits intomasterfrom
fix-s3

Conversation

@alexey-milovidov
Copy link
Copy Markdown
Member

@alexey-milovidov alexey-milovidov commented Apr 15, 2023

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

If a user set max_single_part_upload_size to a very large value, it can lead to a crash due to a bug in the AWS S3 SDK. This fixes #47679.

Note: no tests will be provided for this issue.

@robot-ch-test-poll1 robot-ch-test-poll1 added the pr-improvement Pull request with some product improvements label Apr 15, 2023
@antonio2368 antonio2368 self-assigned this Apr 16, 2023
Comment on lines +80 to +85

/// AWS S3 SDK library has a bug. It is using std::*stream (which is a major offense).
/// LLVM libc++ has a bug. It does not allow std::*stream to work for large strings.

if (max_single_part_upload_size >= 2_GiB)
throw Exception(ErrorCodes::INVALID_SETTING_VALUE, "Uploading parts of 2 GiB or larger is not supported due to a bug in AWS and LLVM. Lower the value of `s3_max_single_part_upload_size` setting.");
Copy link
Copy Markdown
Member

@antonio2368 antonio2368 Apr 16, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just noticed there is validate function for this, auto merge already triggered 🥲

EDIT: requested review again before the job was executed, let's see what will happen

@antonio2368 antonio2368 self-requested a review April 16, 2023 12:35
@robot-ch-test-poll1 robot-ch-test-poll1 merged commit 69cac0a into master Apr 16, 2023
@robot-ch-test-poll1 robot-ch-test-poll1 deleted the fix-s3 branch April 16, 2023 12:46
azat added a commit to azat/ClickHouse that referenced this pull request Apr 17, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

pr-improvement Pull request with some product improvements

Projects

None yet

Development

Successfully merging this pull request may close these issues.

SIGSEGV while uploading to S3 for part > INT_MAX

3 participants