Possibility to change S3 disk settings in runtime#23429
Possibility to change S3 disk settings in runtime#23429vdimir merged 12 commits intoClickHouse:masterfrom
Conversation
robot-clickhouse
added
the
pr-improvement
src/Disks/IDisk.h
Outdated
| friend class DiskDecorator; | ||
|
|
||
| /// Applies new settings for disk in runtime. | ||
| virtual void applyNewSettings(ContextConstPtr) { } |
There was a problem hiding this comment.
Maybe throw 'not implemented' error by default to catch calling method on disks that does not support this it?
src/Disks/S3/DiskS3.cpp
Outdated
| s3_max_single_read_retries = new_settings.s3_max_single_read_retries; | ||
| s3_min_upload_part_size = new_settings.s3_min_upload_part_size; | ||
| s3_max_single_part_upload_size = new_settings.s3_max_single_part_upload_size; | ||
| min_bytes_for_seek = new_settings.min_bytes_for_seek; | ||
| send_metadata = new_settings.send_metadata; | ||
| list_object_keys_size = new_settings.list_object_keys_size; |
There was a problem hiding this comment.
Can we store DiskS3Settings in DiskS3 and replace this object at once? Now we have same logic of putting corresponding fields from DiskS3Settings to DiskS3 fields in constructor and in applyNewSettings, but we can manipulate with whole object.
| swap(*impl); | ||
| auto position = impl->getPosition(); | ||
| swap(*impl); |
There was a problem hiding this comment.
I see that this swap was in deleted code, but didn't get why we swap buffers
There was a problem hiding this comment.
Decorator buffers don't have own buffer in memory, so we need to 'steal' actual working buffer from impl, perform an operation and return buffer back.
|
Internal documentation ticket: DOCSUP-18488 |
|
@sevirov , I was wondering for the documentation for this feature. Thanks! |
|
@HeenaBansal2009 I'm afraid it wasn't finished and is abandoned. I suppose best way to understand how feature work is to read changes in integration test for this PR. |
|
Thanks @vdimir for response. However , I am just curious Since this feature was implemented for backup/restore metadata files from S3 ? If it is not production use and abandoned , How current backup/restore feature of CH are going to be affected . I was searching for any backup restore documentation if available. Thanks in advance. |
|
@HeenaBansal2009 New backup/restore is not related to this (here is the PRs for new backup/restore). Not sure about docs for the new feature. @vitlibar maybe you can answer? |
|
@vdimir Its good to see that backup/restore support is planned for 2022. I did find docs in clickhouse documentation for backup/restore commands. However , I wanted to know is these commands does online backup? OR backup command does freeze the table and take snapshot |
|
@HeenaBansal2009 https://clickhouse.com/docs/en/operations/backup/ some documentation here |
6 participants
I hereby agree to the terms of the CLA available at: https://yandex.ru/legal/cla/?lang=en
Changelog category (leave one):
Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):
Possibility to change S3 disk settings in runtime via new
SYSTEM RESTART DISKSQL command.Detailed description / Documentation draft:
Introduced DiskRestartProxy that re-applies changed S3 disk settings in runtime.
Introduced
SYSTEM RESTART DISKSQL command that invokes DiskRestartProxy->restart().Reduced memory usage for WriteBufferFromS3 decorators (each decorator allocated own buffer in memory that was not actually used).
ReadBufferFromS3/WriteBufferFromS3 decorators code refactoring.