Skip to content

Support batch_delete capability for GCS#37659

Merged
alesapin merged 3 commits intoClickHouse:masterfrom
frew:master
Jun 13, 2022
Merged

Support batch_delete capability for GCS#37659
alesapin merged 3 commits intoClickHouse:masterfrom
frew:master

Conversation

@frew
Copy link
Copy Markdown
Contributor

@frew frew commented May 30, 2022 •
edited by alesapin
Loading

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Add setting support_batch_delete for DiskS3 to disable multiobject delete calls, which Google Cloud Storage doesn't support.

@CLAassistant
Copy link
Copy Markdown

CLAassistant commented May 30, 2022 •
edited
Loading

CLA assistant check
All committers have signed the CLA.

@frew frew changed the title Add support for single DeleteObject when chunk_size == 0 Add support for single DeleteObject when objects_chunk_size_to_delete == 0 May 30, 2022
@robot-ch-test-poll robot-ch-test-poll added the pr-bugfix Pull request with bugfix, not backported by default label May 30, 2022
@alesapin alesapin self-assigned this May 30, 2022
alesapin
alesapin reviewed May 30, 2022
View reviewed changes
alesapin
alesapin reviewed May 30, 2022
View reviewed changes
@alesapin alesapin added can be tested Allows running workflows for external contributors pr-improvement Pull request with some product improvements and removed pr-bugfix Pull request with bugfix, not backported by default labels May 30, 2022
@alesapin
Copy link
Copy Markdown
Member

alesapin commented May 31, 2022

I think the best way is to pass some structure to S3ObjectStorage, like S3Capabilities:

struct S3Capabilities
{
    bool support_batch_delete = true;
};

This struct will be filled from config. Currently, we will have only one setting: support_batch_delete (true by default). For disk, settings check Disks/S3/diskSettings.cpp and registerDiskS3.cpp.

@frew
Copy link
Copy Markdown
Contributor Author

frew commented May 31, 2022

Cool - will take doing that later this week!

@alesapin alesapin changed the title Add support for single DeleteObject when objects_chunk_size_to_delete == 0 Support batch_delete capability for GCS Jun 12, 2022
@alesapin
Copy link
Copy Markdown
Member

alesapin commented Jun 13, 2022

Failures obviously unrelated (this setting is not enabled anywhere).

@alesapin alesapin merged commit 06d94a4 into ClickHouse:master Jun 13, 2022
@alex-zaitsev alex-zaitsev mentioned this pull request Jun 16, 2022
Merged
@frew
Copy link
Copy Markdown
Contributor Author

frew commented Jun 26, 2022

@alesapin Sorry for falling off the face of the earth here (caught COVID and then had a bunch of work to catch up on). Thanks for taking this over the finish line!

@azat azat mentioned this pull request Jun 30, 2022
Closed
@azat
Copy link
Copy Markdown
Member

azat commented Jun 30, 2022 •
edited
Loading

It looks like it worth adding this setting into documentation, and add a note that this is required for GCS.
And maybe it should be enabled by default for GCS (i.e. if hostname contains storage.googleapis.com)

Oh, I see, actually #37882 already adds detection of batch detele.

@UnamedRus UnamedRus mentioned this pull request Jul 13, 2022
Closed
excitoon pushed a commit to Altinity/ClickHouse that referenced this pull request Jul 14, 2022
@alesapin @excitoon
Merge pull request ClickHouse#37659 from frew/master
9fb6a80 
Support `batch_delete` capability for GCS
@excitoon excitoon mentioned this pull request Jul 14, 2022
Merged
arthurpassos added a commit to Altinity/ClickHouse that referenced this pull request Jul 15, 2022
@arthurpassos
Merge pull request #164 from Altinity/backport/22.3_36944_34632
97b66d6 
Window functions and GCP fixes ClickHouse#36944, ClickHouse#34632, ClickHouse#37659 and ClickHouse#37882
Enmk pushed a commit to Altinity/ClickHouse that referenced this pull request Jul 28, 2022
@alesapin @Enmk
Merge pull request ClickHouse#37659 from frew/master
be4c872 
Support `batch_delete` capability for GCS
Enmk pushed a commit to Altinity/ClickHouse that referenced this pull request Jul 28, 2022
@arthurpassos @Enmk
Merge pull request #164 from Altinity/backport/22.3_36944_34632
38f7435 
Window functions and GCP fixes ClickHouse#36944, ClickHouse#34632, ClickHouse#37659 and ClickHouse#37882
@Enmk Enmk mentioned this pull request Jul 28, 2022
Closed
Enmk pushed a commit to Altinity/ClickHouse that referenced this pull request Aug 4, 2022
@alesapin @Enmk
Merge pull request ClickHouse#37659 from frew/master
9f933a0 
Support `batch_delete` capability for GCS
Enmk pushed a commit to Altinity/ClickHouse that referenced this pull request Aug 4, 2022
@arthurpassos @Enmk
Merge pull request #164 from Altinity/backport/22.3_36944_34632
797f3d3 
Window functions and GCP fixes ClickHouse#36944, ClickHouse#34632, ClickHouse#37659 and ClickHouse#37882
This was referenced Aug 4, 2022
Closed
Merged
@Enmk Enmk mentioned this pull request Sep 12, 2022
Merged
4 tasks
Enmk pushed a commit to Altinity/ClickHouse that referenced this pull request Sep 13, 2022
@alesapin @Enmk
Merge pull request ClickHouse#37659 from frew/master
ed4a627 
Support `batch_delete` capability for GCS
Enmk pushed a commit to Altinity/ClickHouse that referenced this pull request Sep 17, 2022
@novikd @Enmk
PR #164 Window functions and GCP fixes ClickHouse#36944, ClickHouse#3…
d80e330 
…4632, ClickHouse#37659 and ClickHouse#37882

Merge pull request ClickHouse#36944 from excitoon-favorites/better_exp_smooth
Merge pull request ClickHouse#34632 from excitoon-favorites/optimizedprocessing
Merge pull request ClickHouse#37659 from frew/master
Support `batch_delete` capability for GCS
Merge pull request ClickHouse#37882 from excitoon-favorites/nodeleteobjects
Fixes for objects removal in `S3ObjectStorage`
@Enmk Enmk mentioned this pull request Dec 22, 2022
Merged
@DanRoscigno DanRoscigno mentioned this pull request Jan 15, 2023
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@alesapin alesapin alesapin approved these changes

Assignees

@alesapin alesapin

Labels

can be tested Allows running workflows for external contributors pr-improvement Pull request with some product improvements

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@frew @CLAassistant @alesapin @azat @robot-ch-test-poll