Skip to content

use removeObjectIfExists instead removeObject, it is retryable #71529

Merged
CheSema merged 3 commits intomasterfrom
chesema-remove-if-exists
Nov 13, 2024
Merged

use removeObjectIfExists instead removeObject, it is retryable #71529
CheSema merged 3 commits intomasterfrom
chesema-remove-if-exists

Conversation

@CheSema
Copy link
Copy Markdown
Member

@CheSema CheSema commented Nov 6, 2024

Changelog category (leave one):

  • Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

The methods removeObject and removeObjects are not idempotent. When retries happen due to network errors, the result could be object not found because it has been deleted at previous attempts.

Documentation entry for user-facing changes

  • Documentation is written (mandatory for new features)

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

CI Settings (Only check the boxes if you know what you are doing):

  • Allow: All Required Checks
  • Allow: Stateless tests
  • Allow: Stateful tests
  • Allow: Integration Tests
  • Allow: Performance tests
  • Allow: All Builds
  • Allow: batch 1, 2 for multi-batch jobs
  • Allow: batch 3, 4, 5, 6 for multi-batch jobs
  • Exclude: Style check
  • Exclude: Fast test
  • Exclude: All with ASAN
  • Exclude: All with TSAN, MSAN, UBSAN, Coverage
  • Exclude: All with aarch64, release, debug
  • Run only fuzzers related jobs (libFuzzer fuzzers, AST fuzzers, etc.)
  • Exclude: AST fuzzers
  • Do not test
  • Woolen Wolfdog
  • Upload binaries for special builds
  • Disable merge-commit
  • Disable CI cache

@robot-ch-test-poll1 robot-ch-test-poll1 added the pr-improvement Pull request with some product improvements label Nov 6, 2024
@robot-ch-test-poll3
Copy link
Copy Markdown
Contributor

robot-ch-test-poll3 commented Nov 6, 2024 •
edited by robot-ch-test-poll2
Loading

This is an automated comment for commit 40c4183 with description of existing statuses. It's updated for the latest CI running

✅ Click here to open a full report in a separate page

Successful checks
Check nameDescriptionStatus
AST fuzzerRuns randomly generated queries to catch program errors. The build type is optionally given in parenthesis. If it fails, ask a maintainer for help✅ success
BuildsThere's no description for the check yet, please add it to tests/ci/ci_config.py:CHECK_DESCRIPTIONS✅ success
ClickBenchRuns [ClickBench](https://github.com/ClickHouse/ClickBench/) with instant-attach table✅ success
Compatibility checkChecks that clickhouse binary runs on distributions with old libc versions. If it fails, ask a maintainer for help✅ success
Docker keeper imageThe check to build and optionally push the mentioned image to docker hub✅ success
Docker server imageThe check to build and optionally push the mentioned image to docker hub✅ success
Docs checkBuilds and tests the documentation✅ success
Fast testNormally this is the first check that is ran for a PR. It builds ClickHouse and runs most of stateless functional tests, omitting some. If it fails, further checks are not started until it is fixed. Look at the report to see which tests fail, then reproduce the failure locally as described here✅ success
Flaky testsChecks if new added or modified tests are flaky by running them repeatedly, in parallel, with more randomization. Functional tests are run 100 times with address sanitizer, and additional randomization of thread scheduling. Integration tests are run up to 10 times. If at least once a new test has failed, or was too long, this check will be red. We don't allow flaky tests, read the doc✅ success
Install packagesChecks that the built packages are installable in a clear environment✅ success
Integration testsThe integration tests report. In parenthesis the package type is given, and in square brackets are the optional part/total tests✅ success
Performance ComparisonMeasure changes in query performance. The performance test report is described in detail here. In square brackets are the optional part/total tests✅ success
Stateful testsRuns stateful functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc✅ success
Stateless testsRuns stateless functional tests for ClickHouse binaries built in various configurations -- release, debug, with sanitizers, etc✅ success
Stress testRuns stateless functional tests concurrently from several clients to detect concurrency-related errors✅ success
Style checkRuns a set of checks to keep the code style clean. If some of tests failed, see the related log from the report✅ success
Unit testsRuns the unit tests for different release types✅ success
Upgrade checkRuns stress tests on server version from last release and then tries to upgrade it to the version from the PR. It checks if the new server can successfully startup without any errors, crashes or sanitizer asserts✅ success

@divanik divanik self-assigned this Nov 6, 2024
divanik
divanik reviewed Nov 7, 2024
View reviewed changes
src/Disks/ObjectStorages/Local/LocalObjectStorage.h Outdated
Comment on lines +45 to +47
void removeObject(const StoredObject & object);

void removeObjects(const StoredObjects & objects) override;
void removeObjects(const StoredObjects & objects);
Copy link
Copy Markdown
Member

@divanik divanik Nov 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make these functions private

Copy link
Copy Markdown
Member

@divanik divanik Nov 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also add const or //NOLIT to definition to suppress clang-tidy

divanik
divanik reviewed Nov 7, 2024
View reviewed changes
src/Disks/ObjectStorages/HDFS/HDFSObjectStorage.h Outdated
Comment on lines +81 to +83
void removeObject(const StoredObject & object);

void removeObjects(const StoredObjects & objects) override;
void removeObjects(const StoredObjects & objects);
Copy link
Copy Markdown
Member

@divanik divanik Nov 7, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Make these functions private

divanik
divanik reviewed Nov 7, 2024
View reviewed changes
Copy link
Copy Markdown
Member

@divanik divanik left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The PR requires minor fixes

@CheSema CheSema requested a review from divanik November 12, 2024 14:04
@divanik divanik added this pull request to the merge queue Nov 13, 2024
@divanik divanik removed this pull request from the merge queue due to a manual request Nov 13, 2024
@CheSema CheSema added this pull request to the merge queue Nov 13, 2024
Merged via the queue into master with commit e1901fa Nov 13, 2024
@CheSema CheSema deleted the chesema-remove-if-exists branch November 13, 2024 14:38
@robot-ch-test-poll2 robot-ch-test-poll2 added the pr-synced-to-cloud The PR is synced to the cloud repo label Nov 13, 2024
tavplubix
tavplubix reviewed Dec 9, 2024
View reviewed changes
src/Disks/ObjectStorages/IObjectStorage.h
Comment on lines 163 to +168
/// Remove object. Throws exception if object doesn't exists.
virtual void removeObject(const StoredObject & object) = 0;
// virtual void removeObject(const StoredObject & object) = 0;

/// Remove multiple objects. Some object storages can do batch remove in a more
/// optimal way.
virtual void removeObjects(const StoredObjects & objects) = 0;
// virtual void removeObjects(const StoredObjects & objects) = 0;
Copy link
Copy Markdown
Member

@tavplubix tavplubix Dec 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove?

Copy link
Copy Markdown
Member Author

@CheSema CheSema Dec 9, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will create new PR for it

Copy link
Copy Markdown
Member Author

@CheSema CheSema Dec 10, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

#73080

@tavplubix tavplubix added the pr-must-backport Pull request should be backported intentionally. Use this label with great care! label Dec 9, 2024
This was referenced Dec 9, 2024
Closed
Merged
Merged
Merged
This was referenced Dec 10, 2024
Closed
Merged
CheSema added a commit that referenced this pull request Dec 12, 2024
@CheSema
Merge pull request #72973 from ClickHouse/cherrypick/24.10/71529
4448a6e 
Cherry pick #71529 to 24.10: use removeObjectIfExists instead removeObject, it is retryable
CheSema added a commit that referenced this pull request Dec 12, 2024
@CheSema
Merge pull request #72972 from ClickHouse/cherrypick/24.9/71529
f8db845 
Cherry pick #71529 to 24.9: use removeObjectIfExists instead removeObject, it is retryable
CheSema added a commit that referenced this pull request Dec 12, 2024
@CheSema
Merge pull request #72971 from ClickHouse/cherrypick/24.8/71529
15f0b60 
Cherry pick #71529 to 24.8: use removeObjectIfExists instead removeObject, it is retryable
robot-clickhouse added a commit that referenced this pull request Dec 12, 2024
@robot-clickhouse
Backport #71529 to 24.8: use removeObjectIfExists instead removeObjec…
11d1a80 
…t, it is retryable
@robot-clickhouse robot-clickhouse mentioned this pull request Dec 12, 2024
Closed
robot-clickhouse added a commit that referenced this pull request Dec 12, 2024
@robot-clickhouse
Backport #71529 to 24.9: use removeObjectIfExists instead removeObjec…
fdc4703 
…t, it is retryable
@robot-clickhouse robot-clickhouse mentioned this pull request Dec 12, 2024
Closed
robot-clickhouse added a commit that referenced this pull request Dec 12, 2024
@robot-clickhouse
Backport #71529 to 24.10: use removeObjectIfExists instead removeObje…
d2951e2 
…ct, it is retryable
@robot-clickhouse robot-clickhouse mentioned this pull request Dec 12, 2024
Merged
@robot-ch-test-poll3 robot-ch-test-poll3 added the pr-backports-created-cloud deprecated label, NOOP label Dec 12, 2024
@robot-ch-test-poll4 robot-ch-test-poll4 added the pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore label Dec 12, 2024
robot-ch-test-poll1 added a commit that referenced this pull request Dec 12, 2024
@robot-ch-test-poll1
Merge pull request #73242 from ClickHouse/backport/24.10/71529
35eabdc 
Backport #71529 to 24.10: use removeObjectIfExists instead removeObject, it is retryable
@yokofly yokofly mentioned this pull request Dec 22, 2025
Merged
3 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@tavplubix tavplubix tavplubix left review comments

@divanik divanik divanik approved these changes

Assignees

@divanik divanik

Labels

pr-backports-created Backport PRs are successfully created, it won't be processed by CI script anymore pr-backports-created-cloud deprecated label, NOOP pr-improvement Pull request with some product improvements pr-must-backport Pull request should be backported intentionally. Use this label with great care! pr-synced-to-cloud The PR is synced to the cloud repo

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@CheSema @robot-ch-test-poll3 @tavplubix @divanik @robot-ch-test-poll1 @robot-ch-test-poll2 @robot-ch-test-poll4