rgw/multisite: bucket instance cleanup after deletion by smanjara · Pull Request #53799 · ceph/ceph

smanjara · 2023-10-03T20:05:10Z

https://tracker.ceph.com/issues/20802

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "pacific"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows

smanjara · 2023-10-03T20:08:24Z

@cbodley this isn't ready yet. but I would like to get your feedback on the layout stuff.

src/rgw/rgw_bucket_layout.h

src/rgw/driver/rados/rgw_bucket.cc

src/rgw/rgw_bucket_layout.h

src/rgw/driver/rados/rgw_trim_bilog.cc

src/rgw/rgw_common.h

cbodley

the bucket layout and trim stuff looks great 👍

src/rgw/rgw_bucket_layout.h

src/rgw/driver/rados/rgw_bucket.cc

src/rgw/driver/rados/rgw_rados.cc

github-actions · 2023-11-23T19:48:22Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

smanjara · 2023-11-23T20:10:35Z

to-do: test the trim logic added and think about test cases.

github-actions · 2023-11-28T14:43:44Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

src/rgw/driver/rados/rgw_rados.cc

src/rgw/driver/rados/rgw_sal_rados.cc

src/rgw/driver/rados/rgw_bucket.cc

cbodley · 2023-11-29T23:07:25Z

looking great except for the s3select submodule change

mattbenjamin · 2023-11-30T02:47:23Z

this is amazing progress, shilpa!

src/rgw/driver/rados/rgw_bucket.cc

github-actions · 2024-01-23T22:21:42Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

smanjara · 2025-04-03T18:53:16Z

with 6544c69 the tests are passing: https://pulpito.ceph.com/smanjara-2024-12-17_09:39:58-rgw:multisite-wip-shilpa-cleanup-bi-distro-default-smithi/

i do love to see green multisite runs!
the test coverage for cleanup on bilog autotrim looks good, but we don't have any coverage for the new BucketNotEmpty check. i was hoping we could copy one of Soumya's sync policy test cases for this, but it looks like they're all disabled with fails_with_rgw :( maybe we could still add a test case and make sure it passes locally, but use fails_with_rgw if it's not reliable in teuthology?

i added a few more testcases after this to test sync policy checks. they pass locally. waiting on shaman to build so I can run another round of multisite suite.

multisite run looks ok. these are the same failures that are on main.
https://qa-proxy.ceph.com/teuthology/smanjara-2025-04-03_17:42:14-rgw:multisite-wip-shilpa-cleanup-bi-distro-default-smithi/8223188/teuthology.log
I had to disable sync policy related tests as they are quite flaky on teuthology. passes reliably when tested locally.

github-actions · 2025-04-04T16:28:19Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

tracker: https://tracker.ceph.com/issues/20802 Signed-off-by: Shilpa Jagannath <[email protected]>

…stanceMetadataHandler. create an async cr for removing bucket instance info in bilog trimming logic Signed-off-by: Shilpa Jagannath <[email protected]>

Signed-off-by: Shilpa Jagannath <[email protected]>

…stance info. - get rid of the helper function that adds the 'deleted' log layout. Signed-off-by: Shilpa Jagannath <[email protected]>

Signed-off-by: Shilpa Jagannath <[email protected]>

…layout Signed-off-by: Shilpa Jagannath <[email protected]>

Signed-off-by: Shilpa Jagannath <[email protected]>

- handle the case when StatusShards returns -ENOENT Signed-off-by: Shilpa Jagannath <[email protected]>

Signed-off-by: Shilpa Jagannath <[email protected]>

we may end up orphaning objects on remote zones when a delete bucket is issued on metadata master. to avoid this, list the buckets on remote zones and delete bucket only when empty. if a zone is unreachable we drop that zone and continue with bucket deletion. such zones might have orphaned objects that will have to be cleaned up using radosgw-admin tool Signed-off-by: Shilpa Jagannath <[email protected]>

function. Signed-off-by: Shilpa Jagannath <[email protected]>

Signed-off-by: Shilpa Jagannath <[email protected]>

…anceMetadataHandler Signed-off-by: Shilpa Jagannath <[email protected]>

removal should happen locally Signed-off-by: Shilpa Jagannath <[email protected]>

Signed-off-by: Shilpa Jagannath <[email protected]>

only handle them when the bucket metadata is deleted. there is a case when we get enoent when status objects have not been created yet, for example when bucket metadata is created and synced but no data exists yet and bucket sync status won't be initialized. these don't need special handling. Signed-off-by: Shilpa Jagannath <[email protected]>

…andler::remove - fix testcase Signed-off-by: Shilpa Jagannath <[email protected]>

Signed-off-by: Shilpa Jagannath <[email protected]>

when returning enotempty during bucket deletion Signed-off-by: Shilpa Jagannath <[email protected]>

Signed-off-by: Shilpa Jagannath <[email protected]>

during bucket deletion Signed-off-by: Shilpa Jagannath <[email protected]>

smanjara requested a review from a team as a code owner October 3, 2023 20:05

github-actions bot added the rgw label Oct 3, 2023

smanjara commented Oct 3, 2023

View reviewed changes

src/rgw/rgw_bucket_layout.h Outdated Show resolved Hide resolved

cbodley reviewed Oct 4, 2023

View reviewed changes

src/rgw/driver/rados/rgw_bucket.cc Outdated Show resolved Hide resolved

src/rgw/rgw_bucket_layout.h Outdated Show resolved Hide resolved

src/rgw/driver/rados/rgw_trim_bilog.cc Outdated Show resolved Hide resolved

smanjara force-pushed the wip-shilpa-cleanup-bi branch from 096c552 to 8836ca1 Compare October 4, 2023 17:21

smanjara force-pushed the wip-shilpa-cleanup-bi branch from 8836ca1 to 005214c Compare November 8, 2023 02:56

cbodley reviewed Nov 8, 2023

View reviewed changes

src/rgw/rgw_common.h Outdated Show resolved Hide resolved

cbodley reviewed Nov 22, 2023

View reviewed changes

src/rgw/rgw_bucket_layout.h Outdated Show resolved Hide resolved

src/rgw/driver/rados/rgw_bucket.cc Outdated Show resolved Hide resolved

src/rgw/driver/rados/rgw_rados.cc Outdated Show resolved Hide resolved

github-actions bot added the needs-rebase label Nov 23, 2023

smanjara force-pushed the wip-shilpa-cleanup-bi branch from 7968773 to 9a244a4 Compare November 23, 2023 20:00

github-actions bot removed the needs-rebase label Nov 23, 2023

github-actions bot added the needs-rebase label Nov 28, 2023

smanjara force-pushed the wip-shilpa-cleanup-bi branch from 9a244a4 to e791922 Compare November 28, 2023 17:59

github-actions bot removed the needs-rebase label Nov 28, 2023

cbodley reviewed Nov 28, 2023

View reviewed changes

src/rgw/driver/rados/rgw_rados.cc Outdated Show resolved Hide resolved

src/rgw/driver/rados/rgw_rados.cc Outdated Show resolved Hide resolved

src/rgw/driver/rados/rgw_sal_rados.cc Outdated Show resolved Hide resolved

src/rgw/driver/rados/rgw_bucket.cc Outdated Show resolved Hide resolved

smanjara force-pushed the wip-shilpa-cleanup-bi branch from e2909e6 to 409bdc0 Compare November 28, 2023 21:46

cbodley added the needs-test label Nov 30, 2023

smanjara force-pushed the wip-shilpa-cleanup-bi branch from 409bdc0 to 05fa85a Compare January 12, 2024 18:02

cbodley reviewed Jan 12, 2024

View reviewed changes

src/rgw/driver/rados/rgw_bucket.cc Outdated Show resolved Hide resolved

smanjara force-pushed the wip-shilpa-cleanup-bi branch from 05fa85a to 8ddcd2a Compare January 16, 2024 04:47

github-actions bot added the tests label Jan 16, 2024

smanjara force-pushed the wip-shilpa-cleanup-bi branch from 8ddcd2a to 273adda Compare January 18, 2024 17:01

github-actions bot added the needs-rebase label Jan 23, 2024

cbodley approved these changes Apr 3, 2025

View reviewed changes

github-actions bot added the needs-rebase label Apr 4, 2025

Shilpa Jagannath added 23 commits April 4, 2025 13:16

rgw/multisite: draft of bucket index cleanup after deletion

ed71755

tracker: https://tracker.ceph.com/issues/20802 Signed-off-by: Shilpa Jagannath <[email protected]>

rgw/multisite: handle the 'deleted' index log addition in RGWBucketIn…

c12c331

…stanceMetadataHandler. create an async cr for removing bucket instance info in bilog trimming logic Signed-off-by: Shilpa Jagannath <[email protected]>

rgw/multisite: add 'deleted' as a RGWBucketInfo flag

0ef0385

Signed-off-by: Shilpa Jagannath <[email protected]>

rgw/multisite: - don't overwrite bucket attrs while storing bucket in…

53c679a

…stance info. - get rid of the helper function that adds the 'deleted' log layout. Signed-off-by: Shilpa Jagannath <[email protected]>

rgw/multisite: retry on detecting bucket instance metadata racing writes

e1ecba0

Signed-off-by: Shilpa Jagannath <[email protected]>

rgw/multisite: helper function to store bucket info flag

175a0e8

Signed-off-by: Shilpa Jagannath <[email protected]>

rgw/multisite: add datalog entry when transitioning to 'DELETED' log …

2e4a7db

…layout Signed-off-by: Shilpa Jagannath <[email protected]>

qa/multisite: add a bilog trim testcase with bucket deletion

3d663ed

Signed-off-by: Shilpa Jagannath <[email protected]>

rgw/multisite: - add deleted layout on master zone

3aef41c

- handle the case when StatusShards returns -ENOENT Signed-off-by: Shilpa Jagannath <[email protected]>

qa/multisite: update test for bucket instance deletion

4ae08eb

Signed-off-by: Shilpa Jagannath <[email protected]>

rgw/multisite: move the remote bucket listing logic into a separate

4b343ac

function. Signed-off-by: Shilpa Jagannath <[email protected]>

rgw/multiiste: fix bucket enotempty check

cd16d5a

Signed-off-by: Shilpa Jagannath <[email protected]>

rgw/multisite: thread svc_datalog through RGWBucketCtl and BucketInst…

65112b8

…anceMetadataHandler Signed-off-by: Shilpa Jagannath <[email protected]>

rgw/multisite: don't sync metadata entry for bucket instance removal.

e2bf9de

removal should happen locally Signed-off-by: Shilpa Jagannath <[email protected]>

qa/multisite: test with checkpoint delays

50bd4bf

Signed-off-by: Shilpa Jagannath <[email protected]>

qa/multisite: test trimming bilog on all zonegroups

5e9b791

Signed-off-by: Shilpa Jagannath <[email protected]>

rgw/trim: restore update_mapping_update in RGWBucketInstanceMetadataH…

126ba3c

…andler::remove - fix testcase Signed-off-by: Shilpa Jagannath <[email protected]>

qa/tests: test for bucket enotempty check during delete

6020608

Signed-off-by: Shilpa Jagannath <[email protected]>

rgw/multisite: check for both zonegroup and bucket level sync policies

9e7385e

when returning enotempty during bucket deletion Signed-off-by: Shilpa Jagannath <[email protected]>

rgw: remove unused objv tracker

52df364

Signed-off-by: Shilpa Jagannath <[email protected]>

rgw/multisite: account for zones opted out of sync and object filters

8fbef97

during bucket deletion Signed-off-by: Shilpa Jagannath <[email protected]>

smanjara force-pushed the wip-shilpa-cleanup-bi branch from 42b6782 to 8fbef97 Compare April 4, 2025 17:20

github-actions bot removed the needs-rebase label Apr 4, 2025

smanjara merged commit e0958b5 into ceph:main Apr 4, 2025
12 checks passed

Conversation

smanjara commented Oct 3, 2023

Contribution Guidelines

Checklist

Uh oh!

smanjara commented Oct 3, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cbodley left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Nov 23, 2023

Uh oh!

smanjara commented Nov 23, 2023

Uh oh!

github-actions bot commented Nov 28, 2023

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

cbodley commented Nov 29, 2023

Uh oh!

mattbenjamin commented Nov 30, 2023

Uh oh!

Uh oh!

github-actions bot commented Jan 23, 2024

Uh oh!

smanjara commented Apr 3, 2025

Uh oh!

github-actions bot commented Apr 4, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants