rgw: only skip versioned to non-versioned buckets by clwluvw · Pull Request #62642 · ceph/ceph

clwluvw · 2025-04-03T02:20:09Z

Since replication entries are compared against the current state of the bucket, delays—regardless of the cause—can lead to inconsistencies when a bucket transitions from non-versioned to versioned. Entries created before versioning was enabled may not have version IDs, which could result in discrepancies.

To address this, we will allow replication of non-versioned objects to versioned buckets as null versions while skipping only those entries that are versioned when the destination bucket is non-versioned.

The same approach applies to suspended versioning. Since suspended buckets generate objects with a null version ID, replication will be permitted to both versioned and non-versioned destinations.

Fixes: https://tracker.ceph.com/issues/70728 & https://tracker.ceph.com/issues/70486
~~Will be rebased with #62637~~

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

cbodley

thanks, the pr description sounds very reasonable. it looks like the tests have solid coverage of creates, deletes, delete markers 👍

cbodley · 2025-04-04T16:22:54Z

src/test/rgw/rgw_multi/tests.py

        # by default, wait up to 5 minutes before giving up on a sync checkpoint
        self.checkpoint_retries = kwargs.get('checkpoint_retries', 60)
-        self.checkpoint_delay = kwargs.get('checkpoint_delay', 5)
+        self.checkpoint_delay = kwargs.get('checkpoint_delay', 15)


why was this part necessary? it seems like it has the potential to slow down the multisite tests in general

I observed an unexpected state during delete replication, where it seemed that two stages of data sync were required, and invoking zone_data_checkpoint() alone wasn't sufficient.

I haven't investigated it in detail yet, but it might be related to the presence of two sync pipelines—one between zones and another between buckets—which could affect how shards are processed or updated. Increasing the timeout might have compensated for the first checkpoint.

For now, I've reverted the timeout to the default value of 5 to see how Teuthology handles it. If it fails there as well, I'll dive deeper to identify the root cause and explore a proper fix without relying on timeout adjustments.

perhaps a zonegroup_bucket_checkpoint() would work better? the data checkpoints are based on data sync status, which doesn't take the 'error repo' into account if retries are necessary

didn’t help. The errors in the ‘error repo’ seem to be mostly related to resharding if I'm not mistaken, so it might not be the case here.

github-actions · 2025-04-04T16:39:49Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

clwluvw · 2025-04-07T12:28:42Z

https://jenkins.ceph.com/job/ceph-pull-requests/155571/

295 - run-tox-qa (Failed)

clwluvw · 2025-04-07T12:28:50Z

jenkins test make check

clwluvw · 2025-04-07T12:28:58Z

jenkins test make check arm64

github-actions · 2025-04-23T18:31:45Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

cbodley · 2025-04-23T20:13:00Z

src/rgw/driver/rados/rgw_data_sync.cc

+          // make sure versioned object only lands on versioned bucket
+          if (!key.instance.empty() && !sync_pipe.dest_bucket_info.versioned()) {


do we know whether key.instance.empty() matches null versions? is this behavior the same for full- vs incremental sync?

i see RGWBucketSyncSingleEntryCR has a bool versioned that comes from rgw_bi_log_entry::is_versioned(), along with a bool null_verid from rgw_bi_log_entry::is_null_verid()

maybe those are relevant here, although RGWBucketFullSyncCR passes false for both

do we know whether key.instance.empty() matches null versions?

yes it does.

there was a discussion around those vars here: #62594
but what i could understand eventually was, is_null_verid is only implemented for delete (#62594 (comment)), and for is_versioned() for null version id it results in false.

github-actions · 2025-04-29T14:27:51Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

Reject PutBucketReplication calls if versioning is not identical between the source and destination buckets. This check also applies to object lock configurations to ensure consistency. Fixes: https://tracker.ceph.com/issues/70486 Signed-off-by: Seena Fallah <[email protected]>

replicating lock enabled objects require the destination bucket to have object lock enabled. Fixes: https://tracker.ceph.com/issues/70486 Signed-off-by: Seena Fallah <[email protected]>

…d buckets When a bucket transitions from non-versioned to versioned, there may be a delay in synchronizing this state and the already ongoing data replication of non-versioned objects. During this period, some older non-versioned objects may still need to be replicated to the now-versioned destination bucket. To handle this, non-versioned objects can be replicated to a versioned bucket with a null version, ensuring no data is lost during the transition. However, replication should still skip versioned objects when syncing to a non-versioned bucket. Signed-off-by: Seena Fallah <[email protected]>

…ed buckets To properly replicate delete requests from a time when a bucket was non-versioned before transitioning to versioned, such deletes should be applied to the null version in the destination bucket. we no need to set version to true for CLS_RGW_OP_UNLINK_INSTANCE as after 92002be those bilogs will have RGW_BILOG_FLAG_VERSIONED_OP flag. Signed-off-by: Seena Fallah <[email protected]>

Signed-off-by: Seena Fallah <[email protected]>

github-actions · 2025-06-28T17:01:27Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

github-actions · 2025-07-28T18:02:34Z

This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution!

github-actions · 2025-09-26T19:01:57Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

github-actions · 2025-12-05T22:02:00Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

github-actions · 2026-01-04T22:02:02Z

This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution!

clwluvw requested a review from a team as a code owner April 3, 2025 02:20

github-actions bot added rgw tests labels Apr 3, 2025

clwluvw requested review from cbodley and smanjara April 3, 2025 02:22

cbodley reviewed Apr 4, 2025

View reviewed changes

github-actions bot added the needs-rebase label Apr 4, 2025

clwluvw force-pushed the mismatch-versioning-fix branch from 43a934d to 62f659d Compare April 6, 2025 10:03

github-actions bot removed the needs-rebase label Apr 6, 2025

clwluvw mentioned this pull request Apr 9, 2025

rgw: add support bucket replication between zonegroups #59911

Closed

github-actions bot added the needs-rebase label Apr 23, 2025

clwluvw force-pushed the mismatch-versioning-fix branch from 62f659d to 7e3ccb8 Compare April 23, 2025 19:03

github-actions bot removed the needs-rebase label Apr 23, 2025

cbodley reviewed Apr 23, 2025

View reviewed changes

github-actions bot added the needs-rebase label Apr 29, 2025

clwluvw added 5 commits April 29, 2025 17:08

qa/rgw: add more test for mismatch state of versioning

7a5a0ac

Signed-off-by: Seena Fallah <[email protected]>

clwluvw force-pushed the mismatch-versioning-fix branch from 7e3ccb8 to 7a5a0ac Compare April 29, 2025 15:17

github-actions bot removed the needs-rebase label Apr 29, 2025

clwluvw mentioned this pull request Apr 29, 2025

rgw: report failed replication status caused by user errors #62147

Closed

4 tasks

github-actions bot added the stale label Jun 28, 2025

github-actions bot closed this Jul 28, 2025

clwluvw reopened this Jul 28, 2025

clwluvw requested a review from cbodley July 28, 2025 18:04

clwluvw removed the stale label Jul 28, 2025

github-actions bot added the stale label Sep 26, 2025

clwluvw removed the stale label Oct 6, 2025

github-actions bot added the stale label Dec 5, 2025

github-actions bot closed this Jan 4, 2026

		// make sure versioned object only lands on versioned bucket
		if (!key.instance.empty() && !sync_pipe.dest_bucket_info.versioned()) {

Comments

Conversation

clwluvw commented Apr 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

cbodley left a comment

Choose a reason for hiding this comment

Uh oh!

cbodley Apr 4, 2025

Choose a reason for hiding this comment

Uh oh!

clwluvw Apr 6, 2025

Choose a reason for hiding this comment

Uh oh!

cbodley Apr 7, 2025

Choose a reason for hiding this comment

Uh oh!

clwluvw Apr 7, 2025

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 4, 2025

Uh oh!

clwluvw commented Apr 7, 2025

Uh oh!

clwluvw commented Apr 7, 2025

Uh oh!

clwluvw commented Apr 7, 2025

Uh oh!

github-actions bot commented Apr 23, 2025

Uh oh!

cbodley Apr 23, 2025

Choose a reason for hiding this comment

Uh oh!

clwluvw Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Apr 29, 2025

Uh oh!

github-actions bot commented Jun 28, 2025

Uh oh!

github-actions bot commented Jul 28, 2025

Uh oh!

github-actions bot commented Sep 26, 2025

Uh oh!

github-actions bot commented Dec 5, 2025

Uh oh!

github-actions bot commented Jan 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

clwluvw commented Apr 3, 2025 •

edited

Loading

clwluvw Apr 23, 2025 •

edited

Loading