without tests or examples, it's hard to tell exactly what this does. maybe you could share a couple examples of replication policies that are enabled by this?

if you have two active-active zones A and B, i would have guessed that cross-bucket policies already work because objects uploaded to A:bucket1 would replicate to B:bucket2, then back to A:bucket2 from there. so is the primary motivation to enable these replication policies in single-zone configurations?

@cbodley the scenario that you describe would work only if there's bidirectional sync between A and B. Also, as you pointed out doesn't work at all if there's only a single zone. From the perspective of the S3 API users this can be very confusing as the topology might not even be visible to them and the expectation is that when you replicate a bucket into another one then the data shows up there.

what kind of configuration management changes would this entail? a user might just want to replicate buckets within the same zone and not opt for multi-zone env at all. they may also expect to access the replicated buckets with separate accounts, as it seems like the main use case for SRR.

is it safe to assume that these sync policy changes cause us to spawn a RGWDataSyncProcessorThread against the local zone's endpoints, consult local sync status when trimming datalogs/bilogs, report on local status in 'radosgw-admin sync status', etc?

This is probably not the most efficient way to achieve cross bucket sync

two thoughts here:

1. it would be great to take advantage of our ref-counting optimization for server-side copy here
2. 'local' data sync still has to process every datalog/bilog entry written, even though these cross-bucket events probably represent a small subset of those. that means the cost of this feature scales faster than the value it provides

perhaps we could add a special case to RGWDefaultDataSyncModule::sync_object() for src_zone==dst_zone that skips all calls with src_obj==dst_obj, and calls RGWRados::copy_obj() otherwise? hopefully there's a way to prevent copy_obj() from overwriting new objects with older ones

that could go a long way to reduce the per-object cost of this feature, but the additional cost of polling seems inescapable. considering a zonegroup with two active-active zones, this would double the number of datalog/bilog listing requests overall

@smanjara we would need to figure out what the correct configuration is. As it is right now this PR is missing the configuration part. It should still work on a single zone (that's how I tested it), but we'll need to create a way to enable/disable this feature.

@cbodley copying object instead of fetching it sounds like a relatively easy optimization. The RGWObjFetchCR does a bit more than just fetching the object, it also in certain cases stats the remote object to get its tags, so that it can match it with the appropriate replication policy rules. I'm not sure copy pasting all this logic would be the best way forward, can probably add the logic into the RGWObjFetchCR itself.

@smanjara we would need to figure out what the correct configuration is. As it is right now this PR is missing the configuration part. It should still work on a single zone (that's how I tested it), but we'll need to create a way to enable/disable this feature.

does the sync policy framework have a way to add a unidirectional pipe from one zone to another? adding such a pipe with src_zone==dst_zone seems like a reasonable way for users to opt in to this

@smanjara we would need to figure out what the correct configuration is. As it is right now this PR is missing the configuration part. It should still work on a single zone (that's how I tested it), but we'll need to create a way to enable/disable this feature.

does the sync policy framework have a way to add a unidirectional pipe from one zone to another? adding such a pipe with src_zone==dst_zone seems like a reasonable way for users to opt in to this

I'm not sure we allow it explicitly, but that's what this PR does implicitly. We can bake some syntactic sugar into the policy that would be easier for the users to digest.

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution!