I’ve been reflecting on where this should be invoked. In #59911, the approach I took was to mark bilog entries with the zones responsible for processing (log_zones), and I initially thought zones should unconditionally process those entries as the criteria (in terms of filters) had already been validated by the source zone when the entry was logged. This would cover most scenarios, aside from two edge cases—bucket deletion during replication or changes in permissions.

That approach was based on zones, as at the moment, sync pipes can only keep zones, not zonegroups. Based on your insights from #61140 (comment), I believe we need to shift focus from zones to zonegroups. The master can change over time, and we want to avoid missing any processing entries.

What I’m envisioning is this: during the logging phase, we still mark the relevant zonegroups based on the matching pipes, rather than individual zones. And if there were no matching pipes, we should log when there is more than one zone per zonegroup with empty log_zonegroups set. Then, during the processing phase, when we fetch entries, we would check if the zonegroup of the requested zone (rgwx-zonegroup) matches ourselves (destination). If it does, we don’t apply any additional filtering on log_zonegroups and return all entries. and if it doesn't we just return the ones having rgwx-zonegroup in the log_zonegroups set.

This would also require modifying the sync pipes to store a bucket’s zonegroup hint, instead of its zone, as constantly looking up the zonegroup for each request wouldn’t be efficient. Buckets don’t frequently move, and in cases of bucket recreation, we can update the policies as needed, maintaining compatibility with AWS—where even if a bucket is moved mid-replication, it still replicates.

So, my question is: do we need to perform this validation on the pull side, or would it be sufficient to handle it during the logging phase, relying on the log_zonegroups field? perhaps just ANDing the case where replication is possible from my zonegorup to the destination bucket's zonegroup. So we change this to should_sync_to rather than from to promote the case on which side this needs to be considered. although if it's on the logging side this would lose the immediate effect if an admin wants to stop replication from zonegroup A to B. So maybe both can be considered anyway.

What I’m envisioning is this: during the logging phase, we still mark the relevant zonegroups based on the matching pipes, rather than individual zones. And if there were no matching pipes, we should log when there is more than one zone per zonegroup with empty log_zonegroups set. Then, during the processing phase, when we fetch entries, we would check if the zonegroup of the requested zone (rgwx-zonegroup) matches ourselves (destination). If it does, we don’t apply any additional filtering on log_zonegroups and return all entries. and if it doesn't we just return the ones having rgwx-zonegroup in the log_zonegroups set.

that sounds right to me.

This would also require modifying the sync pipes to store a bucket’s zonegroup hint, instead of its zone, as constantly looking up the zonegroup for each request wouldn’t be efficient. Buckets don’t frequently move, and in cases of bucket recreation, we can update the policies as needed, maintaining compatibility with AWS—where even if a bucket is moved mid-replication, it still replicates.

sorry not clear, could you clarify what moving a bucket mean?

looks like we will need to introduce the concept of master zone in data sync.

sorry not clear, could you clarify what moving a bucket mean?

So if you have a policy on your source bucket pointing to a destination bucket in Region A and then you delete the destination bucket and create it in Region B, AWS will continue replicating from the source bucket to the newly created bucket in Region B without any modification needed by the user.

So, my question is: do we need to perform this validation on the pull side, or would it be sufficient to handle it during the logging phase, relying on the log_zonegroups field?

for the RGWBucketInfo overload of should_sync_from(), i was thinking we'd only use that on PutBucketReplication to reject any cross-zonegroup policy that's disabled by the admin. though i guess this would open a loophole if the destination bucket was deleted and recreated on a zonegroup that wasn't supposed to be enabled