Scenario

There are two clusters: ClusterA, ClusterB. Both clusters are configured with the standard delete_mode (reaping on 3s timeout).

An object key (K1) belongs to a Preflist of {VnodeA1, VnodeA2, VnodeA3} on ClusterA, and {VnodeB1, VnodeB2, VnodeB3} on ClusterB.

An object is initiated with that key K1 on ClusterA (VnodeA1 say, acting as coordinator). The creation is replicated.

The object is deleted on ClusterB (VnodeB1 say, acting as coordinator). The deletion is replicated.

For some reason, on one cluster (ClusterA say) the reap does not occur on the timeout.

Replication Goes Wrong

1. There is now an object on ClusterA, a tombstone with a vector clock like [{VnodeA1, {1, TS1}}, {VnodeB1, {1, TS2}}], that does not exist on ClusterB.
2. nextgenrepl full-sync now runs with ClusterA as the src, and ClusterB as the snk.
3. The aae_exchange discovers that A > B for K1 as K1 not_found on B. It puts a reference on the replication queue.
4. The fetch, which is run via riak_kv_get_fsm pulls across the tombstone object. But as it prompts a read (via riak_kv_get_fsm), the read final action spots that this is a tombstone, and so triggers a maybe_delete, which causes the tombstone to be reaped from ClusterA.
5. The push of the tombstone to ClusterB, is performed as expected on VnodeB2 and VnodeB3. However on VnodeB1 there is key amnesia - this vnode previously coordinated the write, but has no backend memory because of the reap.
6. Due to key amnesia, the object is written with an updated vector clock.
7. The riak_client:push function which has replicated the object prompts a GET of the object after the asis PUT. This GET is intended to prompt any delete actions (e.g. it is expected to reap). However, in this case it prompts a read-repair instead (there can only be one final action), because the tombstone's vclock differs at VnodeB1 from VnodeB2 and VnodeB3 due to the new actor epoch generated by key amnesia.
8. This means that the object has been reaped from ClusterA, but exists on ClusterB with a vector clock following read repair like [{VnodeA1, {1, TS1}}, {VnodeB1, {1, TS2}}, {VnodeB1.1, {1, TS3}}].
9. Now if full-sync runs from B to A - the same thing occurs in reverse. The net effect is the tombstone is reaped from ClusterB, but re-added to ClusterA with an expanded vector clock due to the new actor epoch (triggered by the object update history from VnodeA1) e.g.: [{VnodeA1, {1, TS1}}, {VnodeB1, {1, TS2}}, {VnodeB1.1, {1, TS3}}, {VnodeA1.1, {1, TS4}}]

... this object can then loop around indefinitely, forever increasing the size of the vector clock.