The riak_kv_overflow_queue has been added as part of this enhancement to the reaper/eraser queues.
Currently the realtime replication queue is in memory, and of a limited fix size. Change this to use the overflow queue, to allow for:
- larger backlogs;
- backlogs to persist across reboots.
The reason why originally this was simply an in-memory queue, was that full-sync will always fill the gaps. Although full-sync is now cheaper than key-listing, especially when gaps are small, it is still a slow operation.
If there are o(100K) or greater gaps, full-sync is going to be more time consuming and expensive than draining from a queue, if such a queue exists.
In implementing this, the queue should persist across backlogs but start empty. There should be an admin command to empty a previous real-time queue on the node, so that replaying such a persisted backlog is an operator choice.
The
riak_kv_overflow_queuehas been added as part of this enhancement to the reaper/eraser queues.Currently the realtime replication queue is in memory, and of a limited fix size. Change this to use the overflow queue, to allow for:
The reason why originally this was simply an in-memory queue, was that full-sync will always fill the gaps. Although full-sync is now cheaper than key-listing, especially when gaps are small, it is still a slow operation.
If there are o(100K) or greater gaps, full-sync is going to be more time consuming and expensive than draining from a queue, if such a queue exists.
In implementing this, the queue should persist across backlogs but start empty. There should be an admin command to empty a previous real-time queue on the node, so that replaying such a persisted backlog is an operator choice.