Have the queue backed to disk, so that beyond a certain size it overflows from memory to disk (and once the on disk part is consumed from the queue the files are removed).

Risk of misconfiguration leading to wiping of wrong data.  Also starting a job may lead to the main process having its disk_log wiped.

Avoid enoent errors

Check for a soft overload on any vnode before reaping.  This will add some delay - but reap has been show to potentially overload a cluster ... availability is more important than reap speed.

There is no catch for {error, mailbox_overload} should it occur - it should not as the mailbox check should prevent it.  If it does, the reaper will crash (and restart without any reaps) - return to a safe known position.

The queue file generated, are still in UUID format, but the id now incorporates creation date.  This should make it easier to detect and clean any garbage that might be accrued.

One use case where '++' is used has also been removed (although a lists:flatten/1 was still required at the end in this case)

Refactor riak_kv_replrtq_src to use the riak_kv_overflow_queue, and allow for larger queues to be supported without leading to memory exhaustion.

Issues still with eqc test.  Also issue of inaccurate accounting of maximum queue size, and setting sizes (and their defaults) at startup.

The popping from rtq is also a bit ugly, would be nice to have some cleaner code for this.

… false

If ObjectLimit == 0, then this might not be the case

This now passes 10K eqc tests of replrtq_eqc.

Stat of discards added to riak_kv_stats

There will be no attempt to re-open overflow on startup.  The queue is persisted to avoid over-consumption of memory, not so that replication requests can be persisted across reboots

Add equivalent to reaper/eraser clear function, which can be used when resetting the size of overflow queues

Default them all to the same value for simplicity, and name the configuration item consistently (although there is a subtle difference in the implementation for replrtq)

Needs to change size limit stored on state.  Also remove previous (and now redundant) configuration: replrtq_srcqueuelimit now replaced with replrtq_overflow_limit

Use the reader queue for repair_keys_range, but also to trigger a read repair when there is key amnesia on a non-coordinating node to resolve the inconsistency without waiting for AAE.

Don't add a corrupted object when rebuilding - instead log, skip past, and prompt read repair.

The behaviour before was to avoid crashing on inserting a corrupted object (whereby the vector clock cannot be extracted for hashing).  Not clear why this is still inserted.  This behaviour is maintained, with the alternative behaviour during the rebuild.

The same handle_corrupted_object/4 fun is then re-used during parallel mode tictac aae rebuilds, with a new version of kv_index_tictactree which catches exceptions and allows an external function to be used for handling them).