Mas i1817 overflowrtq#1829
Merged
Merged
Conversation
Also don't log queues on a crash - avoid over-sized crash dumps
Have the queue backed to disk, so that beyond a certain size it overflows from memory to disk (and once the on disk part is consumed from the queue the files are removed).
Risk of misconfiguration leading to wiping of wrong data. Also starting a job may lead to the main process having its disk_log wiped.
Avoid enoent errors
Check for a soft overload on any vnode before reaping. This will add some delay - but reap has been show to potentially overload a cluster ... availability is more important than reap speed.
There is no catch for {error, mailbox_overload} should it occur - it should not as the mailbox check should prevent it. If it does, the reaper will crash (and restart without any reaps) - return to a safe known position.
The queue file generated, are still in UUID format, but the id now incorporates creation date. This should make it easier to detect and clean any garbage that might be accrued. One use case where '++' is used has also been removed (although a lists:flatten/1 was still required at the end in this case)
Refactor riak_kv_replrtq_src to use the riak_kv_overflow_queue, and allow for larger queues to be supported without leading to memory exhaustion. Issues still with eqc test. Also issue of inaccurate accounting of maximum queue size, and setting sizes (and their defaults) at startup. The popping from rtq is also a bit ugly, would be nice to have some cleaner code for this.
… false If ObjectLimit == 0, then this might not be the case
This now passes 10K eqc tests of replrtq_eqc. Stat of discards added to riak_kv_stats
There will be no attempt to re-open overflow on startup. The queue is persisted to avoid over-consumption of memory, not so that replication requests can be persisted across reboots
Default them all to the same value for simplicity, and name the configuration item consistently (although there is a subtle difference in the implementation for replrtq)
Needs to change size limit stored on state. Also remove previous (and now redundant) configuration: replrtq_srcqueuelimit now replaced with replrtq_overflow_limit
Contributor
Author
yorkshire-steve
approved these changes
May 12, 2022
| {datatype, directory} | ||
| ]}. | ||
|
|
||
| %% @doc A path under which the reaper overload queue will be stored. |
There was a problem hiding this comment.
Should this doc string be updated to indicate this location is used for all queues?
Contributor
Author
There was a problem hiding this comment.
There are separate locations for each queue, which you can't really see from the diff:
%% @doc A path under which the eraser overload queue will be stored.
{mapping, "eraser_dataroot", "riak_kv.eraser_dataroot", [
{default, "$(platform_data_dir)/kv_eraser"},
{datatype, directory}
]}.
%% @doc A path under which the reaper overload queue will be stored.
{mapping, "reaper_dataroot", "riak_kv.reaper_dataroot", [
{default, "$(platform_data_dir)/kv_reaper"},
{datatype, directory}
]}.
%% @doc A path under which the reader overload queue will be stored.
{mapping, "reader_dataroot", "riak_kv.reader_dataroot", [
{default, "$(platform_data_dir)/kv_reader"},
{datatype, directory}
]}.
%% @doc A path under which the repl real-time overload queue will be stored.
{mapping, "replrtq_dataroot", "riak_kv.replrtq_dataroot", [
{default, "$(platform_data_dir)/kv_replrtqsrc"},
{datatype, directory}
]}.
martinsumner
added a commit
that referenced
this pull request
May 31, 2022
hmmr
pushed a commit
to TI-Tokyo/basho-riak_kv
that referenced
this pull request
Nov 23, 2022
Expand on use of riak_kv_overflow_queue so that it is used by the riak_kv_replrtq_src, as well as riak_kv_reaper and riak_kv_eraser. This means that larger queue sizes can be supported for riak_kv_replrtq_src without having to worry about compromising the memory of the node. This should allow for repl_keys_range AAE folds to generate very large replication sets, without clogging the node worker pool by pausing so that real-time replication can keep up. The overflow queues are deleted on shutdown (if there is a queue on disk). The feature is to allow for larger queues without memory exhaustion, persistence is not used to persist queues across restarts. Overflow Queues extended to include a 'reader' queue which may be used for read_repairs. Currently this queue is only used for the repair_keys_range query and the read-repair trigger. # Conflicts: # priv/riak_kv.schema # src/riak_kv_overflow_queue.erl # src/riak_kv_replrtq_src.erl
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Expand on use of riak_kv_overflow_queue so that it is used by the riak_kv_replrtq_src, as well as riak_kv_reaper and riak_kv_eraser.
This means that larger queue sizes can be supported for riak_kv_replrtq_src without having to worry about compromising the memory of the node. This should allow for repl_keys_range AAE folds to generate very large replication sets, without clogging the node worker pool by pausing so that real-time replication can keep up.
The overflow queues are deleted on shutdown (if there is a queue on disk). The feature is to allow for larger queues without memory exhaustion, persistence is not used to persist queues across restarts.
Overflow Queues extended to include a 'reader' queue which may be used for read_repairs. Currently this queue is only used for the repair_keys_range query and the read-repair trigger.