In recent refactoring of cluster changes - #913 and #967 - the focus has been on behaviour on join operations, and not leave.
Recently a customer achieved a balanced ring through a join (as expected with the new cluster claim algorithm), but then leave plans kept creating unbalanced rings i.e. rings whereby partitions were unevenly distributed leading to the potential for "one slow node" problems.
...
[EDIT] There were various incorrect statements made initially here, about how leave works. See later comments for a more accurate representation of the problem
...
There are perhaps some simple things that can be done:
- Perhaps a configuration option to force rebalance on leave. This can then be enabled for users of location. This perhaps could also be enabled if repeated re-planning is not leading to balanced outcomes.
- The
attempt_simple_transfer/4 could check how many partitions are already owned by. a candidate node, and prefer candidate nodes with lower levels of existing ownership. This should be more likely to give balanced results (although it will do nothing in terms of location awareness.
There be demons here. I don't think there's a lot of existing test coverage of leave scenarios (riak_test/src/rt.erl has a staged_join/1 function but no staged_leave/1). There could be the potential for confusing situations when the configuration setting (1) changes between nodes and between staging and committing changes.
In recent refactoring of cluster changes - #913 and #967 - the focus has been on behaviour on
joinoperations, and notleave.Recently a customer achieved a balanced ring through a join (as expected with the new cluster claim algorithm), but then leave plans kept creating unbalanced rings i.e. rings whereby partitions were unevenly distributed leading to the potential for "one slow node" problems.
...
[EDIT] There were various incorrect statements made initially here, about how leave works. See later comments for a more accurate representation of the problem
...
There are perhaps some simple things that can be done:
attempt_simple_transfer/4could check how many partitions are already owned by. a candidate node, and prefer candidate nodes with lower levels of existing ownership. This should be more likely to give balanced results (although it will do nothing in terms of location awareness.There be demons here. I don't think there's a lot of existing test coverage of leave scenarios (riak_test/src/rt.erl has a
staged_join/1function but nostaged_leave/1). There could be the potential for confusing situations when the configuration setting (1) changes between nodes and between staging and committing changes.