Skip to content

Conversation

@ajbeamon
Copy link
Contributor

Add new criteria to DD's GetTeamRequest that allow you to require shards be present on the team and that the team have a minimum free ratio. This can be used to avoid scenarios where the team chosen when processing the request is later rejected by the requestor. Hopefully resolves #1884.

…rds be present on the team and that the team have a minimum free ratio. This avoids scenarios where the team chosen when processing the request is later rejected by the requestor, causing rebalancing movements to get stuck.
bool wantsNewServers;
bool wantsTrueBest;
bool preferLowerUtilization;
bool requiresAssignedData;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would appreciate if a comment can be added to explain requiresAssignedData .
Although I can figure this out by reading this PR, the future me or someone else will take more time to understand this.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'd rather this be clear from the name of the variable than add a comment. I could change it to teamMustHaveShards if that is more obvious, or I'm open to other suggestions.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

teamMustHaveShards is better and good to me.

@xumengpanda
Copy link
Contributor

I'm not sure if this PR wants to improve the visilility of DD or not.
Since it has already touches the "BgDDMountainChopper" and "BgDDValleyFiller", would you mind adding counters in those two actors to tell DBA under which condition those two actors top moving data?

For example, we can count the times when GetTeam does not find a team and the times when chosen or randomTeam does not have healthy space or enough free space.

@xumengpanda
Copy link
Contributor

Also in rebalanceTeams actor, I'm not sure how likely/often the condition if( sourceBytes - destBytes <= 3 * std::max<int64_t>( SERVER_KNOBS->MIN_SHARD_BYTES, metrics.bytes ) || metrics.bytes == 0 ) can be hit.

It seems to me a good place to add a trace so that we know when MountainChopper and ValleyFiller works but reblanceTeams refuses to work.

@ajbeamon
Copy link
Contributor Author

I decided to intentionally defer the work to add extra metrics to another PR.

@ajbeamon ajbeamon merged commit 9e84fa9 into apple:release-6.2 Feb 20, 2020
@ajbeamon ajbeamon deleted the fix-stuck-dd-rebalancing branch February 24, 2020 16:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants