As a result of larger scale performance and operational testing, I found these improvements:

• duplicating other storage server metadata requests that go through load balance, to ensure equal load on SS/TSS
• Not killing a TSS if an individual data move takes too long
• Adding details to traces for, and fixing a bug in TSS recruitment that can cause the cluster to recruit fewer TSS than desired
• Adding the count of wrong_shard_server errors to StorageMetrics, to be able to track any divergence in shards between the SS and TSS and the rate of tss reads not completed due to that divergence

Besides the real cluster testing, passed 250k correctness tests.

Code-Reviewer Section

The general guidelines can be found here.

Please check each of the following things and check all boxes before accepting a PR.

• The PR has a description, explaining both the problem and the solution.
• The description mentions which forms of testing were done and the testing seems reasonable.
• Every function/class/actor that was touched is reasonably well documented.

For Release-Branches

If this PR is made against a release-branch, please also check the following:

• This change/bugfix is a cherry-pick from the next younger branch (younger release-branch or master if this is the youngest branch)
• There is a good reason why this PR needs to go into a release branch and this reason is documented (either in the description above or in a linked GitHub issue)