fix: re-route client batches to child shards after shard split#970
Merged
Conversation
merlimat
requested review from
RobertIndie,
coderzc and
mattisonchao
as code owners
March 24, 2026 00:02
merlimat
force-pushed
the
fix-client-shard-reroute
branch
from
March 24, 2026 00:52
3660ede to
87d6b23
Compare
After a shard split, pending write/read batches for the deleted parent shard would retry with exponential backoff for up to 30s per batch, causing ~70s of client unavailability. The fix detects when a batch's target shard no longer exists in the shard manager (checked before each retry attempt). When detected, the batch stops retrying and re-submits each operation through the normal pipeline, which re-hashes keys and routes them to the correct child shards. This reduces client recovery time from ~70s to sub-second. Changes: - Add Exists(shardId) to ShardManager interface - Add ErrShardNotFound sentinel error - Write/read batch retry loops check shard existence before each retry - On ErrShardNotFound, batches invoke a reroute callback that re-submits operations to the batch manager (which routes to correct child shards based on key hash) - Add PutCall.PartitionKeyOrKey() for correct shard routing of puts with explicit partition keys Signed-off-by: Matteo Merli <[email protected]>
merlimat
force-pushed
the
fix-client-shard-reroute
branch
from
March 24, 2026 02:11
87d6b23 to
d148bb3
Compare
Add tests verifying that write and read batches correctly detect when their target shard is deleted (e.g. after a split) and re-route operations to child shards: - TestWriteBatchRerouteOnShardDeleted: verifies write batch detects shard deletion and invokes reroute callback with all operations - TestWriteBatchNoRerouteWhenShardExists: verifies normal retry behavior when shard still exists (no false reroutes) - TestReadBatchRerouteOnShardDeleted: same pattern for read batches Also rephrase log message from "Shard deleted" to "Shard was split/merged" to avoid implying data loss. Signed-off-by: Matteo Merli <[email protected]>
merlimat
force-pushed
the
fix-client-shard-reroute
branch
from
March 24, 2026 04:02
d148bb3 to
263a695
Compare
mattisonchao
approved these changes
Mar 24, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Exists(shardId)to the shard manager interface so batches can check if their target shard still existsTest plan
TestWriteBatch_RerouteOnShardDeletion)TestReadBatch_RerouteOnShardDeletion)