Skip to content

fix: re-route client batches to child shards after shard split#970

Merged
mattisonchao merged 2 commits into
oxia-db:mainfrom
merlimat:fix-client-shard-reroute
Mar 24, 2026
Merged

fix: re-route client batches to child shards after shard split#970
mattisonchao merged 2 commits into
oxia-db:mainfrom
merlimat:fix-client-shard-reroute

Conversation

@merlimat

@merlimat merlimat commented Mar 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • When a shard is deleted (e.g. after a split), pending write/read batches targeting that shard now detect the deletion and re-route operations to the correct child shards
  • Added Exists(shardId) to the shard manager interface so batches can check if their target shard still exists
  • Write and read batches check shard existence in the retry loop; if the shard is gone, operations are re-submitted through the normal routing pipeline (which re-hashes keys to the correct child shards)
  • Previously, batches would retry with exponential backoff for up to 30s per wave, causing ~70s of client unavailability after a split

Test plan

  • Unit tests for write batch shard rerouting (TestWriteBatch_RerouteOnShardDeletion)
  • Unit tests for read batch shard rerouting (TestReadBatch_RerouteOnShardDeletion)
  • All existing client tests pass
  • CI green

After a shard split, pending write/read batches for the deleted parent
shard would retry with exponential backoff for up to 30s per batch,
causing ~70s of client unavailability.

The fix detects when a batch's target shard no longer exists in the
shard manager (checked before each retry attempt). When detected, the
batch stops retrying and re-submits each operation through the normal
pipeline, which re-hashes keys and routes them to the correct child
shards. This reduces client recovery time from ~70s to sub-second.

Changes:
- Add Exists(shardId) to ShardManager interface
- Add ErrShardNotFound sentinel error
- Write/read batch retry loops check shard existence before each retry
- On ErrShardNotFound, batches invoke a reroute callback that
  re-submits operations to the batch manager (which routes to correct
  child shards based on key hash)
- Add PutCall.PartitionKeyOrKey() for correct shard routing of puts
  with explicit partition keys

Signed-off-by: Matteo Merli <[email protected]>
@merlimat
merlimat force-pushed the fix-client-shard-reroute branch from 87d6b23 to d148bb3 Compare March 24, 2026 02:11
Add tests verifying that write and read batches correctly detect when
their target shard is deleted (e.g. after a split) and re-route
operations to child shards:

- TestWriteBatchRerouteOnShardDeleted: verifies write batch detects
  shard deletion and invokes reroute callback with all operations
- TestWriteBatchNoRerouteWhenShardExists: verifies normal retry
  behavior when shard still exists (no false reroutes)
- TestReadBatchRerouteOnShardDeleted: same pattern for read batches

Also rephrase log message from "Shard deleted" to "Shard was
split/merged" to avoid implying data loss.

Signed-off-by: Matteo Merli <[email protected]>
@merlimat
merlimat force-pushed the fix-client-shard-reroute branch from d148bb3 to 263a695 Compare March 24, 2026 04:02
@mattisonchao
mattisonchao merged commit 2f56840 into oxia-db:main Mar 24, 2026
5 checks passed
@merlimat
merlimat deleted the fix-client-shard-reroute branch March 24, 2026 04:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants