Fix transfers may lock shard holder for too long by timvisee · Pull Request #8373 · qdrant/qdrant

timvisee · 2026-03-12T10:40:17Z

Shard transfers may hold a shard holder lock for a very long time, which can cascade into blocking many operations and eventually freezing the whole cluster.

Some side effects we've seen in real clusters include unresponsive nodes, broken consensus and downtime.

Since we've implemented prevent_unoptimized and the update queue this is a lot more likely to happen.

For example, the last batch of a stream records transfer may block here for a very long time because the last batch sends with wait=true, which holds a shard holder read lock for all that time:

qdrant/lib/collection/src/shards/transfer/stream_records.rs

Lines 136 to 163 in f91e217

    
           loop { 
        
               let shard_holder = shard_holder.read().await; 
        
               let Some(replica_set) = shard_holder.get_shard(shard_id) else { 
        
                   // Forward proxy gone?! 
        
                   // That would be a programming error. 
        
                   return Err(CollectionError::service_error(format!( 
        
                       "Shard {shard_id} is not found" 
        
                   ))); 
        
               }; 
        
               let result = replica_set 
        
                   .transfer_batch(offset, TRANSFER_BATCH_SIZE, None, merge_points) 
        
                   .await?; 
        
               offset = result.next_page_offset; 
        
               progress.lock().add(result.count); 
        
               #[cfg(feature = "staging")] 
        
               if let Some(delay) = staging_delay { 
        
                   tokio::time::sleep(delay).await; 
        
               } 
        
               // If this is the last batch, finalize 
        
               if offset.is_none() { 
        
                   break; 
        
               } 
        
           }

All Submissions:

Contributions should target the dev branch. Did you create your branch from dev?
Have you followed the guidelines in our Contributing document?
Have you checked to ensure there aren't other open Pull Requests for the same update/change?

New Feature Submissions:

Does your submission pass tests?
Have you formatted your code locally using cargo +nightly fmt --all command prior to submission?
Have you checked your code using cargo clippy --workspace --all-features command?

Changes to Core Features:

Have you added an explanation of what your changes do and why you'd like us to include them?
Have you written new tests for your core changes, as applicable?
Have you successfully ran tests with your changes locally?

timvisee · 2026-03-12T12:48:28Z

Fix opened in #8374

…8374) * [ai] Detach shard holder lock from sending update batch * Fix clippy warning * Move stream records delay up to sleep in the middle of the batch * remove unused code --------- Co-authored-by: Andrey Vasnetsov <[email protected]>

timvisee · 2026-03-13T08:11:18Z

@agourlay @generall Same as #8374, but including the test.

* Add basic test that shows shard holder is locked for too long * Rework test, move into stream records test * Fix typos * Don't lock shard holder for a long time in stream records transfer (#8374) * [ai] Detach shard holder lock from sending update batch * Fix clippy warning * Move stream records delay up to sleep in the middle of the batch * remove unused code --------- Co-authored-by: Andrey Vasnetsov <[email protected]> --------- Co-authored-by: Andrey Vasnetsov <[email protected]>

Add basic test that shows shard holder is locked for too long

86b7322

timvisee mentioned this pull request Mar 12, 2026

Don't lock shard holder for a long time in stream records transfer #8374

Merged

6 tasks

timvisee added 2 commits March 12, 2026 15:22

Rework test, move into stream records test

0dd90c2

Fix typos

83457e2

timvisee added the release:1.17.1 label Mar 12, 2026

timvisee requested review from agourlay, coszio, generall and tellet-q March 13, 2026 08:10

timvisee changed the title ~~Draft: transfers may lock shard holder for too long~~ Fix transfers may lock shard holder for too long Mar 13, 2026

timvisee marked this pull request as ready for review March 13, 2026 08:11

generall approved these changes Mar 13, 2026

View reviewed changes

This comment was marked as resolved.

Sign in to view

agourlay approved these changes Mar 13, 2026

View reviewed changes

qdrant deleted a comment from coderabbitai Bot Mar 13, 2026

timvisee merged commit 01a56ef into dev Mar 13, 2026
15 checks passed

timvisee deleted the fix-transfer-locking-shard-holder branch March 13, 2026 08:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix transfers may lock shard holder for too long#8373

Fix transfers may lock shard holder for too long#8373
timvisee merged 4 commits into
devfrom
fix-transfer-locking-shard-holder

timvisee commented Mar 12, 2026 •

edited

Loading

Uh oh!

timvisee commented Mar 12, 2026

Uh oh!

timvisee commented Mar 13, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	loop {
	let shard_holder = shard_holder.read().await;

	let Some(replica_set) = shard_holder.get_shard(shard_id) else {
	// Forward proxy gone?!
	// That would be a programming error.
	return Err(CollectionError::service_error(format!(
	"Shard {shard_id} is not found"
	)));
	};

	let result = replica_set
	.transfer_batch(offset, TRANSFER_BATCH_SIZE, None, merge_points)
	.await?;

	offset = result.next_page_offset;
	progress.lock().add(result.count);

	#[cfg(feature = "staging")]
	if let Some(delay) = staging_delay {
	tokio::time::sleep(delay).await;
	}

	// If this is the last batch, finalize
	if offset.is_none() {
	break;
	}
	}

Conversation

timvisee commented Mar 12, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

All Submissions:

New Feature Submissions:

Changes to Core Features:

Uh oh!

timvisee commented Mar 12, 2026

Uh oh!

timvisee commented Mar 13, 2026

Uh oh!

This comment was marked as resolved.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

timvisee commented Mar 12, 2026 •

edited

Loading