Repository navigation
repair: drop late replies to already answered block id repair requests - #15483
Conversation
|
Why do we need to invalidate the nonce? seems better to just not send the additional request again so that we're fine regardless of whether the late reply comes in first or the retry comes in first |
|
The retry behavior predates this PR. Removing it would be a separate decision that deserves its own PR and discussion. This fix scopes to one thing: once a request times out and a fresh nonce is sent, the old nonce should not still be accepted. Both approaches fix the double-processing problem, but removing retry also changes fault tolerance behavior for slow peers. |
Why not keep the retry and still accept a late reply, instead of relying on responses arriving within 500ms?
Instead of
On queue (first time):
On send, check the table:
On timeout:
On reply:
Extra cleanup:
This only applies to metadata requests, so as mentioned, perhaps a dedicated table would be cleaner 🤔 |
|
That makes sense, accepting late replies and deduplicating at the table level is more correct than killing the nonce. Happy to refactor toward the dedicated table if that's the preferred direction, let me know and I'll update. |
|
Refactored to the dedicated table in 8cd4345, exactly as described: entries are inserted at queue time as |
|
@latent-9 @AshwinSekar Now that the changes are getting bigger, let's step back for a moment 🤔 On the one hand, the table gives more explicit tracking of metadata requests. On the other hand, the scope is much bigger. This duplicate-processing issue isn't severe. So if we want to address it, perhaps a simpler fix gets most of the benefit. We could process a reply only if the request is still in The only gap is a request that timed out and is still waiting in the queue to be resent. It's no longer in the debug!("{my_pubkey}: Received valid response for request {request:?}");
- // Remove from sent_requests since we got a response
- state
+ // A retry sends the request again with a fresh nonce, and the nonce of the
+ // earlier attempt stays live. Only the first valid reply is processed: a
+ // reply to another attempt finds no sent_requests entry and is dropped.
+ if state
.sent_requests
- .remove(&OutgoingMessage::Metadata(request));
+ .remove(&OutgoingMessage::Metadata(request))
+ .is_none()
+ {
+ debug!("{my_pubkey}: Dropping reply for already answered request {request:?}");
+ return;
+ } |
|
Timed-out requests should be rare since responses normally arrive well within 2*DELTA, so the extra round trip in that gap seems like a fair cost. Went with the minimal fix: replies are only processed while the request is still in |
Yeah exactly what i had in mind (see my first message) We don't care if the first request is the earliest response or if a later retry ends up winning. The key is to just not duplicate the follow up requests if multiple responses end up coming in |
requested changes are irrelevant after rework
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #15483 +/- ##
=======================================
Coverage 82.5% 82.5%
=======================================
Files 935 935
Lines 372606 372649 +43
=======================================
+ Hits 307631 307678 +47
+ Misses 64975 64971 -4 🚀 New features to boost your workflow:
|
Problem
In block_id_repair_service, a metadata repair request that times out is retried by requeueing it and sending it again with a fresh nonce, but the nonce of the timed-out attempt stays live in
outstanding_requests. Becauseprocess_block_id_repair_responseonly checks the nonce, a peer that holds its response and later replies once per still-live nonce passes verification each time, so the same semantic request gets processed twice: FecSetRoot requests are re-queued twice and response stats double count. The redundant work is bounded by the outstanding_requests LRU and the 60s TTL, but the stale nonce was only cleaned up lazily.The redundant-work framing is from #14688 by @cfkelly18.
Change
Process a reply only if the request is still in
sent_requests. A retry sends the request again with a fresh nonce and the nonce of the earlier attempt stays live, so only the first valid reply is processed: a reply to another attempt finds nosent_requestsentry and is dropped.The only gap is a request that timed out and is still waiting in the queue to be resent: it is no longer in
sent_requests, so a late reply is dropped, and the cost is one extra round trip (the resent request) rather than duplicate processing.Dropped late replies are counted under their own response stat.
Test plan
A test covers the late-reply drop: a valid reply to a request that is no longer in
sent_requestsis dropped, and the late reply stat is incremented.cargo test -p solana-core --lib repair::block_id_repair_servicepasses (19 tests, run repeatedly), clippy and fmt are green.