Skip to content

repair: drop late replies to already answered block id repair requests - #15483

Merged
AshwinSekar merged 1 commit into
anza-xyz:masterfrom
latent-9:repair-nonce-invalidation
Sep 30, 2026
Merged

AshwinSekar merged 1 commit into
anza-xyz:masterfrom
latent-9:repair-nonce-invalidation

Conversation

@latent-9

@latent-9 latent-9 commented Sep 23, 2026 •

Copy link
Copy Markdown

Problem

In block_id_repair_service, a metadata repair request that times out is retried by requeueing it and sending it again with a fresh nonce, but the nonce of the timed-out attempt stays live in outstanding_requests. Because process_block_id_repair_response only checks the nonce, a peer that holds its response and later replies once per still-live nonce passes verification each time, so the same semantic request gets processed twice: FecSetRoot requests are re-queued twice and response stats double count. The redundant work is bounded by the outstanding_requests LRU and the 60s TTL, but the stale nonce was only cleaned up lazily.

The redundant-work framing is from #14688 by @cfkelly18.

Change

Process a reply only if the request is still in sent_requests. A retry sends the request again with a fresh nonce and the nonce of the earlier attempt stays live, so only the first valid reply is processed: a reply to another attempt finds no sent_requests entry and is dropped.

The only gap is a request that timed out and is still waiting in the queue to be resent: it is no longer in sent_requests, so a late reply is dropped, and the cost is one extra round trip (the resent request) rather than duplicate processing.

Dropped late replies are counted under their own response stat.

Test plan

A test covers the late-reply drop: a valid reply to a request that is no longer in sent_requests is dropped, and the late reply stat is incremented. cargo test -p solana-core --lib repair::block_id_repair_service passes (19 tests, run repeatedly), clippy and fmt are green.

@AshwinSekar

Copy link
Copy Markdown

Why do we need to invalidate the nonce? seems better to just not send the additional request again so that we're fine regardless of whether the late reply comes in first or the retry comes in first

@latent-9

latent-9 commented Sep 24, 2026 •

Copy link
Copy Markdown
Author

The retry behavior predates this PR. Removing it would be a separate decision that deserves its own PR and discussion. This fix scopes to one thing: once a request times out and a fresh nonce is sent, the old nonce should not still be accepted. Both approaches fix the double-processing problem, but removing retry also changes fault tolerance behavior for slow peers.

@stablebits

Copy link
Copy Markdown

The retry behavior predates this PR. Removing it would be a separate decision that deserves its own PR and discussion. This fix scopes to one thing: once a request times out and a fresh nonce is sent, the old nonce should not still be accepted. Both approaches fix the double-processing problem, but removing retry also changes fault tolerance behavior for slow peers.

Why not keep the retry and still accept a late reply, instead of relying on responses arriving within 500ms?

sent_requests tracks requests that were actually sent, right? And nonces are stored (and cleaned up) separately in outstanding_requests. So what if we change the meaning of sent_requests for metadata requests (likely introduce a dedicated table separate from shreds) from "was this request sent" to "is a response to this request wanted"? Then we'd accept a response only if a matching entry exists in the table, which also solves the double-processing problem.

Instead of (request, timestamp), the table would store (request, Option<timestamp>), and entries are no longer removed on timeout/retry:

  • (request, None): a response is wanted, and the request is queued to be (re)sent.
  • (request, Some(t)): a response is wanted, and the request was last sent at t.

On queue (first time):

  • insert (request, None). This has to happen at queue time rather than send time.

On send, check the table:

  • no entry: already answered or dropped by the root cleanup -> skip
  • Some(t): already in flight -> skip (this also drops duplicate queue copies)
  • None: send it and set to Some(now)

On timeout:

  • set Some(t) back to None and requeue
  • don't touch the old nonces, so a late reply to an earlier attempt still counts

On reply:

  • remove the entry and process the reply only if it existed. The first valid reply wins, and later ones are dropped.

Extra cleanup:

  • since entries are no longer removed on timeout, we also need something like retain(|r, _| r.slot() > root) for requests that never get a valid reply.

This only applies to metadata requests, so as mentioned, perhaps a dedicated table would be cleaner 🤔

@latent-9

Copy link
Copy Markdown
Author

That makes sense, accepting late replies and deduplicating at the table level is more correct than killing the nonce. Happy to refactor toward the dedicated table if that's the preferred direction, let me know and I'll update.

@latent-9

Copy link
Copy Markdown
Author

Refactored to the dedicated table in 8cd4345, exactly as described: entries are inserted at queue time as (request, None), the send path skips when the request is already answered (no entry) or in flight (Some(t), which also drops duplicate queue copies), timeouts flip the entry back to None and requeue without touching the old nonces, and the reply is processed only if the entry existed, so the first valid reply wins and later ones are dropped. Added the root-cleanup retain for requests that never get a valid reply, plus tests for the late-reply-still-counts and first-reply-wins paths. Nonce invalidation plumbing is gone.

@stablebits stablebits left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@stablebits

stablebits commented Sep 28, 2026 •

Copy link
Copy Markdown

@latent-9 @AshwinSekar Now that the changes are getting bigger, let's step back for a moment 🤔

On the one hand, the table gives more explicit tracking of metadata requests. On the other hand, the scope is much bigger. This duplicate-processing issue isn't severe. So if we want to address it, perhaps a simpler fix gets most of the benefit. We could process a reply only if the request is still in sent_requests.

The only gap is a request that timed out and is still waiting in the queue to be resent. It's no longer in the sent_requests, so a late reply is dropped. But the cost is one extra round trip (the new request is sent), not duplicate processing. Assuming such cases are rare (can we assume this?), the change is minimal:

         debug!("{my_pubkey}: Received valid response for request {request:?}");

-        // Remove from sent_requests since we got a response
-        state
+        // A retry sends the request again with a fresh nonce, and the nonce of the
+        // earlier attempt stays live. Only the first valid reply is processed: a
+        // reply to another attempt finds no sent_requests entry and is dropped.
+        if state
             .sent_requests
-            .remove(&OutgoingMessage::Metadata(request));
+            .remove(&OutgoingMessage::Metadata(request))
+            .is_none()
+        {
+            debug!("{my_pubkey}: Dropping reply for already answered request {request:?}");
+            return;
+        }

@latent-9

latent-9 commented Sep 28, 2026 •

Copy link
Copy Markdown
Author

Timed-out requests should be rare since responses normally arrive well within 2*DELTA, so the extra round trip in that gap seems like a fair cost. Went with the minimal fix: replies are only processed while the request is still in sent_requests, with a test covering the drop and a stat for it. Pushed in a06647b.

@AshwinSekar

Copy link
Copy Markdown

@latent-9 @AshwinSekar Now that the changes are getting bigger, let's step back for a moment 🤔

On the one hand, the table gives more explicit tracking of metadata requests. On the other hand, the scope is much bigger. This duplicate-processing issue isn't severe. So if we want to address it, perhaps a simpler fix gets most of the benefit. We could process a reply only if the request is still in sent_requests.

The only gap is a request that timed out and is still waiting in the queue to be resent. It's no longer in the sent_requests, so a late reply is dropped. But the cost is one extra round trip (the new request is sent), not duplicate processing. Assuming such cases are rare (can we assume this?), the change is minimal:

         debug!("{my_pubkey}: Received valid response for request {request:?}");

-        // Remove from sent_requests since we got a response
-        state
+        // A retry sends the request again with a fresh nonce, and the nonce of the
+        // earlier attempt stays live. Only the first valid reply is processed: a
+        // reply to another attempt finds no sent_requests entry and is dropped.
+        if state
             .sent_requests
-            .remove(&OutgoingMessage::Metadata(request));
+            .remove(&OutgoingMessage::Metadata(request))
+            .is_none()
+        {
+            debug!("{my_pubkey}: Dropping reply for already answered request {request:?}");
+            return;
+        }

Yeah exactly what i had in mind (see my first message)

We don't care if the first request is the earliest response or if a later retry ends up winning. The key is to just not duplicate the follow up requests if multiple responses end up coming in

@stablebits
stablebits dismissed their stale review September 29, 2026 13:24

requested changes are irrelevant after rework

@alexpyattaev alexpyattaev added the CI Pull Request is ready to enter CI label Sep 29, 2026
@anza-team anza-team removed the CI Pull Request is ready to enter CI label Sep 29, 2026
@codecov-commenter

codecov-commenter commented Sep 29, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82.5%. Comparing base (443ba0e) to head (a06647b).
⚠️ Report is 36 commits behind head on master.

Additional details and impacted files
@@           Coverage Diff           @@
##           master   #15483   +/-   ##
=======================================
  Coverage    82.5%    82.5%           
=======================================
  Files         935      935           
  Lines      372606   372649   +43     
=======================================
+ Hits       307631   307678   +47     
+ Misses      64975    64971    -4     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@AshwinSekar
AshwinSekar added this pull request to the merge queue Sep 30, 2026
Merged via the queue into anza-xyz:master with commit 04539a7 Sep 30, 2026
45 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants