Repository navigation
podcertcontroller: do not rate-limit PCRs assigned to other replicas - #2353
Merged
Taahir Ahmed (ahmedtd) merged 3 commits intoOct 9, 2026
Merged
Conversation
Every replica receives every PodCertificateRequest, but only the replica chosen by rendezvous hashing should sign it. The others returned ErrNotAssigned and requeued with AddRateLimited, which treats a normal condition as a failure: it consumes the shared failure rate limiter and grows the per-item exponential backoff (up to 1000s), slowing down retries of genuinely assigned PCRs and delaying takeover when a replica dies. Requeue unassigned PCRs with AddAfter instead, using the rendezvous lease duration as the recheck interval since assignments cannot change faster than that.
Taahir Ahmed (ahmedtd)
previously approved these changes
Oct 8, 2026
Collaborator
Author
|
Thanks Taahir Ahmed (@ahmedtd) for the quick review. I have fixed the comment. Please review it again. |
Taahir Ahmed (ahmedtd)
approved these changes
Oct 9, 2026
github-merge-queue
Bot
removed this pull request from the merge queue due to failed status checks
Oct 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every replica receives every PodCertificateRequest, but only the replica chosen by rendezvous hashing should sign it. The others return
ErrNotAssignedand were requeued withAddRateLimited, which treats a normal condition as a failure:This slows retries of genuinely assigned PCRs and delays takeover when a replica dies (the PCRs it owned may already be deep in backoff on the surviving replicas).
Unassigned PCRs are now requeued with
AddAfter(rendezvous.LeaseDuration). Assignments only change when a replica joins or its lease expires, so re-checking more often gains nothing.LeaseDurationis exported fromrendezvousfor this purpose; its value is unchanged.A follow-up could make this event-driven (re-enqueue pending PCRs when the active replica set changes) instead of polling.
Testing
go test ./cmd/podcertcontroller/...