Repository navigation
fix: reload frozen CA pools in atelet, ateapiauth, and the worker-side atelet dialer - #1932
Merged
Taahir Ahmed (ahmedtd) merged 3 commits intoOct 6, 2026
Conversation
Maya Wang (mayawang)
added
area/security
Both atelet's gRPC server and atunnel's ingress server verify client certificates against the pod-identity trust bundle projected at /run/podidentity.podcert.ate.dev/trust-bundle.pem, but each read it once at startup into a tls.Config.ClientCAs pool that then stayed fixed for the life of the process. A CA rotation on disk was invisible until the next restart, causing in-cluster peers presenting certificates from the new CA to fail verification with "certificate signed by unknown authority". Reload the pool per handshake through GetConfigForClient, using the existing credbundle.PoolLoader, which stats the file and only re-parses it when it changes. Signed-off-by: Rashad Sirajudeen <[email protected]>
ate-controller's ateapi client (internal/ateapiauth) loaded the CA trusted to verify the ateapi server's certificate once at startup into a fixed tls.Config.RootCAs pool. A rotation of that CA left long-lived clients unable to verify the server's current certificate, failing with "certificate signed by unknown authority" until the client process restarted. Verify the server certificate in VerifyConnection against a pool reloaded through credbundle.PoolLoader instead, mirroring the same reload-on-change pattern internal/ateclient already uses for the equivalent client path. Signed-off-by: Rashad Sirajudeen <[email protected]>
internal/ateletdial builds the TLS config a worker Pod uses to reach the atelet on its own node, verifying atelet's certificate against a CA loaded once at construction into a fixed pool. A rotation of that CA left every caller — capacity reporting, actor suspend requests, and atunnel's credential path — unable to verify atelet's current certificate, failing with "certificate signed by unknown authority" until the worker Pod restarted. A worker that can never report its capacity never gets scheduled onto, indistinguishably from having none. Verify the certificate against a pool reloaded through credbundle.PoolLoader instead, mirroring the same pattern already fixed elsewhere in this codebase. Signed-off-by: Rashad Sirajudeen <[email protected]>
Rashad Sirajudeen (rashadism)
force-pushed
the
reload-frozen-ca-pools
branch
from
October 3, 2026 16:45
dff9771 to
0ca67e3
Compare
Taahir Ahmed (ahmedtd)
approved these changes
Oct 5, 2026
Collaborator
|
If you have the bandwidth, consider a followup PR to centralize the TLS config setup, similar to https://github.com/ahmedtd/tinycert/blob/main/lib/spiffefsd/spiffefsd.go#L29 (I wrote that code, so it's fine to copy verbatim and reassign the copyright to Google / Substrate Authors). |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three components load their peer CA pool once at startup and never refresh it, so a CA rotation breaks them until a restart, same as #1931 describes:
No new machinery. Each fix reloads the pool through credbundle.PoolLoader, which already exists in this codebase and is already used for the equivalent ateapi server case (#1930). On the client side, the reload goes through a VerifyConnection callback with InsecureSkipVerify: true, the same pattern internal/ateclient/dial.go already uses for its own client path.
Also tested on a live cluster: with these three fixes deployed, a worker's capacity reporting recovered from failing every attempt to succeeding immediately, and a stuck golden-snapshot build that had been failing on every retry completed within about a minute.
Test plan