Skip to content

fix: reload the pod-identity CA pool for client-cert verification - #1930

Merged
Taahir Ahmed (ahmedtd) merged 1 commit into
agent-substrate:mainfrom
rashadism:ateapi-reload-pod-identity-ca
Oct 6, 2026
Merged

Taahir Ahmed (ahmedtd) merged 1 commit into
agent-substrate:mainfrom
rashadism:ateapi-reload-pod-identity-ca

Conversation

@rashadism

@rashadism Rashad Sirajudeen (rashadism) commented Sep 28, 2026 •

Copy link
Copy Markdown
Contributor

buildServerCreds read --pod-identity-ca-certs into an x509.CertPool once at startup and froze it into the gRPC server's tls.Config, so a pod-identity CA rotation was never picked up without restarting ateapi. GetCertificate for the server's own certificate already reloads per handshake via credbundle.Loader; ClientCAs did not.

This uses credbundle.PoolLoader — already implemented and already used by the credential-provider server — via GetConfigForClient, so the trust pool is re-read (and re-parsed only when the file actually changes) on every handshake, matching the reload behavior the server-cert side already had.

Split buildServerCreds into a thin wrapper plus a testable buildServerTLSConfig taking explicit paths, and added coverage proving a CA rotation on disk is picked up without a restart.

Related: #1931

Test plan

  • go build ./cmd/ateapi/...
  • go vet ./cmd/ateapi/...
  • go test ./cmd/ateapi/...
  • gofmt -l clean

@google-cla

google-cla Bot commented Sep 28, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

ateapi's buildServerCreds read --pod-identity-ca-certs into an
x509.CertPool once at startup and froze it into the gRPC server's
tls.Config, so a pod-identity CA rotation was never picked up without
restarting ateapi. GetCertificate for the server's own certificate
already reloaded per handshake via credbundle.Loader; ClientCAs did
not.

Use credbundle.PoolLoader, already implemented and used by the
credential-provider server, via GetConfigForClient so the trust pool
is re-read (and re-parsed only when the file actually changes) on
every handshake.

Split buildServerCreds into a thin wrapper plus a testable
buildServerTLSConfig taking explicit paths, and add coverage proving
a CA rotation on disk is picked up without a restart.

Signed-off-by: Rashad Sirajudeen <[email protected]>
@ahmedtd
Taahir Ahmed (ahmedtd) added this pull request to the merge queue Oct 5, 2026
Merged via the queue into agent-substrate:main with commit f1adf1e Oct 6, 2026
10 checks passed
a4-a4s1 Bot pushed a commit to AlexBulankou/substrate that referenced this pull request Oct 6, 2026
…e atelet dialer (agent-substrate#1932)

Three components load their peer CA pool once at startup and never
refresh it, so a CA rotation breaks them until a restart, same as agent-substrate#1931
describes:

- atelet's own gRPC server verifying client certs (ClientCAs).
- ate-controller's client verifying ateapi's server cert (RootCAs).
- a worker Pod's client verifying its node-local atelet, used by
capacity reporting, actor suspend requests, and atunnel's egress
credential path.

No new machinery. Each fix reloads the pool through
credbundle.PoolLoader, which already exists in this codebase and is
already used for the equivalent ateapi server case (agent-substrate#1930). On the
client side, the reload goes through a VerifyConnection callback with
InsecureSkipVerify: true, the same pattern internal/ateclient/dial.go
already uses for its own client path.

Also tested on a live cluster: with these three fixes deployed, a
worker's capacity reporting recovered from failing every attempt to
succeeding immediately, and a stuck golden-snapshot build that had been
failing on every retry completed within about a minute.

## Test plan
- go build ./..., go vet, go test all pass
- gofmt -l clean on every touched file
- each fix has a regression test that reproduces the failure against the
old one-shot-load code and passes against the fix

---------

Signed-off-by: Rashad Sirajudeen <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/security Security related issue/pr kind/bug Something isn't working / bugfixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants