Skip to content

Authenticate bundled PostgreSQL logins with separate pod certificates - #2266

Merged
Eitan Yarmush (EItanya) merged 3 commits into
agent-substrate:mainfrom
iplay88keys:iplay88keys/postgres-client-cert-auth
Oct 8, 2026
Merged

Eitan Yarmush (EItanya) merged 3 commits into
agent-substrate:mainfrom
iplay88keys:iplay88keys/postgres-client-cert-auth

Conversation

@iplay88keys

@iplay88keys Jeremy Alvis (iplay88keys) commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Part of: #2097
Part of: #2071

Bundled PostgreSQL currently accepts any client certificate signed by the pod-identity CA without binding it to the requested database username. Require certificate authentication with a common name matching the login, using a dedicated PostgreSQL client CA.

  • Add a pod-certificate signer restricted to ate-system pods using the ate-api-server service account. It issues only the two permitted identities: substrate_owner_user and substrate_readwrite_user.
  • Project separate certificates for the owner and runtime connections. Configure passwordless connection strings with server certificate verification, and retain the existing owner/runtime role grants.
  • Have ate-setup provision the CA and configure bundled PostgreSQL automatically, including on an existing installation. Disable passwords on the bundled application users and keep administrator access local to the PostgreSQL pod.
  • Reuse existing rotation support: new database connections from ateapi reread projected credentials and trust roots, while the bundled PostgreSQL TLS-reloader detects server-certificate or client-trust-bundle changes every 60 seconds and requests a configuration reload.

External PostgreSQL and Cloud SQL continue to use their configured connection strings and authentication settings. Both application certificates are available to the API-server process; separate credentials preserve database role separation but do not isolate the roles from a compromised API server.

Validation

  • Passed go test -race ./cmd/podcertcontroller/... ./cmd/ate-setup/... ./cmd/ateapi/internal/store/atepg ./pkg/postgressetup ./internal/installdefaults after rebasing onto updated main.
  • Signer tests cover both allowed identities, unauthorized requests, pod identity checks, certificate lifetimes, and trust bundles.
  • Docker integration tests use the bundled PostgreSQL configuration and verify successful passwordless connections, bootstrap/migrations, and runtime operations. They reject mismatched usernames, untrusted certificates, missing certificates, remote administrator access, plaintext connections, and runtime access to owner privileges.

Issue owner and read/write certificates only to API-server pods and require each certificate common name to match its database login. Replace bundled database passwords with projected credentials while preserving separate role grants.

@maxsmythe Max Smythe (maxsmythe) left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@maxsmythe

Copy link
Copy Markdown
Collaborator

I'll defer to Eitan and Taahir for merge

@EItanya

Copy link
Copy Markdown
Collaborator

LGTM as well, I think should leave final to Taahir Ahmed (@ahmedtd)

// Each bundled connection presents a certificate for its own login user.
func bundledPostgresDSN(user string) string {
bundle := "/run/postgres.podcert.ate.dev/" + user + ".pem"
return fmt.Sprintf("postgresql://%[email protected]:5432/atepg?sslmode=verify-full&sslrootcert=/run/servicedns.podcert.ate.dev/trust-bundle.pem&sslcert=%s&sslkey=%s", user, bundle, bundle)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to investigate if the postgres client properly handles when the cert and key rotate (the default lifetime is 24 hours, but it can be lowered to 10 minutes for testing this).

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a live integration test that replaces the client cert bundles, server cert, and trust roots. It runs the tls-reload script and forces fresh connections.

I see that queries are succeeding with the replacements, cached TLS configurations are rejected, and an existing connection remains usable.

From what I can tell, the podcertificate API requires a minimum lifetime of 1 hour, not 10 minutes, unfortunately. So this test simulates the projected file replacement rather than waiting for kubelet renewal.

@EItanya
Eitan Yarmush (EItanya) added this pull request to the merge queue Oct 8, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Oct 8, 2026
@EItanya
Eitan Yarmush (EItanya) added this pull request to the merge queue Oct 8, 2026
Merged via the queue into agent-substrate:main with commit df78882 Oct 8, 2026
14 checks passed
@iplay88keys
Jeremy Alvis (iplay88keys) deleted the iplay88keys/postgres-client-cert-auth branch October 8, 2026 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants