Repository navigation
On GCS, snapshot transfers run as a dedicated service account and their tokens no longer run short near the end of each hour - #3093
Merged
Benjamin Elder (BenTheElder) merged 4 commits intoOct 22, 2026
Conversation
ILLUM1N0X
requested review from
Taahir Ahmed (ahmedtd) and
Michelle Au (msau42)
October 21, 2026 21:46
…e account The credential exchange downscoped ate-api-server's own token, so GCS audit logs named ate-api-server for every node's snapshot transfer, and the GKE metadata server's cached token left downscoped tokens short for the last twenty minutes of each hour. The control sidecar now impersonates the service account --transfer-service-account names through the IAM Credentials API, and fetches a new token once less than 40 minutes are left, so every downscoped token starts with at least that long. While a fetch fails, it keeps using the last token until that expires.
ate-setup takes the account's name from SNAPSHOT_TRANSFER_GSA_NAME, builds its email in gcp.projectID, writes it to the ate-api-server ConfigMap, and the manifest passes it to the control sidecar's --transfer-service-account. A GKE deploy of ate-api-server refuses to run without it. kind stores snapshots in S3 and does not use it.
ILLUM1N0X
force-pushed
the
snapshot-transfer-account
branch
from
October 21, 2026 22:48
5158770 to
ca9795d
Compare
create iam and bootstrap create ate-snapshot-transfer, let ate-api-server's Workload Identity principal create tokens for it, and grant it object admin and bucket viewer on the snapshot bucket. create cloudsql gives its own account the same token creator grant, since ate-api-server's pod runs as that account once Cloud SQL is on. enable and bootstrap turn on the IAM and IAM Credentials APIs. teardown.sh deletes the account.
ILLUM1N0X
force-pushed
the
snapshot-transfer-account
branch
from
October 21, 2026 23:06
ca9795d to
8b87d67
Compare
setup-gcp retries the token creator grant, since a service account created moments earlier can be rejected as missing. deploy ate-api-server checks for the transfer account before it touches the cluster, and kind ignores SNAPSHOT_TRANSFER_GSA_NAME from the environment. The exchange falls back to the last source token only while it outlasts the STS call.
Michelle Au (msau42)
approved these changes
Oct 21, 2026
This was referenced Oct 25, 2026
Steven Shriver (stshrive)
added a commit
to stshrive/substrate-gke
that referenced
this pull request
Oct 26, 2026
…e/substrate Pulls in the dedicated snapshot transfer service account (`ate-snapshot-transfer`) that ate-api-server's credential exchange impersonates from [Agent Substrate PR 3093](agent-substrate/substrate#3093). From Substrate v0.6.0, ate-setup refuses to deploy ate-api-server on GKE unless SNAPSHOT_TRANSFER_GSA_NAME names this account, so v0.6.0 cannot be installed without it. Differences from upstream: - The default name is a constant in setup-gcp (DefaultTransferGSAName), since this module cannot import upstream's internal/installdefaults. - The service account policy is read at version 3 and written back with its conditional bindings (ai-on-gke#55), and the IAM client uses the test fakes' client options. createCloudSQLGSA uses the shared addServiceAccountBinding. - bucketGrant is keyed by member. atelet's snapshot bucket roles stay behind --atelet-snapshot-access. The bucket policy retry, which waits for a new service account to propagate, uses the version 3 handle. - teardown.sh goes through gcloud_step and bucket_gone, so a missing account or binding counts as already deleted and other failures are reported. cleanup-gcp's text names the account. - Installer: SNAPSHOT_TRANSFER_GSA_NAME=ate-snapshot-transfer is in the shared environment, so bootstrap creates the account under the name ate-setup is given. Trees before v0.6.0 ignore it. The bootstrap checklist and dry-run lines have the new step. The project step's permission sample adds iam.serviceAccounts.create. The upgrade hand-over exports the variable and runs `create iam` before the new release is deployed. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Steven Shriver (stshrive)
added a commit
to stshrive/substrate-gke
that referenced
this pull request
Oct 26, 2026
…e/substrate Pulls in the dedicated snapshot transfer service account (`ate-snapshot-transfer`) that ate-api-server's credential exchange impersonates from [Agent Substrate PR 3093](agent-substrate/substrate#3093). From Substrate v0.6.0, ate-setup refuses to deploy ate-api-server on GKE unless SNAPSHOT_TRANSFER_GSA_NAME names this account, so v0.6.0 cannot be installed without it. Differences from upstream: - The default name is a constant in setup-gcp (DefaultTransferGSAName), since this module cannot import upstream's internal/installdefaults. - The service account policy is read at version 3 and written back with its conditional bindings (ai-on-gke#55), and the IAM client uses the test fakes' client options. createCloudSQLGSA uses the shared addServiceAccountBinding. - bucketGrant is keyed by member. atelet's snapshot bucket roles stay behind --atelet-snapshot-access. The bucket policy retry, which waits for a new service account to propagate, uses the version 3 handle. - teardown.sh goes through gcloud_step and bucket_gone, so a missing account or binding counts as already deleted and other failures are reported. cleanup-gcp's text names the account. - Installer: SNAPSHOT_TRANSFER_GSA_NAME=ate-snapshot-transfer is in the shared environment, so bootstrap creates the account under the name ate-setup is given. Trees before v0.6.0 ignore it. The bootstrap checklist and dry-run lines have the new step. The project step's permission sample adds iam.serviceAccounts.create. The upgrade hand-over exports the variable and runs `create iam` before the new release is deployed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #2722
The credential exchange downscoped ate-api-server's own token, so GCS audit logs named ate-api-server for every node's snapshot transfer, and the metadata server's cached token left downscoped tokens short for the last twenty minutes of each hour. The exchange now impersonates a dedicated Google service account,
ate-snapshot-transferby default, and fetches a new token once less than 40 minutes are left. setup-gcp creates the account and its grants, and ate-setup passes it to the exchange.Breaking change
GKE installs must run
go run ./tools/setup-gcp create iamand setSNAPSHOT_TRANSFER_GSA_NAMEbefore deploying. ate-setup refuses to deploy ate-api-server on GKE without it. With Cloud SQL, also reruncreate cloudsql, so its account may create the transfer account's tokens.Tested on a GKE cluster: the full e2e suite passed on gVisor workers, a micro-VM actor suspended and resumed, and GCS audit logs named the transfer account for every node snapshot transfer.
AI use: AI-assisted, I reviewed and tested it.