Repository navigation
setup-gcp: say which node pools need replacing after enabling the beta APIs - #1852
Merged
Haven Xia (HavenXia) merged 1 commit intoOct 22, 2026
Conversation
Bowei Du (bowei)
added
kind/bug
Gari Singh (mastersingh24)
force-pushed
the
setup-gcp-node-recycle
branch
from
October 2, 2026 10:52
691dc1d to
fb28575
Compare
Contributor
Author
|
Haven Xia (@HavenXia) Aditya Shantanu (@aditya-shantanu) could one of you approve the CI run for this PR? I rebased onto main, which needed a small README conflict resolution after #1924, so the workflow is waiting on approval for the new commit. The previous run's |
Contributor
Author
|
FYI the installer side of this landed in ai-on-gke/substrate-gke#41: it lists the pools to replace before Substrate starts. Leaving this open for anyone still running setup-gcp from here. |
Comment on lines
+175
to
+176
| // It copies what this tool sets on the pools it creates, so for a cluster | ||
| // this tool made the new pool is equivalent. Anything else customized on the |
Member
|
need rebase |
…a APIs When bootstrap finds an existing cluster without the PodCertificate beta APIs, its reconcile path turns them on in place. That works: the APIs are served once the update finishes. On 1.36 node pools it is only half the repair. Pod certificate projection is a kubelet feature, gated through 1.36, and a kubelet that was already running never picks it up, so pods scheduled onto those nodes fail with "MountVolume.SetUp failed: unimplemented" while the control plane looks healthy. Nothing said so. After a successful beta-API update, warn once per node pool below 1.37 and log the two commands that fix it: create a replacement pool, with the machine type, disk, node count and nested virtualization copied from the old one, then delete the old pool. It is the pool's version that decides this, not the control plane's, since a 1.37 control plane can still run 1.36 nodes. A new pool is the remedy because it is the one GKE operation certain to produce new kubelets. An upgrade to the version a pool already runs looks like the obvious fix, but GKE skips it: the UPGRADE_NODES operation finishes in seconds without replacing any node. Replacing the pool is left to the user rather than done here, because it moves running workloads. Verified end to end on a throwaway 1.36 cluster: this reconcile path enabled the APIs and logged the warning; a pod on the existing node failed with "unimplemented"; the logged create command produced a pool whose kubelet issued a PodCertificateRequest for the same pod. The README warning is rewritten to match, and it drops the claim that enabling the APIs on an existing cluster does not make them served, which did not survive testing. Fixes agent-substrate#1819
Contributor
Author
|
Rebased onto main. Kept the new sysctl check next to this, and dropped the commit that only moved tests around to avoid clashing with #1797, since that one's closed. |
Gari Singh (mastersingh24)
force-pushed
the
setup-gcp-node-recycle
branch
from
October 21, 2026 20:37
fb28575 to
e7d4af6
Compare
Haven Xia (HavenXia)
approved these changes
Oct 22, 2026
Steven Shriver (stshrive)
added a commit
to stshrive/substrate-gke
that referenced
this pull request
Oct 26, 2026
…bstrate Pulls in the warning that names the node pools to replace after setup-gcp turns the beta APIs on for an existing cluster from [Agent Substrate PR 1852](agent-substrate/substrate#1852). The setup-gcp README's beta API warning is upstream's rewrite. Upstream also fixed hack/ate-dev-env.sh.example, which this repository does not have. One addition: the replacement node-pools create command also copies provisioned boot disk IOPS and throughput (ai-on-gke#64), so a replacement for a pool created with them keeps its disk performance. A test case covers it. Co-Authored-By: Claude Opus 5.5 <[email protected]>
Steven Shriver (stshrive)
added a commit
to stshrive/substrate-gke
that referenced
this pull request
Oct 26, 2026
…bstrate Pulls in the warning that names the node pools to replace after setup-gcp turns the beta APIs on for an existing cluster from [Agent Substrate PR 1852](agent-substrate/substrate#1852). The setup-gcp README's beta API warning is upstream's rewrite. Upstream also fixed hack/ate-dev-env.sh.example, which this repository does not have. One addition: the replacement node-pools create command also copies provisioned boot disk IOPS and throughput (ai-on-gke#64), so a replacement for a pool created with them keeps its disk performance. A test case covers it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #1819
When bootstrap finds an existing cluster without the PodCertificate beta APIs, its reconcile path turns them on in place. That works: the APIs are served once the update finishes. On 1.36 node pools it is only half the repair, and nothing said so. Pod certificate projection is a kubelet feature, gated through 1.36, so pods scheduled onto nodes that were already running fail with
MountVolume.SetUp failed: unimplementedwhile the control plane looks healthy.What changes
After a successful beta-API update, setup-gcp warns once per node pool below 1.37 and logs the two commands that fix it:
gcloud container node-pools create <pool>-2 …, with the machine type, disk, node count and nested virtualization copied from the old pool;gcloud container node-pools delete <pool> …for once workloads have moved.The warning also says to carry over anything else customized on the old pool (taints, service account, autoscaling). The pool's version decides this, not the control plane's, since a 1.37 control plane can still run 1.36 nodes.
The README warning is rewritten to match, building on the
v1discovery wording from ClusterTrustBundle v1 support #1924. It drops the claim that enabling the APIs on an existing cluster doesn't make them served, which didn't hold up in testing.hack/ate-dev-env.sh.examplehad the same claim in a comment; fixed there too.Why a new pool, not an in-place node recreation
The issue proposed recreating nodes, and the obvious command is upgrading a pool to the version it already runs. GKE skips that: in testing, the
UPGRADE_NODESoperation finished in 4 seconds and no node was replaced. A pool can only be upgraded forward when GKE happens to offer a newer patch. A new pool is the one GKE operation certain to produce new kubelets.Replacing the pool is left to the user rather than done automatically, because it moves running workloads. A flag to do it could be a follow-up if that would be useful.
Verified end to end on a throwaway 1.36.4 cluster created without the beta APIs:
create clusterreconcile path enabled the APIs and logged the warning fordefault-pool.MountVolume.SetUp failed … unimplemented.node-pools createcommand, pasted as printed, created a pool whose kubelet issued a PodCertificateRequest for the same pod (the mount then waited oncredential bundle is not issued yet, as expected with no signer running).node-pools deletecommand removed the old pool.go test ./tools/setup-gcp/...)