Repository navigation
deploy: upgrade GKE boot disks to 50 GB balanced storage - #1739
Merged
Merged
Conversation
rdimitrov
added a commit
that referenced
this pull request
Oct 25, 2026
Promote the 50 GB `pd-balanced` node boot disks from #1739 through the existing production workflow while retaining registry application image `1.8.4`. This comment-only change to `deploy/Pulumi.gcpProd.yaml` deliberately triggers that workflow on merge. The two-node pool retains `e2-standard-2` machines and uses surge upgrades with one extra node and zero unavailable nodes. Staging rollout succeeded: https://github.com/modelcontextprotocol/registry/actions/runs/40243009855. Verified both original nodes were replaced with Ready nodes whose actual boot disks are 50 GB `pd-balanced`. All four original persistent volumes retain their identities, volume handles and sizes, and both PostgreSQL clusters recovered. Every active workload is ready; the two registry replicas are on different nodes, and registry/ingress containers have zero restarts. Health, listing and name-search checks returned HTTP 200 in about 0.6 seconds; a fresh unique search also returned HTTP 200. **Allow for a multi-minute database interruption when merging.** During staging replacement, registry PostgreSQL was unready from 23:16:45 to 23:20:29 UTC on October 25 (3m44s), including its configured 180-second smart-shutdown timeout before disk reattachment/startup. This is an observed staging duration, not a production guarantee. Grafana PostgreSQL also moved. The single staging ingress replica caused three failed health samples during its move; production has two ingress replicas but still only one instance of each database. `/health` does not query PostgreSQL, so database readiness was checked separately. Surge capacity does not provide database HA. Refreshed production Pulumi preview at `c71e488`, after #1738 deployed, passed: node-pool boot-disk size/type update, registry deploy-commit annotation update, recreation of two automatically cleaned-up ingress admission Jobs, and existing Grafana empty-rule normalization. No Pulumi resource deletions/replacements or database/persistent-volume changes. GKE nevertheless replaces the node VMs incrementally. All PR CI checks passed. Check the latest successful registry backup when merging; it last succeeded at 2026-10-25 04:47:13 UTC during preparation. Both projects have surge quota headroom; actual zonal capacity is established during provisioning. After production rollout, verify both nodes have 50 GB balanced boot disks, all original persistent volumes are retained, both databases recover, and workloads/API smoke checks pass. Compare disk queues, ingress restarts and request latency during busy traffic before concluding that #1252 is resolved.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Production ingress stalls correlated with high I/O latency and queues on the 20 GB
pd-standardboot disks, which also serve the ingress cache and container logs. Increase both hosted environments to 50 GBpd-balancedboot disks and explicitly retain GKE's surge strategy (maxSurge: 1,maxUnavailable: 0). Node count and machine type stay the same; database and monitoring persistent volumes keep their existing sizes.Merge and verify production promotion #1738 before merging this PR. Merging this PR automatically rolls staging; production needs a subsequent promotion after staging verification. Production deployment uses all infrastructure code at the triggering commit, so reversing this merge order would combine the pod and disk changes. GKE updates the existing node pool by replacing its VMs incrementally. The single registry and Grafana PostgreSQL instances will be interrupted while their separate persistent disks move; surge capacity does not provide database HA. The README documents preparation and verification.
Validation:
make -C deploy build,go vet ./...indeploy, andgit diff --checkpassed.1048543succeeded. The node-pool diff contains onlynodeConfig.diskSizeGbandnodeConfig.diskType; no Pulumi resource deletions or replacements and no persistent-volume/database changes. The rest of the plan is the pending deploy: promote ingress stability fixes to production #1738 Deployment updates, two automatically cleaned-up ingress admission Jobs, and existing Grafana empty-rule normalization. Re-preview after deploy: promote ingress stability fixes to production #1738 deploys before production promotion.No infrastructure has been applied from this branch. After rollout, verify disk configuration, workload readiness/placement and API smoke checks, then compare disk queues, ingress restarts and request latency during busy traffic. Related to #1252; this change does not establish that the latency target has been met.
GKE surge upgrade behavior