Skip to content

Commit cfc66cb

Browse files
authored
deploy: upgrade GKE boot disks to 50 GB balanced storage (#1739)
Production ingress stalls correlated with high I/O latency and queues on the 20 GB `pd-standard` boot disks, which also serve the ingress cache and container logs. Increase both hosted environments to 50 GB `pd-balanced` boot disks and explicitly retain GKE's surge strategy (`maxSurge: 1`, `maxUnavailable: 0`). Node count and machine type stay the same; database and monitoring persistent volumes keep their existing sizes. **Merge and verify production promotion #1738 before merging this PR.** Merging this PR automatically rolls staging; production needs a subsequent promotion after staging verification. Production deployment uses all infrastructure code at the triggering commit, so reversing this merge order would combine the pod and disk changes. GKE updates the existing node pool by replacing its VMs incrementally. The single registry and Grafana PostgreSQL instances will be interrupted while their separate persistent disks move; surge capacity does not provide database HA. The README documents preparation and verification. Validation: - `make -C deploy build`, `go vet ./...` in `deploy`, and `git diff --check` passed. - Refreshed production Pulumi preview at `1048543` succeeded. The node-pool diff contains only `nodeConfig.diskSizeGb` and `nodeConfig.diskType`; no Pulumi resource deletions or replacements and no persistent-volume/database changes. The rest of the plan is the pending #1738 Deployment updates, two automatically cleaned-up ingress admission Jobs, and existing Grafana empty-rule normalization. Re-preview after #1738 deploys before production promotion. - Both projects have regional CPU, SSD-disk, and address quota headroom for a surge node; actual zonal capacity is only established during provisioning. - Both production PostgreSQL clusters are healthy with one instance and PDBs disabled. Their disks use the GCE persistent-disk CSI driver. The production registry backup last succeeded at 2026-10-25 04:47:13 UTC; check freshness again at rollout. No infrastructure has been applied from this branch. After rollout, verify disk configuration, workload readiness/placement and API smoke checks, then compare disk queues, ingress restarts and request latency during busy traffic. Related to #1252; this change does not establish that the latency target has been met. [GKE surge upgrade behavior](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/node-pool-upgrade-strategies)
1 parent 4e2df3f commit cfc66cb

2 files changed

Lines changed: 31 additions & 11 deletions

File tree

‎deploy/README.md‎

Lines changed: 23 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -130,15 +130,29 @@ ingress probe failures/restarts, node CPU pressure, and request latency at
130130
comparable traffic over the following day, including a busy period. Application
131131
metrics omit ingress cache hits, so include ingress logs in the latency check.
132132
133-
Keep the node disk change in a subsequent rollout. The current 20 GB
134-
`pd-standard` boot disks also hold the ingress cache and container logs. The
135-
next step for the observed disk queues is larger `pd-balanced` boot disks,
136-
starting with 50 GB, using GKE surge upgrades with one extra node and zero
137-
unavailable nodes. A disk-type change recreates nodes; it is not an in-place
138-
filesystem change. Preview the actual Pulumi plan before applying it, verify a
139-
recent successful database backup, and account for a possible interruption
140-
while the single PostgreSQL instance moves. Complete the pod rollout before
141-
starting this node rollout. No node-pool change is included in the pod repair.
133+
### Node boot disks
134+
135+
Hosted node pools use 50 GB `pd-balanced` boot disks for the ingress cache and
136+
container logs. This replaces the 20 GB `pd-standard` disks that showed sustained
137+
I/O queues. GKE surge upgrades explicitly allow one extra node and zero
138+
unavailable nodes. Changing disk type or size rolls the nodes; the PostgreSQL
139+
and monitoring persistent volumes are separate and keep their existing sizes.
140+
141+
Complete the pod rollout above before promoting the disk change to production.
142+
Merging the disk change rolls staging first; production requires a separate
143+
promotion. Preview the actual Pulumi plan before each rollout and confirm a
144+
node-pool update rather than resource replacement. Check quota and capacity for
145+
the extra node, verify a recent successful database backup, and schedule for a
146+
database interruption: the single registry and Grafana PostgreSQL instances
147+
must stop and reattach their persistent disks on another node. Surge capacity
148+
does not make these databases highly available. Monitoring workloads can also
149+
be interrupted while moving.
150+
151+
After the rollout, verify both nodes use 50 GB `pd-balanced` disks, all workloads
152+
are ready, and the two registry and production ingress replicas are spread
153+
across nodes. Check the registry health, listing, and search endpoints. Compare
154+
boot-disk latency/queues, ingress restarts, and request latency during a busy
155+
period before concluding that the disk upgrade resolved the observed stalls.
142156
143157
## Server-name search index
144158

‎deploy/pkg/providers/gcp/provider.go‎

Lines changed: 8 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -252,8 +252,14 @@ func (p *Provider) CreateCluster(ctx *pulumi.Context, environment string) (*prov
252252
NodeCount: pulumi.Int(2),
253253
NodeConfig: &container.NodePoolNodeConfigArgs{
254254
MachineType: pulumi.String("e2-standard-2"),
255-
DiskSizeGb: pulumi.Int(20),
256-
DiskType: pulumi.String("pd-standard"),
255+
// Boot disks also serve the ingress cache and container logs.
256+
DiskSizeGb: pulumi.Int(50),
257+
DiskType: pulumi.String("pd-balanced"),
258+
},
259+
UpgradeSettings: &container.NodePoolUpgradeSettingsArgs{
260+
Strategy: pulumi.String("SURGE"),
261+
MaxSurge: pulumi.Int(1),
262+
MaxUnavailable: pulumi.Int(0),
257263
},
258264

259265
// Node management configuration

0 commit comments

Comments
 (0)