Repository navigation
Commit cfc66cb
authored
deploy: upgrade GKE boot disks to 50 GB balanced storage (#1739)
Production ingress stalls correlated with high I/O latency and queues on
the 20 GB `pd-standard` boot disks, which also serve the ingress cache
and container logs. Increase both hosted environments to 50 GB
`pd-balanced` boot disks and explicitly retain GKE's surge strategy
(`maxSurge: 1`, `maxUnavailable: 0`). Node count and machine type stay
the same; database and monitoring persistent volumes keep their existing
sizes.
**Merge and verify production promotion #1738 before merging this PR.**
Merging this PR automatically rolls staging; production needs a
subsequent promotion after staging verification. Production deployment
uses all infrastructure code at the triggering commit, so reversing this
merge order would combine the pod and disk changes. GKE updates the
existing node pool by replacing its VMs incrementally. The single
registry and Grafana PostgreSQL instances will be interrupted while
their separate persistent disks move; surge capacity does not provide
database HA. The README documents preparation and verification.
Validation:
- `make -C deploy build`, `go vet ./...` in `deploy`, and `git diff
--check` passed.
- Refreshed production Pulumi preview at `1048543` succeeded. The
node-pool diff contains only `nodeConfig.diskSizeGb` and
`nodeConfig.diskType`; no Pulumi resource deletions or replacements and
no persistent-volume/database changes. The rest of the plan is the
pending #1738 Deployment updates, two automatically cleaned-up ingress
admission Jobs, and existing Grafana empty-rule normalization.
Re-preview after #1738 deploys before production promotion.
- Both projects have regional CPU, SSD-disk, and address quota headroom
for a surge node; actual zonal capacity is only established during
provisioning.
- Both production PostgreSQL clusters are healthy with one instance and
PDBs disabled. Their disks use the GCE persistent-disk CSI driver. The
production registry backup last succeeded at 2026-10-25 04:47:13 UTC;
check freshness again at rollout.
No infrastructure has been applied from this branch. After rollout,
verify disk configuration, workload readiness/placement and API smoke
checks, then compare disk queues, ingress restarts and request latency
during busy traffic. Related to #1252; this change does not establish
that the latency target has been met.
[GKE surge upgrade
behavior](https://docs.cloud.google.com/kubernetes-engine/docs/concepts/node-pool-upgrade-strategies)1 parent 4e2df3f commit cfc66cb
2 files changed
Lines changed: 31 additions & 11 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
130 | 130 | | |
131 | 131 | | |
132 | 132 | | |
133 | | - | |
134 | | - | |
135 | | - | |
136 | | - | |
137 | | - | |
138 | | - | |
139 | | - | |
140 | | - | |
141 | | - | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
142 | 156 | | |
143 | 157 | | |
144 | 158 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
252 | 252 | | |
253 | 253 | | |
254 | 254 | | |
255 | | - | |
256 | | - | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
257 | 263 | | |
258 | 264 | | |
259 | 265 | | |
| |||
0 commit comments