Repository navigation
fix(deploy): reduce ingress restarts under load - #1737
Merged
Merged
Conversation
rdimitrov
marked this pull request as ready for review
October 25, 2026 22:37
rdimitrov
added a commit
that referenced
this pull request
Oct 25, 2026
Promote the infrastructure changes from #1737 through the existing production workflow while retaining registry application image `1.8.4`. The comment-only change to `deploy/Pulumi.gcpProd.yaml` deliberately triggers that workflow on merge; it does not change application configuration or encrypted secrets. Staging deployment of merge commit `5b2a1a3` succeeded: https://github.com/modelcontextprotocol/registry/actions/runs/40241181207. Verified both registry replicas are available on different nodes with 250m CPU requests, ingress is available with a 300m request and 3s/5s readiness/liveness timeouts, and per-revision spreading is applied. All new containers had zero restarts. Health, list, and name-search requests returned HTTP 200 in approximately 0.6 seconds. Staging has one ingress replica, so production's two-replica placement still needs post-deployment verification; these smoke checks do not establish peak-load performance. Production preview at this promotion commit succeeded: two intended Deployment updates plus existing Grafana empty-rule normalization. The refreshed preview of the identical infrastructure code also showed recreation of the two automatically cleaned-up ingress admission setup Jobs. No node, database, or backup changes, deletions, or replacements are planned. After deployment, verify both production Deployments and node placement, then compare probe failures/restarts and ingress request latency at comparable traffic during a busy period. The disk upgrade remains a separate rollout. Related: #1252.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Production ingress repeatedly exceeded its one-second health-check deadline under load. Both registry replicas were co-located with PostgreSQL despite soft spreading; 12 of 13 ingress restarts on that node followed near-saturated CPU. A second node also showed disk pressure, which needs a separate node rollout.
Enforce per-revision spreading for registry and ingress in staging/production, using
pod-template-hashso old replicas cannot mask placement of new ones. Reserve 300m CPU per ingress and 250m per registry replica, retaining unlimited CPU bursts. Set ingress readiness/liveness timeouts to 3s/5s and explicitly retain a one-pod surge with zero unavailable replicas. Local development keeps soft registry spreading and 100m CPU requests.Document staging-first verification and a separate subsequent upgrade from the 20 GB standard boot disks to 50 GB balanced disks. This PR changes no node pool or production promotion config and does not close #1252.
Validation:
make -C deploy build,go vet ./...in deploy, andgit diff --checkpass. Executed the actual deployment constructors with Pulumi mocks and rendered pinned ingress chart 4.13.0 for local/staging/prod; verified selectors, revision spreading, CPU requests, merged probe defaults, and rollout policies. Both staging and production API servers accepted the generated Deployments withkubectl apply --dry-run=server. A CPU scheduling model of the current production placement found no deadlocks across 70 reachable states, assuming ready replacements and unchanged background requests; final requests are approximately 1490m/1489m of 1930m per node.Production
pulumi preview --stack gcpProd --refresh --diffcompleted successfully for commit 5bac183. It plans the two intended Deployment updates, two existing Grafana RBAC resources changing onlyrules: nulltorules: [](no permission change), and recreation of the two existing ingress admission setup Jobs withttlSecondsAfterFinished: 0. There are no node, database, or backup-resource changes and no deletions or replacements. The non-refresh preview showed the same four updates without the cleaned-up Jobs. No infrastructure update has been applied; verify staging before production rollout.