Repository navigation
Commit 5b2a1a3
authored
fix(deploy): reduce ingress restarts under load (#1737)
Production ingress repeatedly exceeded its one-second health-check
deadline under load. Both registry replicas were co-located with
PostgreSQL despite soft spreading; 12 of 13 ingress restarts on that
node followed near-saturated CPU. A second node also showed disk
pressure, which needs a separate node rollout.
Enforce per-revision spreading for registry and ingress in
staging/production, using `pod-template-hash` so old replicas cannot
mask placement of new ones. Reserve 300m CPU per ingress and 250m per
registry replica, retaining unlimited CPU bursts. Set ingress
readiness/liveness timeouts to 3s/5s and explicitly retain a one-pod
surge with zero unavailable replicas. Local development keeps soft
registry spreading and 100m CPU requests.
Document staging-first verification and a separate subsequent upgrade
from the 20 GB standard boot disks to 50 GB balanced disks. This PR
changes no node pool or production promotion config and does not close
#1252.
Validation: `make -C deploy build`, `go vet ./...` in deploy, and `git
diff --check` pass. Executed the actual deployment constructors with
Pulumi mocks and rendered pinned ingress chart 4.13.0 for
local/staging/prod; verified selectors, revision spreading, CPU
requests, merged probe defaults, and rollout policies. Both staging and
production API servers accepted the generated Deployments with `kubectl
apply --dry-run=server`. A CPU scheduling model of the current
production placement found no deadlocks across 70 reachable states,
assuming ready replacements and unchanged background requests; final
requests are approximately 1490m/1489m of 1930m per node.
Production `pulumi preview --stack gcpProd --refresh --diff` completed
successfully for commit 5bac183. It plans the two intended Deployment
updates, two existing Grafana RBAC resources changing only `rules: null`
to `rules: []` (no permission change), and recreation of the two
existing ingress admission setup Jobs with `ttlSecondsAfterFinished: 0`.
There are no node, database, or backup-resource changes and no deletions
or replacements. The non-refresh preview showed the same four updates
without the cleaned-up Jobs. No infrastructure update has been applied;
verify staging before production rollout.1 parent df677de commit 5b2a1a3
3 files changed
Lines changed: 98 additions & 17 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
104 | 104 | | |
105 | 105 | | |
106 | 106 | | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
107 | 143 | | |
108 | 144 | | |
109 | 145 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
51 | 73 | | |
52 | 74 | | |
53 | 75 | | |
| |||
59 | 81 | | |
60 | 82 | | |
61 | 83 | | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
62 | 103 | | |
63 | 104 | | |
64 | 105 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
41 | 41 | | |
42 | 42 | | |
43 | 43 | | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
44 | 63 | | |
45 | 64 | | |
46 | 65 | | |
| |||
96 | 115 | | |
97 | 116 | | |
98 | 117 | | |
99 | | - | |
100 | | - | |
101 | | - | |
102 | | - | |
103 | | - | |
104 | | - | |
105 | | - | |
106 | | - | |
107 | | - | |
108 | | - | |
109 | | - | |
110 | | - | |
111 | | - | |
112 | | - | |
113 | | - | |
114 | | - | |
| 118 | + | |
115 | 119 | | |
116 | 120 | | |
117 | 121 | | |
| |||
234 | 238 | | |
235 | 239 | | |
236 | 240 | | |
237 | | - | |
| 241 | + | |
238 | 242 | | |
239 | 243 | | |
240 | 244 | | |
| |||
0 commit comments