Repository navigation
Commit 9c18260
authored
fix(e2e): stop the install pipeline flaking (VPA OOM/churn + wait budgets) (#2957)
## What
The e2e install pipeline still flakes on a loaded runner, failing at a
different point each run. Root-caused via repeated runs and a live SSH
breakpoint; the platform itself is fine (the components do converge),
the failures are slowness + self-inflicted churn under resource
pressure.
### Fixes
1. **VPA updater OOM** — the `vertical-pod-autoscaler-updater` memory
limit was `110Mi`. The updater watches every pod in the cluster, so on a
full install it OOM-kills → `ProgressDeadlineExceeded` → Flux fails the
HelmRelease "early due to stalled resources", cascading NotReady to
monitoring, velero, etcd-operator and everything downstream. Raised to
`512Mi` (request `200Mi`).
2. **VPA-for-VPA eviction churn** — the two VPA-for-VPA
`VerticalPodAutoscaler` objects ran `updateMode: Auto`, evicting the VPA
recommender(s) during install and adding pod churn on an already busy
cluster. Switched to `Initial`, matching the existing monitoring/etcd
`VPA->Initial` fixes.
3. **Install wait budgets** — components reach readiness but overshoot
tight waits on a busy runner: keycloak HR chain `10m→20m`, tenant
aggregate wait `10m→20m`, per-app tenant readiness waits `5m→10m`.
(The crust-gather failure-snapshot timeout that kept whole jobs hanging
to the 3h cancel is already on `main`.)
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Increased several installation and readiness wait times to better
handle slower environment startup and reduce premature failures.
* Updated vertical autoscaling behavior to avoid unnecessary pod
restarts during deployment.
* Raised resource limits for the autoscaling updater to improve
reliability under load.
* **Tests**
* Aligned automated checks with the new autoscaling behavior and longer
readiness expectations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->4 files changed
Lines changed: 31 additions & 20 deletions
File tree
- hack
- packages/system/vertical-pod-autoscaler
- templates
- tests
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
193 | 193 | | |
194 | 194 | | |
195 | 195 | | |
196 | | - | |
197 | | - | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
198 | 201 | | |
199 | | - | |
| 202 | + | |
200 | 203 | | |
201 | 204 | | |
202 | 205 | | |
203 | 206 | | |
204 | 207 | | |
205 | 208 | | |
206 | 209 | | |
207 | | - | |
| 210 | + | |
208 | 211 | | |
209 | 212 | | |
210 | 213 | | |
211 | | - | |
| 214 | + | |
212 | 215 | | |
213 | 216 | | |
214 | 217 | | |
215 | 218 | | |
216 | 219 | | |
217 | 220 | | |
218 | | - | |
| 221 | + | |
219 | 222 | | |
220 | | - | |
| 223 | + | |
221 | 224 | | |
222 | 225 | | |
223 | 226 | | |
224 | | - | |
| 227 | + | |
225 | 228 | | |
226 | | - | |
| 229 | + | |
227 | 230 | | |
228 | 231 | | |
229 | 232 | | |
| |||
243 | 246 | | |
244 | 247 | | |
245 | 248 | | |
246 | | - | |
| 249 | + | |
247 | 250 | | |
248 | 251 | | |
249 | 252 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
69 | 69 | | |
70 | 70 | | |
71 | 71 | | |
72 | | - | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
73 | 76 | | |
74 | 77 | | |
75 | 78 | | |
| |||
82 | 85 | | |
83 | 86 | | |
84 | 87 | | |
85 | | - | |
| 88 | + | |
86 | 89 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
29 | 29 | | |
30 | 30 | | |
31 | 31 | | |
32 | | - | |
33 | | - | |
34 | | - | |
35 | | - | |
36 | | - | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
37 | 37 | | |
38 | 38 | | |
39 | 39 | | |
| |||
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
45 | | - | |
| 45 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
11 | 11 | | |
12 | 12 | | |
13 | 13 | | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
14 | 19 | | |
15 | | - | |
| 20 | + | |
16 | 21 | | |
17 | 22 | | |
18 | | - | |
| 23 | + | |
19 | 24 | | |
20 | 25 | | |
21 | 26 | | |
| |||
0 commit comments