Skip to content

Commit 9c18260

Browse files
authored
fix(e2e): stop the install pipeline flaking (VPA OOM/churn + wait budgets) (#2957)
## What The e2e install pipeline still flakes on a loaded runner, failing at a different point each run. Root-caused via repeated runs and a live SSH breakpoint; the platform itself is fine (the components do converge), the failures are slowness + self-inflicted churn under resource pressure. ### Fixes 1. **VPA updater OOM** — the `vertical-pod-autoscaler-updater` memory limit was `110Mi`. The updater watches every pod in the cluster, so on a full install it OOM-kills → `ProgressDeadlineExceeded` → Flux fails the HelmRelease "early due to stalled resources", cascading NotReady to monitoring, velero, etcd-operator and everything downstream. Raised to `512Mi` (request `200Mi`). 2. **VPA-for-VPA eviction churn** — the two VPA-for-VPA `VerticalPodAutoscaler` objects ran `updateMode: Auto`, evicting the VPA recommender(s) during install and adding pod churn on an already busy cluster. Switched to `Initial`, matching the existing monitoring/etcd `VPA->Initial` fixes. 3. **Install wait budgets** — components reach readiness but overshoot tight waits on a busy runner: keycloak HR chain `10m→20m`, tenant aggregate wait `10m→20m`, per-app tenant readiness waits `5m→10m`. (The crust-gather failure-snapshot timeout that kept whole jobs hanging to the 3h cancel is already on `main`.) ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Increased several installation and readiness wait times to better handle slower environment startup and reduce premature failures. * Updated vertical autoscaling behavior to avoid unnecessary pod restarts during deployment. * Raised resource limits for the autoscaling updater to improve reliability under load. * **Tests** * Aligned automated checks with the new autoscaling behavior and longer readiness expectations. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents ec42c52 + d6f66d2 commit 9c18260

4 files changed

Lines changed: 31 additions & 20 deletions

File tree

‎hack/e2e-install-cozystack.bats‎

Lines changed: 13 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -193,37 +193,40 @@ EOF
193193
# failure messages without redundant separate waits. seaweedfs now
194194
# installs as a serial chain seaweedfs-db (CNPG bootstrap) ->
195195
# seaweedfs-system (master raft quorum) -> seaweedfs wrapper, which
196-
# pushes the parent's Ready flip to ~5-6 min; tenant-root HR.spec.timeout
197-
# is 15m and this 10m wait stays inside it.
196+
# pushes the parent's Ready flip to ~5-6 min on an idle runner. On a loaded
197+
# runner the tenant stack only starts creating pods ~9-10 min in, so the
198+
# parent's Ready can land past the HR's single 15m timeout window; the HR
199+
# re-reconciles every 1m until it converges, so this wait is 20m to observe
200+
# that eventual Ready rather than expiring first.
198201
kubectl wait hr/etcd hr/ingress hr/monitoring hr/seaweedfs hr/tenant-root \
199-
-n tenant-root --timeout=10m --for=condition=ready
202+
-n tenant-root --timeout=20m --for=condition=ready
200203

201204

202205
# Expose Cozystack services through ingress
203206
kubectl patch package cozystack.cozystack-platform --type merge -p '{"spec":{"components":{"platform":{"values":{"publishing":{"exposedServices":["api","dashboard","cdi-uploadproxy","vm-exportproxy","keycloak"]}}}}}}'
204207

205208
# NGINX ingress controller
206209
timeout 60 sh -ec 'until kubectl get deploy root-ingress-controller -n tenant-root >/dev/null 2>&1; do sleep 1; done'
207-
kubectl wait deploy/root-ingress-controller -n tenant-root --timeout=5m --for=condition=available
210+
kubectl wait deploy/root-ingress-controller -n tenant-root --timeout=10m --for=condition=available
208211

209212
# etcd statefulset
210213
timeout 60 sh -ec 'until kubectl get sts/etcd -n tenant-root >/dev/null 2>&1; do sleep 2; done'
211-
kubectl wait sts/etcd -n tenant-root --for=jsonpath='{.status.readyReplicas}'=3 --timeout=5m
214+
kubectl wait sts/etcd -n tenant-root --for=jsonpath='{.status.readyReplicas}'=3 --timeout=10m
212215

213216
# VictoriaMetrics components
214217
timeout 60 sh -ec 'until kubectl get vmalert/vmalert-shortterm -n tenant-root >/dev/null 2>&1; do sleep 2; done'
215218
timeout 60 sh -ec 'until kubectl get vmalertmanager/alertmanager -n tenant-root >/dev/null 2>&1; do sleep 2; done'
216219
kubectl wait vmalert/vmalert-shortterm vmalertmanager/alertmanager -n tenant-root --for=jsonpath='{.status.updateStatus}'=operational --timeout=15m
217220
timeout 60 sh -ec 'until kubectl get vlclusters/generic -n tenant-root >/dev/null 2>&1; do sleep 2; done'
218-
kubectl wait vlclusters/generic -n tenant-root --for=jsonpath='{.status.updateStatus}'=operational --timeout=5m
221+
kubectl wait vlclusters/generic -n tenant-root --for=jsonpath='{.status.updateStatus}'=operational --timeout=10m
219222
timeout 60 sh -ec 'until kubectl get vmcluster/shortterm vmcluster/longterm -n tenant-root >/dev/null 2>&1; do sleep 2; done'
220-
kubectl wait vmcluster/shortterm vmcluster/longterm -n tenant-root --for=jsonpath='{.status.updateStatus}'=operational --timeout=5m
223+
kubectl wait vmcluster/shortterm vmcluster/longterm -n tenant-root --for=jsonpath='{.status.updateStatus}'=operational --timeout=10m
221224

222225
# Grafana
223226
timeout 60 sh -ec 'until kubectl get clusters.postgresql.cnpg.io/grafana-db -n tenant-root >/dev/null 2>&1; do sleep 2; done'
224-
kubectl wait clusters.postgresql.cnpg.io/grafana-db -n tenant-root --for=condition=ready --timeout=5m
227+
kubectl wait clusters.postgresql.cnpg.io/grafana-db -n tenant-root --for=condition=ready --timeout=10m
225228
timeout 60 sh -ec 'until kubectl get deploy/grafana-deployment -n tenant-root >/dev/null 2>&1; do sleep 2; done'
226-
kubectl wait deploy/grafana-deployment -n tenant-root --for=condition=available --timeout=5m
229+
kubectl wait deploy/grafana-deployment -n tenant-root --for=condition=available --timeout=10m
227230

228231
# Verify Grafana via ingress
229232
ingress_ip=$(kubectl get svc root-ingress-controller -n tenant-root -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
@@ -243,7 +246,7 @@ EOF
243246
kubectl patch package cozystack.cozystack-platform --type merge -p '{"spec":{"components":{"platform":{"values":{"authentication":{"oidc":{"enabled":true,"keycloakInternalUrl":"http://keycloak-http.cozy-keycloak.svc:8080/realms/cozy"}}}}}}}'
244247

245248
timeout 120 sh -ec 'until kubectl get hr -n cozy-keycloak keycloak keycloak-configure keycloak-operator >/dev/null 2>&1; do sleep 1; done'
246-
kubectl wait hr/keycloak hr/keycloak-configure hr/keycloak-operator -n cozy-keycloak --timeout=10m --for=condition=ready
249+
kubectl wait hr/keycloak hr/keycloak-configure hr/keycloak-operator -n cozy-keycloak --timeout=20m --for=condition=ready
247250
}
248251

249252
@test "Aggregated API rejects Tenant name with dashes" {

‎packages/system/vertical-pod-autoscaler/templates/vpa-for-vpa.yaml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,10 @@ spec:
6969
kind: Deployment
7070
name: {{ dig "vertical-pod-autoscaler" "nameOverride" "" .Values.AsMap | default "vertical-pod-autoscaler" | trunc 63 | trimSuffix "-" }}-recommender
7171
updatePolicy:
72-
updateMode: Recreate
72+
# Initial (not Recreate): size at pod creation without ever evicting.
73+
# Recreate here evicts the VPA recommender during install, adding pod churn
74+
# on an already busy cluster. Matches the monitoring/etcd VPA->Initial fixes.
75+
updateMode: Initial
7376
---
7477
apiVersion: autoscaling.k8s.io/v1
7578
kind: VerticalPodAutoscaler
@@ -82,5 +85,5 @@ spec:
8285
kind: Deployment
8386
name: vpa-for-vpa-recommender
8487
updatePolicy:
85-
updateMode: Recreate
88+
updateMode: Initial
8689
{{- end }}

‎packages/system/vertical-pod-autoscaler/tests/vpa_test.yaml‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -29,11 +29,11 @@ tests:
2929
- hasDocuments:
3030
count: 0
3131

32-
# updateMode Auto is deprecated since VPA 1.4 (an alias for Recreate that will
33-
# be removed in a future API version). Pin the vpa-for-vpa
34-
# VerticalPodAutoscalers to the explicit, non-deprecated Recreate so the bump
35-
# does not leave a deprecated value behind.
36-
- it: pins the vpa-for-vpa VerticalPodAutoscalers to updateMode Recreate
32+
# The vpa-for-vpa VerticalPodAutoscalers use updateMode Initial: they size the
33+
# VPA components at pod creation without ever evicting them. Recreate (and its
34+
# deprecated Auto alias) would evict the recommender mid-install, adding pod
35+
# churn on an already busy cluster. Pin Initial so a bump cannot regress it.
36+
- it: pins the vpa-for-vpa VerticalPodAutoscalers to updateMode Initial
3737
template: templates/vpa-for-vpa.yaml
3838
documentSelector:
3939
path: kind
@@ -42,4 +42,4 @@ tests:
4242
asserts:
4343
- equal:
4444
path: spec.updatePolicy.updateMode
45-
value: Recreate
45+
value: Initial

‎packages/system/vertical-pod-autoscaler/values.yaml‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,11 +11,16 @@ vertical-pod-autoscaler:
1111

1212
updater:
1313
resources:
14+
# The updater watches every pod in the cluster; a 110Mi limit OOM-kills it
15+
# on a full cozystack install, which trips ProgressDeadlineExceeded and
16+
# makes Flux fail the HelmRelease (cascading to monitoring/velero/etc).
17+
# Give it real headroom; the recommender (which holds more state) is
18+
# already unlimited.
1419
limits:
15-
memory: 110Mi
20+
memory: 512Mi
1621
requests:
1722
cpu: 200m
18-
memory: 110Mi
23+
memory: 200Mi
1924

2025
recommender:
2126
extraArgs:

0 commit comments

Comments
 (0)