Skip to content

Commit d6f66d2

Browse files
kvapsclaude
authored andcommitted
fix(vertical-pod-autoscaler): stop the updater OOM + VPA-for-VPA eviction churn
Two root causes of e2e install flakiness, both in the VPA package: 1. The updater memory limit was 110Mi. The updater watches every pod in the cluster, so on a full cozystack install it OOM-kills, which trips the Deployment's ProgressDeadlineExceeded and makes Flux fail the HelmRelease 'early due to stalled resources' - cascading NotReady to monitoring, velero, etcd-operator and everything downstream. Raise the limit to 512Mi (request 200Mi); the recommender is already unlimited. 2. The two VPA-for-VPA VerticalPodAutoscaler objects ran updateMode: Auto, which evicts the VPA recommender(s) during install and adds pod churn on an already busy cluster. Switch to Initial, matching the existing monitoring/etcd VPA->Initial fixes. Co-Authored-By: Claude <[email protected]> Signed-off-by: Andrei Kvapil <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 433203c commit d6f66d2

3 files changed

Lines changed: 18 additions & 10 deletions

File tree

‎packages/system/vertical-pod-autoscaler/templates/vpa-for-vpa.yaml‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -69,7 +69,10 @@ spec:
6969
kind: Deployment
7070
name: {{ dig "vertical-pod-autoscaler" "nameOverride" "" .Values.AsMap | default "vertical-pod-autoscaler" | trunc 63 | trimSuffix "-" }}-recommender
7171
updatePolicy:
72-
updateMode: Recreate
72+
# Initial (not Recreate): size at pod creation without ever evicting.
73+
# Recreate here evicts the VPA recommender during install, adding pod churn
74+
# on an already busy cluster. Matches the monitoring/etcd VPA->Initial fixes.
75+
updateMode: Initial
7376
---
7477
apiVersion: autoscaling.k8s.io/v1
7578
kind: VerticalPodAutoscaler
@@ -82,5 +85,5 @@ spec:
8285
kind: Deployment
8386
name: vpa-for-vpa-recommender
8487
updatePolicy:
85-
updateMode: Recreate
88+
updateMode: Initial
8689
{{- end }}

‎packages/system/vertical-pod-autoscaler/tests/vpa_test.yaml‎

Lines changed: 6 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -29,11 +29,11 @@ tests:
2929
- hasDocuments:
3030
count: 0
3131

32-
# updateMode Auto is deprecated since VPA 1.4 (an alias for Recreate that will
33-
# be removed in a future API version). Pin the vpa-for-vpa
34-
# VerticalPodAutoscalers to the explicit, non-deprecated Recreate so the bump
35-
# does not leave a deprecated value behind.
36-
- it: pins the vpa-for-vpa VerticalPodAutoscalers to updateMode Recreate
32+
# The vpa-for-vpa VerticalPodAutoscalers use updateMode Initial: they size the
33+
# VPA components at pod creation without ever evicting them. Recreate (and its
34+
# deprecated Auto alias) would evict the recommender mid-install, adding pod
35+
# churn on an already busy cluster. Pin Initial so a bump cannot regress it.
36+
- it: pins the vpa-for-vpa VerticalPodAutoscalers to updateMode Initial
3737
template: templates/vpa-for-vpa.yaml
3838
documentSelector:
3939
path: kind
@@ -42,4 +42,4 @@ tests:
4242
asserts:
4343
- equal:
4444
path: spec.updatePolicy.updateMode
45-
value: Recreate
45+
value: Initial

‎packages/system/vertical-pod-autoscaler/values.yaml‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -11,11 +11,16 @@ vertical-pod-autoscaler:
1111

1212
updater:
1313
resources:
14+
# The updater watches every pod in the cluster; a 110Mi limit OOM-kills it
15+
# on a full cozystack install, which trips ProgressDeadlineExceeded and
16+
# makes Flux fail the HelmRelease (cascading to monitoring/velero/etc).
17+
# Give it real headroom; the recommender (which holds more state) is
18+
# already unlimited.
1419
limits:
15-
memory: 110Mi
20+
memory: 512Mi
1621
requests:
1722
cpu: 200m
18-
memory: 110Mi
23+
memory: 200Mi
1924

2025
recommender:
2126
extraArgs:

0 commit comments

Comments
 (0)