Skip to content

Commit a4cb00e

Browse files
committed
fix(monitoring): spread monitoring replicas over nodes
The VMCluster, VLCluster, VTCluster and VMAlertmanager the chart renders carry no affinity and no topology spread, and vm-operator copies both fields onto the pods as written without adding any of its own. Nothing in those specs kept two replicas of one tier off the same node. With replicationFactor 2 both copies of a series can then sit on one node, and on a node-local class the volumes keep them there, so losing that node loses both copies. A StatefulSet pod on a lost node is not replaced elsewhere until its Pod or Node object is deleted, which holds for the Alertmanager replicas without a volume as well. The tiers that run as StatefulSets, vmstorage, vmselect, vlstorage, vtstorage and the Alertmanager, now get a per-node spread with whenUnsatisfiable DoNotSchedule and nodeTaintsPolicy Ignore. The tiers that run as Deployments, vminsert, vlinsert, vlselect, vtinsert and vtselect, get the same spread with ScheduleAnyway, since their pods are replaced on another node without help. A platform that sets scheduling.globalAppTopologySpreadConstraints gets that policy for every tier instead, with the tier's own selector added to each constraint that has none. vm-operator rolls a changed StatefulSet one pod at a time. A replica on a node-local volume whose peer shares its node cannot be placed again under the new rule, so its new pod stays Pending and the rollout of that cluster stops there until the replica is moved. Assisted-by: LLM Signed-off-by: Yan Bondarenko <[email protected]>
1 parent aed7fa8 commit a4cb00e

6 files changed

Lines changed: 382 additions & 0 deletions

File tree

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
{{- define "monitoring.topologySpreadConstraints" -}}
2+
{{- $root := index . 0 -}}
3+
{{- $selector := dict "matchLabels" (dict "app.kubernetes.io/name" (index . 1) "app.kubernetes.io/instance" (index . 2)) -}}
4+
{{- $global := list -}}
5+
{{- with ($root.Values._cluster | default dict).scheduling -}}
6+
{{- with get . "globalAppTopologySpreadConstraints" -}}
7+
{{- $global = dig "topologySpreadConstraints" (list) (fromYaml .) -}}
8+
{{- end -}}
9+
{{- end -}}
10+
{{- $constraints := list -}}
11+
{{- range $global -}}
12+
{{- $constraints = append $constraints (ternary . (merge (dict "labelSelector" $selector) .) (hasKey . "labelSelector")) -}}
13+
{{- end -}}
14+
{{- if not $constraints -}}
15+
{{- $default := dict "maxSkew" 1 "topologyKey" "kubernetes.io/hostname" "whenUnsatisfiable" (index . 3) "labelSelector" $selector -}}
16+
{{- if eq (index . 3) "DoNotSchedule" -}}
17+
{{- $_ := set $default "nodeTaintsPolicy" "Ignore" -}}
18+
{{- end -}}
19+
{{- $constraints = list $default -}}
20+
{{- end -}}
21+
{{- toYaml (dict "topologySpreadConstraints" $constraints) -}}
22+
{{- end -}}

‎packages/system/monitoring/templates/alerta/alerta.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -421,6 +421,7 @@ spec:
421421
apps.cozystack.io/application.name: {{ $.Release.Name }}
422422
replicaCount: {{ include "monitoring.alertmanager.replicas" $ }}
423423
configSecret: alertmanager
424+
{{- include "monitoring.topologySpreadConstraints" (list $ "vmalertmanager" "alertmanager" "DoNotSchedule") | nindent 2 }}
424425
{{- if and $emailEnabled $email.authUsername }}
425426
secrets:
426427
- {{ $emailPasswordSecret | quote }}

‎packages/system/monitoring/templates/vlogs/vlogs.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -22,13 +22,16 @@ spec:
2222
apps.cozystack.io/application.name: {{ $.Release.Name }}
2323
vlinsert:
2424
replicaCount: 2
25+
{{- include "monitoring.topologySpreadConstraints" (list $ "vlinsert" .name "ScheduleAnyway") | nindent 4 }}
2526
resources: {}
2627
vlselect:
2728
replicaCount: 2
29+
{{- include "monitoring.topologySpreadConstraints" (list $ "vlselect" .name "ScheduleAnyway") | nindent 4 }}
2830
resources: {}
2931
vlstorage:
3032
retentionPeriod: {{ .retentionPeriod | quote }}
3133
replicaCount: 2
34+
{{- include "monitoring.topologySpreadConstraints" (list $ "vlstorage" .name "DoNotSchedule") | nindent 4 }}
3235
resources: {}
3336
storage:
3437
volumeClaimTemplate:

‎packages/system/monitoring/templates/vm/vmcluster.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,9 +18,11 @@ spec:
1818
# it's usually more than default 30
1919
maxLabelsPerTimeseries: "60"
2020
replicaCount: 2
21+
{{- include "monitoring.topologySpreadConstraints" (list $ "vminsert" .name "ScheduleAnyway") | nindent 4 }}
2122
resources: {}
2223
vmselect:
2324
replicaCount: 2
25+
{{- include "monitoring.topologySpreadConstraints" (list $ "vmselect" .name "DoNotSchedule") | nindent 4 }}
2426
resources: {}
2527
extraArgs:
2628
search.maxUniqueTimeseries: "600000"
@@ -46,6 +48,7 @@ spec:
4648
storage: 2Gi
4749
vmstorage:
4850
replicaCount: 2
51+
{{- include "monitoring.topologySpreadConstraints" (list $ "vmstorage" .name "DoNotSchedule") | nindent 4 }}
4952
resources: {}
5053
storage:
5154
volumeClaimTemplate:

‎packages/system/monitoring/templates/vtraces/vtraces.yaml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -71,9 +71,11 @@ spec:
7171
apps.cozystack.io/application.name: {{ $.Release.Name }}
7272
insert:
7373
replicaCount: 2
74+
{{- include "monitoring.topologySpreadConstraints" (list $ "vtinsert" .name "ScheduleAnyway") | nindent 4 }}
7475
resources: {}
7576
select:
7677
replicaCount: 2
78+
{{- include "monitoring.topologySpreadConstraints" (list $ "vtselect" .name "ScheduleAnyway") | nindent 4 }}
7779
resources: {}
7880
storage:
7981
retentionPeriod: {{ .retentionPeriod | quote }}
@@ -84,6 +86,7 @@ spec:
8486
retention.maxDiskUsagePercent: "80"
8587
{{- end }}
8688
replicaCount: 2
89+
{{- include "monitoring.topologySpreadConstraints" (list $ "vtstorage" .name "DoNotSchedule") | nindent 4 }}
8790
resources: {}
8891
storage:
8992
volumeClaimTemplate:

0 commit comments

Comments
 (0)