You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a4cb00e
Browse filesBrowse the repository at this point in the historyBrowse files
fix(monitoring): spread monitoring replicas over nodes
The VMCluster, VLCluster, VTCluster and VMAlertmanager the chart renders
carry no affinity and no topology spread, and vm-operator copies both
fields onto the pods as written without adding any of its own. Nothing in
those specs kept two replicas of one tier off the same node. With
replicationFactor 2 both copies of a series can then sit on one node, and
on a node-local class the volumes keep them there, so losing that node
loses both copies. A StatefulSet pod on a lost node is not replaced
elsewhere until its Pod or Node object is deleted, which holds for the
Alertmanager replicas without a volume as well.
The tiers that run as StatefulSets, vmstorage, vmselect, vlstorage,
vtstorage and the Alertmanager, now get a per-node spread with
whenUnsatisfiable DoNotSchedule and nodeTaintsPolicy Ignore. The tiers
that run as Deployments, vminsert, vlinsert, vlselect, vtinsert and
vtselect, get the same spread with ScheduleAnyway, since their pods are
replaced on another node without help. A platform that sets
scheduling.globalAppTopologySpreadConstraints gets that policy for every
tier instead, with the tier's own selector added to each constraint that
has none.
vm-operator rolls a changed StatefulSet one pod at a time. A replica on a
node-local volume whose peer shares its node cannot be placed again under
the new rule, so its new pod stays Pending and the rollout of that
cluster stops there until the replica is moved.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <[email protected]>
0 commit comments