[Backport release-1.6] fix(etcd): require an even spread of members over nodes by default - #4988

fix(etcd): tenant etcd members now spread over nodes with whenUnsatisfiable DoNotSchedule by default, so two members of one etcd no longer start on the same node while another node has none. Members that exist are not moved, only members created after the upgrade get the new constraint.

An etcd created while the cluster had fewer nodes than members puts two or three members on one node. Once a node is added, such a member whose Pod is recreated (drain, eviction, graceful node shutdown) stays Pending, because maxSkew rejects its own node and its node-local volume rules out the others. With two nodes grown to three the etcd runs two of three members and its PodDisruptionBudget blocks draining the other nodes. With one node grown to three a reboot of that node brings back one member of three and quorum is lost. A stranded member is replaced by deleting its EtcdMember while the other members keep quorum. Its Pod and PVC go with it, and the replacement gets a new Pod and PVC on the node without a member. Right after adding nodes, doing this for one co-located member at a time while all members are Ready avoids the case.

With nodeTaintsPolicy Ignore a node the etcd Pods cannot tolerate, such as a tainted control plane, counts as a node with zero members, so no node can take a second member. On a cluster with such a node and fewer schedulable nodes than members, a new etcd starts only as many members as there are schedulable nodes, one of three on a single schedulable node. An existing etcd there keeps its members, but a member it replaces stays Pending the same way.

Either case is avoided by setting scheduling.globalAppTopologySpreadConstraints in spec.components.platform.values of the cozystack.cozystack-platform Package to a topologySpreadConstraints entry with maxSkew 1, topologyKey kubernetes.io/hostname and whenUnsatisfiable ScheduleAnyway before the upgrade. Every app chart that reads this value applies it, two members of a new etcd can share a node again, and the operator gives it only to members created after it adopts the changed EtcdCluster spec into status.observed. Once status.observed.topologySpreadConstraints of an EtcdCluster shows ScheduleAnyway, a member of it that stays Pending is replaced by deleting its EtcdMember.