Upgrade image pulls stall the management etcd on Talos nodes #4962

What we see

Upgrading two 3-node clusters from v1.6.4 to v1.6.5 (Talos 1.13.6, etcd 3.6.12, the management etcd and the containerd image store on the same system disk), we saw the management etcd react to the image pulls of the release.

On the first cluster we ran the installer upgrade as usual. Starting about two minutes after helm upgrade and over the next five minutes, the management etcd elected a new leader four times (terms 272 to 275). etcd logged slow fdatasync up to 5.0 s and leader failed to send out heartbeat on time; took too long, leader is overloaded likely from slow disk while the nodes pulled the new platform images. Later a single pull of piraeus-server (about 370 MB) on the leader node was enough for one more election, with slow fdatasync of 5.16 s. Quorum held every time, but API writes stall during each election in the middle of the platform rollout.

On the second cluster we pulled the 25 changed images before the upgrade, one node at a time with talosctl image pull --namespace cri, the etcd leader node last. That took about 5.5 minutes per node and produced two slow fdatasync entries of about 1.0 and 1.2 s and no election. The installer upgrade, the Cilium IPAM migration hook and the rolling of all platform components and LINSTOR satellites then ran with no leader change at all. One node still logged nine slow fdatasync warnings while the network components restarted, without an election.

Question

Would it make sense to make the pull part of the upgrade explicit? Two options we can think of:

1. Document a pre-pull step in the upgrade guide, with the list of images of the release and the advice to pull node by node, leader last.
2. Ship something with the release that pulls the images onto every node before the HelmReleases start rolling, throttled to one node at a time.

We are happy to send the docs part if option 1 is welcome. The image list of a release is easy to derive from the packages artifact, we used the images whose digests differ between the two releases.

Environment

• Cozystack v1.6.4 to v1.6.5, installer chart via helm upgrade
• Talos 1.13.6, Kubernetes 1.33 and 1.34, three control-plane nodes per cluster
• etcd 3.6.12 on the system disk shared with the containerd image store