Skip to content

Commit d9a1655

Browse files
committed
docs(operations): pre-pull the new images before an upgrade on Talos
On Talos the management etcd and the containerd image store share the EPHEMERAL volume, and the kubelet pulls images in parallel, so the platform rollout makes every node pull and unpack the new images at once. On a three-node cluster going from v1.6.4 to v1.6.5 that stalled etcd fdatasync for up to 5 s and the leader changed four times in five minutes. A second cluster pulled the same images node by node before the upgrade, leader last, and kept its leader through the whole rollout. The new optional step describes that procedure, see cozystack/cozystack#4962. Assisted-by: LLM Signed-off-by: Yan Bondarenko <[email protected]>
1 parent b248316 commit d9a1655

2 files changed

Lines changed: 114 additions & 4 deletions

File tree

‎content/en/docs/next/operations/cluster/upgrade.md‎

Lines changed: 57 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,62 @@ kubectl annotate configmap -n cozy-system cozystack-version helm.sh/resource-pol
5454
release could delete the `cozy-system` namespace and all resources within it.
5555
{{% /alert %}}
5656

57-
### 3. Upgrade the Cozystack Operator
57+
### 3. Pre-pull the new images (optional)
58+
59+
By default Talos Linux keeps `/var/lib/etcd` and `/var/lib/containerd` as directories on the same `EPHEMERAL` volume, and its kubelet pulls images in parallel.
60+
During the upgrade many platform components restart at once, so every node pulls and unpacks their new images at the same time.
61+
If this delays the disk writes of the etcd leader long enough, etcd logs `slow fdatasync` and `failed to send out heartbeat on time`, the other members elect a new leader, and requests to the Kubernetes API stall during each election.
62+
Pulling the new images before the upgrade, one node at a time, keeps this load out of the rollout.
63+
64+
Build the list of new images from the Cozystack images the cluster runs now.
65+
Set `FROM` to the installed version and `TO` to the target version.
66+
67+
```bash
68+
FROM=v1.6.4
69+
TO=v1.6.5
70+
kubectl get pods --all-namespaces --output json \
71+
| jq --raw-output '.items[].spec | (.initContainers // [])[], .containers[] | .image' \
72+
| grep -E "^ghcr.io/cozystack/cozystack/[^:]+:${FROM}(@|$)" \
73+
| sed -E "s/:${FROM}(@sha256:[0-9a-f]+)?$/:${TO}/" \
74+
| sort -u > images.txt
75+
```
76+
77+
The list covers the images that carry the Cozystack version in their tag.
78+
Add other images that change in the release to `images.txt` by hand if you want them pulled too.
79+
80+
Find the etcd leader.
81+
It is the node whose `MEMBER` matches the `LEADER` column.
82+
83+
```bash
84+
talosctl --nodes <node1>,<node2>,<node3> etcd status
85+
```
86+
87+
Then pull the images on each control plane node in turn, the leader last.
88+
Nodes that do not run etcd can skip this step.
89+
90+
```bash
91+
while read -r image; do
92+
talosctl --nodes <node> image pull --namespace cri "$image"
93+
done < images.txt
94+
```
95+
96+
Before you move to the next node, run `etcd status` again and check that the leader and the `RAFT TERM` are the same, and that the etcd log of the node has no new `slow fdatasync` warnings.
97+
98+
```bash
99+
talosctl --nodes <node> logs etcd | grep 'slow fdatasync'
100+
```
101+
102+
A pull on the leader node can still delay its heartbeats.
103+
To avoid that, move the leadership to a node that already has the images before you pull on the leader.
104+
105+
```bash
106+
talosctl --nodes <leader-node> etcd forfeit-leadership
107+
```
108+
109+
Start the upgrade soon after the pull.
110+
The pre-pulled images stay unused until the rollout, and the kubelet removes unused images once the disk usage goes above its garbage collection threshold, 85% by default.
111+
112+
### 4. Upgrade the Cozystack Operator
58113

59114
Upgrade the Cozystack operator Helm release to the target version:
60115

@@ -72,7 +127,7 @@ You can read the logs of the operator:
72127
kubectl logs -n cozy-system deploy/cozystack-operator -f
73128
```
74129

75-
### 4. Check the cluster status after upgrading
130+
### 5. Check the cluster status after upgrading
76131

77132
```bash
78133
kubectl get pods -n cozy-system

‎content/en/docs/v1.6/operations/cluster/upgrade.md‎

Lines changed: 57 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -54,7 +54,62 @@ kubectl annotate configmap -n cozy-system cozystack-version helm.sh/resource-pol
5454
release could delete the `cozy-system` namespace and all resources within it.
5555
{{% /alert %}}
5656

57-
### 3. Upgrade the Cozystack Operator
57+
### 3. Pre-pull the new images (optional)
58+
59+
By default Talos Linux keeps `/var/lib/etcd` and `/var/lib/containerd` as directories on the same `EPHEMERAL` volume, and its kubelet pulls images in parallel.
60+
During the upgrade many platform components restart at once, so every node pulls and unpacks their new images at the same time.
61+
If this delays the disk writes of the etcd leader long enough, etcd logs `slow fdatasync` and `failed to send out heartbeat on time`, the other members elect a new leader, and requests to the Kubernetes API stall during each election.
62+
Pulling the new images before the upgrade, one node at a time, keeps this load out of the rollout.
63+
64+
Build the list of new images from the Cozystack images the cluster runs now.
65+
Set `FROM` to the installed version and `TO` to the target version.
66+
67+
```bash
68+
FROM=v1.6.4
69+
TO=v1.6.5
70+
kubectl get pods --all-namespaces --output json \
71+
| jq --raw-output '.items[].spec | (.initContainers // [])[], .containers[] | .image' \
72+
| grep -E "^ghcr.io/cozystack/cozystack/[^:]+:${FROM}(@|$)" \
73+
| sed -E "s/:${FROM}(@sha256:[0-9a-f]+)?$/:${TO}/" \
74+
| sort -u > images.txt
75+
```
76+
77+
The list covers the images that carry the Cozystack version in their tag.
78+
Add other images that change in the release to `images.txt` by hand if you want them pulled too.
79+
80+
Find the etcd leader.
81+
It is the node whose `MEMBER` matches the `LEADER` column.
82+
83+
```bash
84+
talosctl --nodes <node1>,<node2>,<node3> etcd status
85+
```
86+
87+
Then pull the images on each control plane node in turn, the leader last.
88+
Nodes that do not run etcd can skip this step.
89+
90+
```bash
91+
while read -r image; do
92+
talosctl --nodes <node> image pull --namespace cri "$image"
93+
done < images.txt
94+
```
95+
96+
Before you move to the next node, run `etcd status` again and check that the leader and the `RAFT TERM` are the same, and that the etcd log of the node has no new `slow fdatasync` warnings.
97+
98+
```bash
99+
talosctl --nodes <node> logs etcd | grep 'slow fdatasync'
100+
```
101+
102+
A pull on the leader node can still delay its heartbeats.
103+
To avoid that, move the leadership to a node that already has the images before you pull on the leader.
104+
105+
```bash
106+
talosctl --nodes <leader-node> etcd forfeit-leadership
107+
```
108+
109+
Start the upgrade soon after the pull.
110+
The pre-pulled images stay unused until the rollout, and the kubelet removes unused images once the disk usage goes above its garbage collection threshold, 85% by default.
111+
112+
### 4. Upgrade the Cozystack Operator
58113

59114
Upgrade the Cozystack operator Helm release to the target version:
60115

@@ -72,7 +127,7 @@ You can read the logs of the operator:
72127
kubectl logs -n cozy-system deploy/cozystack-operator -f
73128
```
74129

75-
### 4. Check the cluster status after upgrading
130+
### 5. Check the cluster status after upgrading
76131

77132
```bash
78133
kubectl get pods -n cozy-system

0 commit comments

Comments
 (0)