Skip to content

Commit ec6bf1a

Browse files
committed
Troubleshoot Kube-OVN and remove failed nodes
Signed-off-by: Andrei Kvapil <[email protected]>
1 parent f642be9 commit ec6bf1a

1 file changed

Lines changed: 180 additions & 0 deletions

File tree

‎content/en/docs/operations/troubleshooting.md‎

Lines changed: 180 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -128,3 +128,183 @@ bash -x talos-bootstrap
128128
```
129129

130130
Pay attention to the last command displayed before the error; it often indicates the command that failed and can provide clues for further troubleshooting.
131+
132+
133+
### Kube-OVN crash
134+
135+
In difficult cases, you may encounter issues where the Kube-OVN DaemonSet pods crash or fail to start properly.
136+
This usually indicates a corrupted OVN database. You can confirm this by checking the logs of the Kube-OVN CNI pods:
137+
138+
```
139+
# k get pod -n cozy-kubeovn
140+
NAME READY STATUS RESTARTS AGE
141+
kube-ovn-cni-5rsvz 0/1 Running 5 (35s ago) 4m37s
142+
kube-ovn-cni-jq2zz 0/1 Running 5 (33s ago) 4m39s
143+
kube-ovn-cni-p4gz2 0/1 Running 3 (23s ago) 4m38s
144+
```
145+
146+
```
147+
# k logs -n cozy-kubeovn kube-ovn-cni-jq2zz
148+
W0725 08:21:12.479452 87678 ovs.go:35] 100.64.0.4 network not ready after 3 ping to gateway 100.64.0.1
149+
W0725 08:21:15.479600 87678 ovs.go:35] 100.64.0.4 network not ready after 6 ping to gateway 100.64.0.1
150+
W0725 08:21:18.479628 87678 ovs.go:35] 100.64.0.4 network not ready after 9 ping to gateway 100.64.0.1
151+
W0725 08:21:21.479355 87678 ovs.go:35] 100.64.0.4 network not ready after 12 ping to gateway 100.64.0.1
152+
W0725 08:21:24.479322 87678 ovs.go:35] 100.64.0.4 network not ready after 15 ping to gateway 100.64.0.1
153+
W0725 08:21:27.479664 87678 ovs.go:35] 100.64.0.4 network not ready after 18 ping to gateway 100.64.0.1
154+
W0725 08:21:30.478907 87678 ovs.go:35] 100.64.0.4 network not ready after 21 ping to gateway 100.64.0.1
155+
W0725 08:21:33.479738 87678 ovs.go:35] 100.64.0.4 network not ready after 24 ping to gateway 100.64.0.1
156+
W0725 08:21:36.479607 87678 ovs.go:35] 100.64.0.4 network not ready after 27 ping to gateway 100.64.0.1
157+
W0725 08:21:39.479753 87678 ovs.go:35] 100.64.0.4 network not ready after 30 ping to gateway 100.64.0.1
158+
W0725 08:21:42.479480 87678 ovs.go:35] 100.64.0.4 network not ready after 33 ping to gateway 100.64.0.1
159+
W0725 08:21:45.478754 87678 ovs.go:35] 100.64.0.4 network not ready after 36 ping to gateway 100.64.0.1
160+
W0725 08:21:48.479396 87678 ovs.go:35] 100.64.0.4 network not ready after 39 ping to gateway 100.64.0.1
161+
```
162+
163+
To resolve this issue, you can clean up the OVN database. This involves running a DaemonSet that removes the OVN configuration files from each node.
164+
It is safe to perform this cleanup — the Kube-OVN DaemonSet will automatically recreate the necessary files from the Kubernetes A
165+
166+
Apply the following YAML to deploy the cleanup DaemonSet:
167+
168+
```yaml
169+
apiVersion: apps/v1
170+
kind: DaemonSet
171+
metadata:
172+
name: ovn-cleanup
173+
namespace: cozy-kubeovn
174+
spec:
175+
selector:
176+
matchLabels:
177+
app: ovn-cleanup
178+
template:
179+
metadata:
180+
labels:
181+
app: ovn-cleanup
182+
component: network
183+
type: infra
184+
spec:
185+
affinity:
186+
podAffinity:
187+
requiredDuringSchedulingIgnoredDuringExecution:
188+
- labelSelector:
189+
matchLabels:
190+
app: ovn-central
191+
topologyKey: "kubernetes.io/hostname"
192+
containers:
193+
- name: cleanup
194+
image: busybox
195+
command: ["/bin/sh", "-xc", "rm -rf /host-config-ovn/*; rm -rf /host-config-ovn/.*; exec sleep infinity"]
196+
volumeMounts:
197+
- name: host-config-ovn
198+
mountPath: /host-config-ovn
199+
nodeSelector:
200+
kubernetes.io/os: linux
201+
node-role.kubernetes.io/control-plane: ""
202+
tolerations:
203+
- operator: "Exists"
204+
volumes:
205+
- name: host-config-ovn
206+
hostPath:
207+
path: /var/lib/ovn
208+
type: ""
209+
hostNetwork: true
210+
restartPolicy: Always
211+
terminationGracePeriodSeconds: 1
212+
```
213+
214+
Verify that the DaemonSet is running:
215+
216+
```bash
217+
# kubectl get pod -n cozy-kubeovn
218+
ovn-cleanup-hjzxb 1/1 Running 0 6s
219+
ovn-cleanup-wmzdv 1/1 Running 0 6s
220+
ovn-cleanup-ztm86 1/1 Running 0 6s
221+
```
222+
223+
Once the cleanup is complete, delete the DaemonSet and restart the Kube-OVN DaemonSet pods to apply the new configuration:
224+
225+
```bash
226+
kubectl delete ds
227+
kubectl get pod -n cozy-kubeovn
228+
```
229+
230+
231+
232+
233+
### Remove failed node from a cluster
234+
235+
Cozystack automatically handles high availability by recreating replicated PVCs and workloads on other nodes when a node fails.
236+
However, local storage PVs may remain bound to the failed node, which can cause issues with new pods. These need to be cleaned up manually.
237+
238+
Additionally, the failed node will still exist in the cluster, which can lead to inconsistencies in the cluster state and affect pod scheduling.
239+
240+
241+
##### Step 1: Remove the Node from the Cluster
242+
243+
Run the following command to remove the failed node (replace mynode with the actual node name):
244+
245+
```
246+
kubectl delete node mynode
247+
```
248+
249+
If the failed node is a control-plane node, you must also remove its etcd member from the etcd cluster:
250+
251+
```bash
252+
# talm -f nodes/node1.yaml etcd member list
253+
```
254+
255+
Example output:
256+
257+
```
258+
NODE ID HOSTNAME PEER URLS CLIENT URLS LEARNER
259+
37.27.60.28 2ba6e48b8cf1a0c1 node1 https://192.168.100.11:2380 https://192.168.100.11:2379 false
260+
37.27.60.28 b82e2194fb76ee42 node2 https://192.168.100.12:2380 https://192.168.100.12:2379 false
261+
37.27.60.28 f24f4de3d01e5e88 node3 https://192.168.100.13:2380 https://192.168.100.13:2379 false
262+
```
263+
264+
Then remove the corresponding member (replace the ID with the one for your failed node):
265+
266+
```bash
267+
talm -f nodes/node1.yaml etcd remove-member f24f4de3d01e5e88
268+
```
269+
270+
##### Step 2: Remove PVCs and Pods Bound to the Failed Node
271+
272+
Here are few commands to help you clean up the failed node:
273+
274+
To delete PVCs bound to the failed node use the following command. Replace `mynode` with the name of your failed node:
275+
276+
```bash
277+
kubectl get pv -o json | jq -r '.items[] | select(.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0] == "mynode").spec.claimRef | "kubectl delete pvc -n \(.namespace) \(.name)"' | sh -x
278+
```
279+
280+
To delete pods stuck in the Pending state across all namespaces:
281+
282+
```bash
283+
kubectl get pod -A | awk '/Pending/ {print "kubectl delete pod -n " $1 " " $2}' | sh -x
284+
```
285+
286+
#### Step 3: Check Resource Status
287+
288+
After cleanup, check for any resource issues using linstor advise:
289+
290+
```bash
291+
# linstor advise resource
292+
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
293+
┊ Resource ┊ Issue ┊ Possible fix ┊
294+
╞═══════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════════╡
295+
┊ pvc-02b0c0a1-e0b6-4e98-9384-60ff24f3b3b6 ┊ Resource expected to have 3 replicas, got only 2. ┊ linstor rd ap --place-count 3 pvc-02b0c0a1-e0b6-4e98-9384-60ff24f3b3b6 ┊
296+
┊ pvc-06e3b406-23f0-4f10-8b03-84063c1b2a12 ┊ Resource expected to have 3 replicas, got only 2. ┊ linstor rd ap --place-count 3 pvc-06e3b406-23f0-4f10-8b03-84063c1b2a12 ┊
297+
┊ pvc-a0b8aeaf-076e-4bd9-93ed-c4db09c04d0b ┊ Resource expected to have 3 replicas, got only 2. ┊ linstor rd ap --place-count 3 pvc-a0b8aeaf-076e-4bd9-93ed-c4db09c04d0b ┊
298+
┊ pvc-a523ebeb-c3b6-468d-abe5-f6afbbf31081 ┊ Resource expected to have 3 replicas, got only 2. ┊ linstor rd ap --place-count 3 pvc-a523ebeb-c3b6-468d-abe5-f6afbbf31081 ┊
299+
┊ pvc-cf7e87b5-3e6d-4034-903d-4625830fb5b4 ┊ Resource expected to have 1 replicas, got only 0. ┊ linstor rd ap --place-count 1 pvc-cf7e87b5-3e6d-4034-903d-4625830fb5b4 ┊
300+
┊ pvc-d344bc83-97fd-4489-bbe7-5399eea57165 ┊ Resource expected to have 3 replicas, got only 2. ┊ linstor rd ap --place-count 3 pvc-d344bc83-97fd-4489-bbe7-5399eea57165 ┊
301+
┊ pvc-d39345a9-5446-4c64-a5ba-957ff7c7a31f ┊ Resource expected to have 3 replicas, got only 2. ┊ linstor rd ap --place-count 3 pvc-d39345a9-5446-4c64-a5ba-957ff7c7a31f ┊
302+
┊ pvc-db6d4236-93bd-4268-9dcc-0ed275b17067 ┊ Resource expected to have 1 replicas, got only 0. ┊ linstor rd ap --place-count 1 pvc-db6d4236-93bd-4268-9dcc-0ed275b17067 ┊
303+
┊ pvc-ebb412c3-083c-4eee-93dc-70917ea6d87e ┊ Resource expected to have 1 replicas, got only 0. ┊ linstor rd ap --place-count 1 pvc-ebb412c3-083c-4eee-93dc-70917ea6d87e ┊
304+
┊ pvc-f107aacb-78d7-4ac6-97f8-8ed529a9c292 ┊ Resource expected to have 3 replicas, got only 2. ┊ linstor rd ap --place-count 3 pvc-f107aacb-78d7-4ac6-97f8-8ed529a9c292 ┊
305+
┊ pvc-f347d71a-b646-45e5-a717-f0a745061beb ┊ Resource expected to have 1 replicas, got only 0. ┊ linstor rd ap --place-count 1 pvc-f347d71a-b646-45e5-a717-f0a745061beb ┊
306+
┊ pvc-f6e96c83-6144-4510-b0ab-61936db52391 ┊ Resource expected to have 3 replicas, got only 2. ┊ linstor rd ap --place-count 3 pvc-f6e96c83-6144-4510-b0ab-61936db52391 ┊
307+
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
308+
```
309+
310+
Run the suggested linstor rd ap commands to restore the desired replica count.

0 commit comments

Comments
 (0)