@@ -132,18 +132,23 @@ Pay attention to the last command displayed before the error; it often indicates
132132
133133## Kube-OVN crash
134134
135- In difficult cases, you may encounter issues where the Kube-OVN DaemonSet pods crash or fail to start properly.
136- This usually indicates a corrupted OVN database. You can confirm this by checking the logs of the Kube-OVN CNI pods:
135+ In complex cases, you may encounter issues where the Kube-OVN DaemonSet pods crash or fail to start properly.
136+ This usually indicates a corrupted OVN database.
137+ You can confirm this by checking the logs of the Kube-OVN CNI pods.
137138
138- ```
139+ Get the list of pods in ` cozy-kubeovn ` namespace:
140+
141+ ``` console
139142# kubectl get pod -n cozy-kubeovn
140143NAME READY STATUS RESTARTS AGE
141144kube-ovn-cni-5rsvz 0/1 Running 5 (35s ago) 4m37s
142145kube-ovn-cni-jq2zz 0/1 Running 5 (33s ago) 4m39s
143146kube-ovn-cni-p4gz2 0/1 Running 3 (23s ago) 4m38s
144147```
145148
146- ```
149+ Read the logs of a pod by its name (` kube-ovn-cni-jq2zz ` in this example):
150+
151+ ``` console
147152# kubectl logs -n cozy-kubeovn kube-ovn-cni-jq2zz
148153W0725 08:21:12.479452 87678 ovs.go:35] 100.64.0.4 network not ready after 3 ping to gateway 100.64.0.1
149154W0725 08:21:15.479600 87678 ovs.go:35] 100.64.0.4 network not ready after 6 ping to gateway 100.64.0.1
@@ -160,8 +165,9 @@ W0725 08:21:45.478754 87678 ovs.go:35] 100.64.0.4 network not ready after 36 p
160165W0725 08:21:48.479396 87678 ovs.go:35] 100.64.0.4 network not ready after 39 ping to gateway 100.64.0.1
161166```
162167
163- To resolve this issue, you can clean up the OVN database. This involves running a DaemonSet that removes the OVN configuration files from each node.
164- It is safe to perform this cleanup — the Kube-OVN DaemonSet will automatically recreate the necessary files from the Kubernetes A
168+ To resolve this issue, you can clean up the OVN database.
169+ This involves running a DaemonSet that removes the OVN configuration files from each node.
170+ It is safe to perform this cleanup — the Kube-OVN DaemonSet will automatically recreate the necessary files from the Kubernetes API.
165171
166172Apply the following YAML to deploy the cleanup DaemonSet:
167173
@@ -213,8 +219,8 @@ spec:
213219
214220Verify that the DaemonSet is running:
215221
216- ` ` ` bash
217- # kubectl get pod -n cozy-kubeovn
222+ ` ` ` console
223+ kubectl get pod -n cozy-kubeovn
218224ovn-cleanup-hjzxb 1/1 Running 0 6s
219225ovn-cleanup-wmzdv 1/1 Running 0 6s
220226ovn-cleanup-ztm86 1/1 Running 0 6s
@@ -228,21 +234,22 @@ kubectl get pod -n cozy-kubeovn
228234```
229235
230236
237+ ## Remove a failed node from the cluster
231238
239+ When a cluster node fails, Cozystack automatically handles high availability by recreating replicated PVCs and workloads on other nodes.
240+ However, there can be issues that require removing the node to resolve:
232241
233- ## Remove failed node from a cluster
234-
235- Cozystack automatically handles high availability by recreating replicated PVCs and workloads on other nodes when a node fails.
236- However, local storage PVs may remain bound to the failed node, which can cause issues with new pods. These need to be cleaned up manually.
242+ - Local storage PVs may remain bound to the failed node, which can cause issues with new pods.
243+ These need to be cleaned up manually.
237244
238- Additionally, the failed node will still exist in the cluster, which can lead to inconsistencies in the cluster state and affect pod scheduling.
245+ - The failed node will still exist in the cluster, which can lead to inconsistencies in the cluster state and affect pod scheduling.
239246
240247
241- #### Step 1: Remove the Node from the Cluster
248+ ### Step 1: Remove the Node from the Cluster
242249
243250Run the following command to remove the failed node (replace mynode with the actual node name):
244251
245- ```
252+ ``` bash
246253kubectl delete node mynode
247254```
248255
@@ -254,7 +261,7 @@ talm -f nodes/node1.yaml etcd member list
254261
255262Example output:
256263
257- ```
264+ ``` console
258265NODE ID HOSTNAME PEER URLS CLIENT URLS LEARNER
25926637.27.60.28 2ba6e48b8cf1a0c1 node1 https://192.168.100.11:2380 https://192.168.100.11:2379 false
26026737.27.60.28 b82e2194fb76ee42 node2 https://192.168.100.12:2380 https://192.168.100.12:2379 false
@@ -267,27 +274,28 @@ Then remove the corresponding member (replace the ID with the one for your faile
267274talm -f nodes/node1.yaml etcd remove-member f24f4de3d01e5e88
268275```
269276
270- #### Step 2: Remove PVCs and Pods Bound to the Failed Node
277+ ### Step 2: Remove PVCs and Pods Bound to the Failed Node
271278
272279Here are few commands to help you clean up the failed node:
273280
274- To delete PVCs bound to the failed node use the following command. Replace ` mynode ` with the name of your failed node:
281+ - ** Delete PVCs** bound to the failed node:<br >
282+ (Replace ` mynode ` with the name of your failed node)
283+
284+ ``` bash
285+ kubectl get pv -o json | jq -r ' .items[] | select(.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0] == "mynode").spec.claimRef | "kubectl delete pvc -n \(.namespace) \(.name)"' | sh -x
286+ ```
287+
288+ - ** Delete pods** stuck in ` Pending` state across all namespaces:
289+
290+ ` ` ` bash
291+ kubectl get pod -A | awk ' /Pending/ {print "kubectl delete pod -n " $1 " " $2}' | sh -x
292+ ` ` `
275293
276- ``` bash
277- kubectl get pv -o json | jq -r ' .items[] | select(.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0] == "mynode").spec.claimRef | "kubectl delete pvc -n \(.namespace) \(.name)"' | sh -x
278- ```
294+ # ## Step 3: Check Resource Status
279295
280- To delete pods stuck in the Pending state across all namespaces :
296+ After cleanup, check for any resource issues using ` linstor advise ` :
281297
282- ``` bash
283- kubectl get pod -A | awk ' /Pending/ {print "kubectl delete pod -n " $1 " " $2}' | sh -x
284- ```
285-
286- #### Step 3: Check Resource Status
287-
288- After cleanup, check for any resource issues using linstor advise:
289-
290- ``` bash
298+ ` ` ` console
291299# linstor advise resource
292300╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
293301┊ Resource ┊ Issue ┊ Possible fix ┊
@@ -307,4 +315,4 @@ After cleanup, check for any resource issues using linstor advise:
307315╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
308316` ` `
309317
310- Run the suggested linstor rd ap commands to restore the desired replica count.
318+ Run the ` linstor rd ap` commands suggested in the " Possible fix " column to restore the desired replica count.
0 commit comments