Skip to content

Commit d67060f

Browse files
committed
[docs] Proofread OVN crash and Removing Nodes
Signed-off-by: Nick Volynkin <[email protected]>
1 parent c9fc5d6 commit d67060f

1 file changed

Lines changed: 40 additions & 32 deletions

File tree

‎content/en/docs/operations/troubleshooting.md‎

Lines changed: 40 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -132,18 +132,23 @@ Pay attention to the last command displayed before the error; it often indicates
132132

133133
## Kube-OVN crash
134134

135-
In difficult cases, you may encounter issues where the Kube-OVN DaemonSet pods crash or fail to start properly.
136-
This usually indicates a corrupted OVN database. You can confirm this by checking the logs of the Kube-OVN CNI pods:
135+
In complex cases, you may encounter issues where the Kube-OVN DaemonSet pods crash or fail to start properly.
136+
This usually indicates a corrupted OVN database.
137+
You can confirm this by checking the logs of the Kube-OVN CNI pods.
137138

138-
```
139+
Get the list of pods in `cozy-kubeovn` namespace:
140+
141+
```console
139142
# kubectl get pod -n cozy-kubeovn
140143
NAME READY STATUS RESTARTS AGE
141144
kube-ovn-cni-5rsvz 0/1 Running 5 (35s ago) 4m37s
142145
kube-ovn-cni-jq2zz 0/1 Running 5 (33s ago) 4m39s
143146
kube-ovn-cni-p4gz2 0/1 Running 3 (23s ago) 4m38s
144147
```
145148

146-
```
149+
Read the logs of a pod by its name (`kube-ovn-cni-jq2zz` in this example):
150+
151+
```console
147152
# kubectl logs -n cozy-kubeovn kube-ovn-cni-jq2zz
148153
W0725 08:21:12.479452 87678 ovs.go:35] 100.64.0.4 network not ready after 3 ping to gateway 100.64.0.1
149154
W0725 08:21:15.479600 87678 ovs.go:35] 100.64.0.4 network not ready after 6 ping to gateway 100.64.0.1
@@ -160,8 +165,9 @@ W0725 08:21:45.478754 87678 ovs.go:35] 100.64.0.4 network not ready after 36 p
160165
W0725 08:21:48.479396 87678 ovs.go:35] 100.64.0.4 network not ready after 39 ping to gateway 100.64.0.1
161166
```
162167

163-
To resolve this issue, you can clean up the OVN database. This involves running a DaemonSet that removes the OVN configuration files from each node.
164-
It is safe to perform this cleanup — the Kube-OVN DaemonSet will automatically recreate the necessary files from the Kubernetes A
168+
To resolve this issue, you can clean up the OVN database.
169+
This involves running a DaemonSet that removes the OVN configuration files from each node.
170+
It is safe to perform this cleanup — the Kube-OVN DaemonSet will automatically recreate the necessary files from the Kubernetes API.
165171

166172
Apply the following YAML to deploy the cleanup DaemonSet:
167173

@@ -213,8 +219,8 @@ spec:
213219
214220
Verify that the DaemonSet is running:
215221
216-
```bash
217-
# kubectl get pod -n cozy-kubeovn
222+
```console
223+
kubectl get pod -n cozy-kubeovn
218224
ovn-cleanup-hjzxb 1/1 Running 0 6s
219225
ovn-cleanup-wmzdv 1/1 Running 0 6s
220226
ovn-cleanup-ztm86 1/1 Running 0 6s
@@ -228,21 +234,22 @@ kubectl get pod -n cozy-kubeovn
228234
```
229235

230236

237+
## Remove a failed node from the cluster
231238

239+
When a cluster node fails, Cozystack automatically handles high availability by recreating replicated PVCs and workloads on other nodes.
240+
However, there can be issues that require removing the node to resolve:
232241

233-
## Remove failed node from a cluster
234-
235-
Cozystack automatically handles high availability by recreating replicated PVCs and workloads on other nodes when a node fails.
236-
However, local storage PVs may remain bound to the failed node, which can cause issues with new pods. These need to be cleaned up manually.
242+
- Local storage PVs may remain bound to the failed node, which can cause issues with new pods.
243+
These need to be cleaned up manually.
237244

238-
Additionally, the failed node will still exist in the cluster, which can lead to inconsistencies in the cluster state and affect pod scheduling.
245+
- The failed node will still exist in the cluster, which can lead to inconsistencies in the cluster state and affect pod scheduling.
239246

240247

241-
#### Step 1: Remove the Node from the Cluster
248+
### Step 1: Remove the Node from the Cluster
242249

243250
Run the following command to remove the failed node (replace mynode with the actual node name):
244251

245-
```
252+
```bash
246253
kubectl delete node mynode
247254
```
248255

@@ -254,7 +261,7 @@ talm -f nodes/node1.yaml etcd member list
254261

255262
Example output:
256263

257-
```
264+
```console
258265
NODE ID HOSTNAME PEER URLS CLIENT URLS LEARNER
259266
37.27.60.28 2ba6e48b8cf1a0c1 node1 https://192.168.100.11:2380 https://192.168.100.11:2379 false
260267
37.27.60.28 b82e2194fb76ee42 node2 https://192.168.100.12:2380 https://192.168.100.12:2379 false
@@ -267,27 +274,28 @@ Then remove the corresponding member (replace the ID with the one for your faile
267274
talm -f nodes/node1.yaml etcd remove-member f24f4de3d01e5e88
268275
```
269276

270-
#### Step 2: Remove PVCs and Pods Bound to the Failed Node
277+
### Step 2: Remove PVCs and Pods Bound to the Failed Node
271278

272279
Here are few commands to help you clean up the failed node:
273280

274-
To delete PVCs bound to the failed node use the following command. Replace `mynode` with the name of your failed node:
281+
- **Delete PVCs** bound to the failed node:<br>
282+
(Replace `mynode` with the name of your failed node)
283+
284+
```bash
285+
kubectl get pv -o json | jq -r '.items[] | select(.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0] == "mynode").spec.claimRef | "kubectl delete pvc -n \(.namespace) \(.name)"' | sh -x
286+
```
287+
288+
- **Delete pods** stuck in `Pending` state across all namespaces:
289+
290+
```bash
291+
kubectl get pod -A | awk '/Pending/ {print "kubectl delete pod -n " $1 " " $2}' | sh -x
292+
```
275293

276-
```bash
277-
kubectl get pv -o json | jq -r '.items[] | select(.spec.nodeAffinity.required.nodeSelectorTerms[0].matchExpressions[0].values[0] == "mynode").spec.claimRef | "kubectl delete pvc -n \(.namespace) \(.name)"' | sh -x
278-
```
294+
### Step 3: Check Resource Status
279295

280-
To delete pods stuck in the Pending state across all namespaces:
296+
After cleanup, check for any resource issues using `linstor advise`:
281297

282-
```bash
283-
kubectl get pod -A | awk '/Pending/ {print "kubectl delete pod -n " $1 " " $2}' | sh -x
284-
```
285-
286-
#### Step 3: Check Resource Status
287-
288-
After cleanup, check for any resource issues using linstor advise:
289-
290-
```bash
298+
```console
291299
# linstor advise resource
292300
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
293301
┊ Resource ┊ Issue ┊ Possible fix ┊
@@ -307,4 +315,4 @@ After cleanup, check for any resource issues using linstor advise:
307315
╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
308316
```
309317

310-
Run the suggested linstor rd ap commands to restore the desired replica count.
318+
Run the `linstor rd ap` commands suggested in the "Possible fix" column to restore the desired replica count.

0 commit comments

Comments
 (0)