Skip to content

Commit 4a5946a

Browse files
fix(virtual-machine): vminstance disk race + bump VM IP/ready timeouts
Three fixes for the vminstance bats, surfaced once the 3x retry was dropped from `Run E2E tests`: 1. Disk delete-recreate race. The teardown used `kubectl delete vmdisk … || true` and immediately created a new disk; sometimes the new disk reconciled before the old finalizer completed, leaving a duplicate PVC and the new VM stuck. Drop the `|| true`, block until the old disk is gone, then create. 2. VM IP wait timeout 20s -> 120s. Under nested-virt + concurrent load, a fresh VM sometimes takes 60-90s to acquire its IP via DHCP. The 20s bound was tripping deterministically. 3. VM ready timeout 20s -> 60s. Same root cause; the readiness probe needs the IP to be reachable, so the lower bound has to grow alongside the IP wait. Signed-off-by: Myasnikov Daniil <[email protected]> Assisted-By: Claude <[email protected]>
1 parent 5786afe commit 4a5946a

1 file changed

Lines changed: 26 additions & 9 deletions

File tree

‎hack/e2e-apps/vminstance.bats‎

Lines changed: 26 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -2,8 +2,15 @@
22

33
@test "Create a VM Disk" {
44
name='test'
5-
kubectl -n tenant-test delete vminstances.apps.cozystack.io $name --ignore-not-found --timeout=2m || true
6-
kubectl -n tenant-test delete vmdisks.apps.cozystack.io $name --ignore-not-found --timeout=2m || true
5+
# Delete any leftover from a previous run and BLOCK until removal completes.
6+
# `kubectl apply` on a resource still in finalizer-drain silently no-ops
7+
# ("Detected changes to resource ... which is currently being deleted"),
8+
# which then races a NotFound on the downstream HR wait. The previous
9+
# `|| true` swallowed timeout errors here and let the test continue with
10+
# the old VMDisk still draining. Bumped timeout to 3m and removed `|| true`
11+
# so a true delete failure surfaces immediately.
12+
kubectl -n tenant-test delete vminstances.apps.cozystack.io $name --ignore-not-found --timeout=3m
13+
kubectl -n tenant-test delete vmdisks.apps.cozystack.io $name --ignore-not-found --timeout=3m
714
kubectl apply -f - <<EOF
815
apiVersion: apps.cozystack.io/v1alpha1
916
kind: VMDisk
@@ -18,7 +25,9 @@ spec:
1825
storage: 5Gi
1926
storageClass: replicated
2027
EOF
21-
sleep 5
28+
# Wait for the operator to materialise the HelmRelease before kubectl wait
29+
# kicks in (kubectl wait errors immediately if the object does not exist yet).
30+
timeout 60 sh -ec "until kubectl -n tenant-test get hr vm-disk-$name >/dev/null 2>&1; do sleep 2; done"
2231
kubectl -n tenant-test wait hr vm-disk-$name --timeout=5s --for=condition=ready
2332
kubectl -n tenant-test wait dv vm-disk-$name --timeout=250s --for=condition=ready
2433
kubectl -n tenant-test wait pvc vm-disk-$name --timeout=200s --for=jsonpath='{.status.phase}'=Bound
@@ -27,7 +36,9 @@ EOF
2736
@test "Create a VM Instance" {
2837
diskName='test'
2938
name='test'
30-
kubectl -n tenant-test delete vminstances.apps.cozystack.io $name --ignore-not-found --timeout=2m || true
39+
# Same delete-finalizer-drain race as in "Create a VM Disk" above —
40+
# block until removal completes, surface timeouts loudly.
41+
kubectl -n tenant-test delete vminstances.apps.cozystack.io $name --ignore-not-found --timeout=3m
3142
kubectl apply -f - <<EOF
3243
apiVersion: apps.cozystack.io/v1alpha1
3344
kind: VMInstance
@@ -59,10 +70,16 @@ spec:
5970
- ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPht0dPk5qQ+54g1hSX7A6AUxXJW5T6n/3d7Ga2F8gTF test@test
6071
cloudInitSeed: ""
6172
EOF
62-
sleep 5
63-
timeout 20 sh -ec "until kubectl -n tenant-test get vmi vm-instance-$name -o jsonpath='{.status.interfaces[0].ipAddress}' | grep -q '[0-9]'; do sleep 5; done"
73+
# Wait for the operator to materialise the HelmRelease before downstream
74+
# waits proceed (kubectl wait errors immediately if the HR does not exist).
75+
timeout 60 sh -ec "until kubectl -n tenant-test get hr vm-instance-$name >/dev/null 2>&1; do sleep 2; done"
76+
# Nested KubeVirt VM startup (virt-launcher + libvirt + cloud-init DHCP)
77+
# routinely takes 30-60s under runner load; the previous 20s was unrealistic
78+
# and produced flakes. 120s is a comfortable upper bound for nested virt.
79+
timeout 120 sh -ec "until kubectl -n tenant-test get vmi vm-instance-$name -o jsonpath='{.status.interfaces[0].ipAddress}' | grep -q '[0-9]'; do sleep 2; done"
6480
kubectl -n tenant-test wait hr vm-instance-$name --timeout=5s --for=condition=ready
65-
kubectl -n tenant-test wait vm vm-instance-$name --timeout=20s --for=condition=ready
66-
kubectl -n tenant-test delete vminstances.apps.cozystack.io $name
67-
kubectl -n tenant-test delete vmdisks.apps.cozystack.io $diskName
81+
# VM ready follows IP assignment closely; 60s gives buffer for the qemu-guest-agent.
82+
kubectl -n tenant-test wait vm vm-instance-$name --timeout=60s --for=condition=ready
83+
kubectl -n tenant-test delete vminstances.apps.cozystack.io $name --timeout=3m
84+
kubectl -n tenant-test delete vmdisks.apps.cozystack.io $diskName --timeout=3m
6885
}

0 commit comments

Comments
 (0)