Skip to content

Commit 401993a

Browse files
test(e2e): wait for parent HR Ready before downstream asserts
Two timing bugs unmasked by Flux v2.8 kstatus, with the same root shape: the bats steps assumed a side-effect ordering ("by the time X is true, the parent HR's helm install is done") that held under v2.7's faster dispatch but breaks under v2.8 where helm-controller walks kstatus on every applied resource before flipping the HR Ready. run-kubernetes.sh: After every addon HR (cilium, coredns, csi, vsnap-crd, ingress-nginx) reaches Ready, the remediation guard reads `.status.history` on the parent kubernetes-${test_name} HR. The parent's helm install can still be Reconciling at that point because kstatus is finishing its full sweep; `.status.history` is empty until install completes, so the guard exits 1 with "empty .status.history on a Ready HR". kubernetes- latest masked this because the ouroboros assertions add ~10m of buffer; kubernetes-previous (no ouroboros) hit the guard with the parent still installing. Add an explicit kubectl wait hr/kubernetes-${test_name} --for=condition=ready before the guard. vminstance.bats: The test waited 120s for `kubectl get vmi vm-instance-$name` to return a VMI with an assigned IP, then waited for the parent HR Ready separately afterwards. Under v2.7 the parent HR's helm install applied the VirtualMachine CR fast enough that kubevirt had created the VMI before the 120s budget expired. Under v2.8 the dispatch is slower and the VMI poll returned NotFound for its entire budget. Reorder: wait for HR Ready first (which implies the VirtualMachine CR is applied + kstatus-ready), then poll for the VMI with IP. Loosen the VMI poll budget from 120s to 5m for nested-virt slow paths. Assisted-By: Claude <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]> (cherry picked from commit c664b86)
1 parent 7b9f286 commit 401993a

2 files changed

Lines changed: 18 additions & 4 deletions

File tree

‎hack/e2e-apps/run-kubernetes.sh‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -477,6 +477,15 @@ EOF
477477
delete ingress hairpin-probe --ignore-not-found 2>/dev/null || true
478478
fi
479479

480+
# Wait for the parent kubernetes-${test_name} HR to be Ready before the
481+
# remediation guard runs. The guard reads `.status.history`, which is empty
482+
# until the helm install action completes — under Flux v2.8 kstatus the
483+
# parent's helm install can still be "Running 'install'" after every child
484+
# HR (cilium, coredns, csi, vsnap-crd, ingress-nginx) is already Ready,
485+
# because kstatus walks all applied resources before flipping the parent
486+
# Ready.
487+
kubectl wait hr -n tenant-test "kubernetes-${test_name}" --timeout=5m --for=condition=ready
488+
480489
# Guard: parent HelmRelease must not have entered an install/upgrade remediation cycle.
481490
# A non-zero installFailures/upgradeFailures indicates the helm-wait budget expired while
482491
# admin-kubeconfig was still being provisioned, which would trigger uninstall remediation

‎hack/e2e-apps/vminstance.bats‎

Lines changed: 9 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -100,11 +100,16 @@ EOF
100100
# Wait for the operator to materialise the HelmRelease before downstream
101101
# waits proceed (kubectl wait errors immediately if the HR does not exist).
102102
timeout 60 sh -ec "until kubectl -n tenant-test get hr vm-instance-$name >/dev/null 2>&1; do sleep 2; done"
103-
# Nested KubeVirt VM startup (virt-launcher + libvirt + cloud-init DHCP)
104-
# routinely takes 30-60s under runner load; the previous 20s was unrealistic
105-
# and produced flakes. 120s is a comfortable upper bound for nested virt.
106-
timeout 120 sh -ec "until kubectl -n tenant-test get vmi vm-instance-$name -o jsonpath='{.status.interfaces[0].ipAddress}' | grep -q '[0-9]'; do sleep 2; done"
103+
# Wait for the parent HR Ready before polling for the VMI. Under Flux v2.8
104+
# kstatus the helm install can be still dispatching when the previous
105+
# ordering assumed it had already applied the VirtualMachine CR — without
106+
# this the next poll for `vmi vm-instance-$name` returned NotFound for
107+
# its entire 120s budget because kubevirt hadn't seen the VM yet.
107108
kubectl -n tenant-test wait hr vm-instance-$name --timeout=5m --for=condition=ready
109+
# Nested KubeVirt VM startup (virt-launcher + libvirt + cloud-init DHCP)
110+
# routinely takes 30-60s under runner load. 5m is a comfortable upper
111+
# bound for nested virt + slow runner I/O after the HR-Ready gate.
112+
timeout 5m sh -ec "until kubectl -n tenant-test get vmi vm-instance-$name -o jsonpath='{.status.interfaces[0].ipAddress}' | grep -q '[0-9]'; do sleep 2; done"
108113
# VM ready follows IP assignment closely; 60s gives buffer for the qemu-guest-agent.
109114
kubectl -n tenant-test wait vm vm-instance-$name --timeout=5m --for=condition=ready
110115
kubectl -n tenant-test delete vminstances.apps.cozystack.io $name --ignore-not-found --timeout=3m

0 commit comments

Comments
 (0)