ControllerUnpublishVolume leaks orphan hot-plug volumes when VM teardown precedes VMI termination #182

Is this a BUG REPORT or FEATURE REQUEST?:

/kind bug

What happened:

In production we observe hot-plug "zombie" pods that survive after a tenant Pod is rescheduled from one tenant Node (KubeVirt VM A) to another (VM B). The CSI driver issues RemoveVolume to VM A and AddVolume to VM B, but in several edge cases the VM-A detach is silently dropped. The hot-plug pod on VM A stays Running, the underlying infra storage device stays exclusively attached to the source host, and the new attach on VM B fails with:

MapVolume.MapPodDevice failed for volume "pvc-XXXX" : rpc error: code = Internal
desc = NodePublishVolume failed for pvc-XXXX: failed to set source device readwrite: exit status 1

(on DRBD-backed infra storage; other RWO backends surface this as Multi-Attach error for volume "pvc-XXXX" from the kube-controller-manager attach/detach controller — same root cause).

We hit this 12 times concurrently across two tenant clusters today; the manual workaround is virtctl removevolume <vm-A> -n <ns> --volume-name=<dvName> for each, after which the new attach succeeds.

What you expected to happen:

ControllerUnpublishVolume should always release the hot-plug from the VMI, even when the parent VM object is missing or out of sync with the VMI status. Returning success without doing so leaves orphan hot-plug pods that are unsafe for any storage backend enforcing exclusive attachment.

How to reproduce it (as minimally and precisely as possible):

1. Tenant cluster running on top of KubeVirt (e.g. via CAPK).
2. Deploy a StatefulSet with one RWO PVC backed by the kubevirt-csi driver (storageClassName: kubevirt).
3. Cordon + drain the tenant Node hosting the pod so the StatefulSet pod moves to a different tenant Node (i.e. a different KubeVirt VM in the infra cluster).
4. While step 3 is in flight, force-delete the source VirtualMachine object (kubectl delete vm <vm-A>) before its VMI finishes terminating. This simulates CAPI-driven KubeVirtMachine deletion.
5. Observe:
   • The new tenant Pod stays in ContainerCreating.
   • kubectl describe pod hp-volume-XXXX -n <ns> on the destination reports MapVolume.MapPodDevice failed ... failed to set source device readwrite.
   • The source VMI (still alive) keeps the hot-plug in .status.volumeStatus[].hotplugVolume; the hot-plug pod on that VMI is still Running.

It is also reproducible without the force-delete, just less reliably: virt-api updates VM.spec optimistically on removevolume but virt-handler can take seconds to actually unplug; if external-attacher gives up in that window the detach is never re-attempted.

Anything else we need to know?:

Root cause

pkg/service/controller.go ControllerUnpublishVolume:

_, err = c.virtClient.GetWorkloadManagingVirtualMachine(ctx, c.infraClusterNamespace, vmName)
if err != nil {
    if !errors.IsNotFound(err) {
        return nil, err
    }
    // VM removed, vacuously succeeded in unpublish
    return nil, nil
}

That short-circuit was introduced by #119 to fix #83 ("volume attached to a node that was removed could not be detached"). It correctly unblocks the cleanup of a VolumeAttachment whose Node is permanently gone, but when the VMI (and its hot-plug pod, and the infra storage attachment) outlive the VM by even a few seconds — which happens routinely during CAPK machine deletion, PVC finalizer waits, or live-migration — the synchronous detach path silently skips work that still needs to be done.

The same kind of false-positive lives inside EnsureVolumeRemovedVM (pkg/kubevirt/client.go) which returns (true, nil) on VM-not-found, and inside removeVolumeFromVm (pkg/service/controller.go) which exits early before reaching the existing VMI-fallback branch added by #134.

Proposed fix

Make ControllerUnpublishVolume and removeVolumeFromVm consult both VM.spec.template.spec.volumes and VMI.status.volumeStatus, short-circuiting only when both surfaces agree the volume is gone. A new EnsureVolumeRemovedVMI helper would mirror the existing EnsureVolumeRemovedVM API.

I have a PR ready for this.

Environment

• KubeVirt CSI driver built from a recent rebase of upstream main (verified against commit 51c6bca).
• KubeVirt: v1.8.x in the infra cluster, CAPK-managed VMs.
• Infra storage: LINSTOR + DRBD 9.2.x. The same code path also breaks on any infra StorageClass that enforces single-writer semantics.

Happy to provide raw logs, VMI/VM YAML dumps, or to test patches.