Skip to content

Commit cb2f606

Browse files
authored
fix(csi): verify VMI Ready after kubevirt-csi Publish to surface stuck-PVC failures (#2659)
## What this PR does Stops `ControllerPublishVolume` from silently reporting success when the volume is in VM spec but the hotplug attach never actually completed. ### Root cause Upstream's fast path (`EnsureVolumeAvailableVM`) checks only `vm.Spec.Template.Spec.Volumes`. If a prior Publish wrote that entry but timed out waiting for VMI Ready (e.g. the infra PVC is stuck `ClaimPending` because LINSTOR cannot autoplace), the next Publish short-circuits to success. external-attacher sets `Attached=true`, but QEMU was never sent `device_add`. The tenant kubelet's NodeStageVolume then fails with: ``` rpc error: code = Unknown desc = couldn't find device by serial id ``` Observed in production. Sequence from kubevirt-csi-driver controller logs: ``` 16:53:13 /ControllerPublishVolume called 16:55:13 volume failed to be ready in time (2m), context deadline exceeded 16:55:14 /ControllerPublishVolume called (retry) 16:55:14 Volume X already attached to VM Y - skipping hot-plug 16:55:14 /ControllerPublishVolume returned with response: {} ``` VMI volumeStatus during that window: `phase=Pending message="PVC is in phase ClaimPending"`. VolumeAttachment then goes `Attached=true`; tenant kubelet surfaces "couldn't find device by serial id". ### Fix After upstream Publish returns success for a hotplug volume, the wrapper re-reads VMI status and checks `Phase == VolumeReady` for this DataVolume. If it is not Ready (or not present), the RPC returns `codes.Unavailable` with the upstream reason, so external-attacher keeps retrying instead of declaring the volume attached. The underlying provisioning failure surfaces at the CSI layer instead of becoming a confusing kubelet-side "device not found by serial id". This is a wrapper-side workaround for a defect in upstream `kubevirt.io/csi-driver`; the structural fix belongs upstream (tighten the fast path to require VMI Ready) and will be filed separately. The wrapper guard is safe to keep as defense-in-depth even after the upstream fix lands. Relates to #2634. ### Release note ```release-note fix(csi): kubevirt-csi-driver Publish now verifies VMI Ready before reporting attached, surfacing stuck-PVC failures at the CSI layer instead of as "couldn't find device by serial id" on the tenant kubelet ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved hotplug volume publishing: non‑NFS volumes are now verified on the VM before being marked published; if the volume is missing or not ready, publishing returns unavailable, improving reliability for hotplugged storage. <!-- review_stack_entry_start --> [![Review Change Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2659?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack) <!-- review_stack_entry_end --> <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents 60357eb + cbe4b86 commit cb2f606

1 file changed

Lines changed: 50 additions & 2 deletions

File tree

‎packages/apps/kubernetes/images/kubevirt-csi-driver/controller.go‎

Lines changed: 50 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ import (
2121
"k8s.io/client-go/tools/cache"
2222
"k8s.io/client-go/util/retry"
2323
"k8s.io/klog/v2"
24+
kubevirtv1 "kubevirt.io/api/core/v1"
2425
cdiv1 "kubevirt.io/containerized-data-importer-api/pkg/apis/core/v1beta1"
2526

2627
kubevirtclient "kubevirt.io/csi-driver/pkg/kubevirt"
@@ -237,12 +238,59 @@ func (w *WrappedControllerService) determineDvSource(ctx context.Context, req *c
237238
return res, nil
238239
}
239240

241+
// publishHotplugVolume delegates to upstream for the hotplug attach, then re-verifies
242+
// that the volume reached VolumeReady in VMI.Status.VolumeStatus before returning
243+
// success. Upstream's fast path (EnsureVolumeAvailableVM) considers the volume
244+
// attached based only on its presence in VM.spec.template.spec.volumes — so when an
245+
// earlier attempt wrote that entry but the volume never became Ready (e.g. the infra
246+
// PVC is stuck ClaimPending because the storage backend cannot provision), the
247+
// retried Publish reports success and external-attacher sets Attached=true.
248+
// QEMU never received device_add, and the tenant kubelet's NodeStageVolume fails
249+
// with "couldn't find device by serial id". Verifying VMI Ready independently of
250+
// upstream propagates the not-ready condition back to external-attacher so it keeps
251+
// retrying instead of silently declaring the volume attached.
252+
func (w *WrappedControllerService) publishHotplugVolume(ctx context.Context, req *csi.ControllerPublishVolumeRequest) (*csi.ControllerPublishVolumeResponse, error) {
253+
resp, err := w.ControllerService.ControllerPublishVolume(ctx, req)
254+
if err != nil {
255+
return nil, err
256+
}
257+
258+
vmNamespace, vmName, parseErr := cache.SplitMetaNamespaceKey(req.GetNodeId())
259+
if parseErr != nil {
260+
return nil, status.Errorf(codes.Internal,
261+
"cannot verify VMI readiness after publish: failed to parse node ID %q: %v",
262+
req.GetNodeId(), parseErr)
263+
}
264+
dvName := req.GetVolumeId()
265+
// GetVirtualMachine on this client returns a VirtualMachineInstance, not a VM.
266+
vmi, getErr := w.virtClient.GetVirtualMachine(ctx, vmNamespace, vmName)
267+
if getErr != nil {
268+
return nil, status.Errorf(codes.Unavailable,
269+
"cannot verify VMI %s/%s readiness after publish of %s: %v",
270+
vmNamespace, vmName, dvName, getErr)
271+
}
272+
for _, vs := range vmi.Status.VolumeStatus {
273+
if vs.Name != dvName {
274+
continue
275+
}
276+
if vs.Phase == kubevirtv1.VolumeReady {
277+
return resp, nil
278+
}
279+
return nil, status.Errorf(codes.Unavailable,
280+
"volume %s is in VM %s/%s spec but not Ready in VMI status (phase=%q, message=%q)",
281+
dvName, vmNamespace, vmName, vs.Phase, vs.Message)
282+
}
283+
return nil, status.Errorf(codes.Unavailable,
284+
"volume %s not present in VMI %s/%s status after publish", dvName, vmNamespace, vmName)
285+
}
286+
240287
// ControllerPublishVolume for NFS volumes: annotates infra PVC for WFFC binding,
241288
// waits for PVC bound, extracts NFS export from PV, and creates CiliumNetworkPolicy.
242-
// For RWO volumes, delegates to upstream (hotplug SCSI).
289+
// For hotplug volumes (RWO and RWX Block), delegates to upstream and then verifies
290+
// the volume is actually Ready in VMI.Status.VolumeStatus before returning success.
243291
func (w *WrappedControllerService) ControllerPublishVolume(ctx context.Context, req *csi.ControllerPublishVolumeRequest) (*csi.ControllerPublishVolumeResponse, error) {
244292
if req.GetVolumeContext()[nfsVolumeKey] != "true" {
245-
return w.ControllerService.ControllerPublishVolume(ctx, req)
293+
return w.publishHotplugVolume(ctx, req)
246294
}
247295

248296
if len(req.GetVolumeId()) == 0 {

0 commit comments

Comments
 (0)