Repository navigation
Conversation
|
Skipping CI for Draft Pull Request. |
|
@kvaps: The label(s) DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
fb8fc11 to
91d3595
Compare
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
…migration (#3006) ## What this PR does Temporary revert. v1.5.0 restored persistent EFI/TPM state for the `windows.11` / `windows.2k22` / `windows.2k25` profiles. With persistence on, KubeVirt creates a backend-storage PVC (`persistent-state-for-<vm>`) which, on the default `replicated` StorageClass, is **ReadWriteOnce** — it pins the VM to its node and **blocks live-migration and node drains**. With `evictionStrategy: LiveMigrate` this can stall cluster upgrades. See #3005. This strips persistent EFI/TPM from the preferences again — but via `yq del` instead of the original `sed -d`, so an emptied `preferredTPM` stays `{}` (the v1.8 CRD rejects `null`, which is the bug the original strip caused). **Secure Boot and the vTPM remain present, just non-persistent** — Windows VMs still boot; they just don't persist EFI/TPM across reboots.⚠️ **Temporary stopgap** until vm-state can request an RWX backend — either RWX-Block via the upstream KubeVirt change (kubevirt/kubevirt#18215), or an RWX-Filesystem `vmStateStorageClass`. Once that lands, persistence can be re-enabled with live-migration intact. ### Screenshots No UI changes. ### Release note ```release-note fix(kubevirt-instancetypes): temporarily disable persistent EFI/TPM state to unblock live-migration of Windows VMs. Persistence created a ReadWriteOnce backend-storage PVC on the default storage class, pinning the VM to its node. Secure Boot and vTPM remain available (non-persistent); to be re-enabled once vm-state can use RWX storage. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Updated Windows virtual machine preferences to enable live migration and drain operations * Fixed compatibility issues with KubeVirt v1.8 virtual machine preference configurations * Removed persistent settings from TPM and EFI configurations to prevent blocking behaviors on Windows 11 and Windows Server 2022/2025 variants <!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
Thanks @mhenriks @jean-edouard! VEP opened: kubevirt/enhancements#349 (PR kubevirt/enhancements#350). Quick answers to @jean-edouard's questions (full detail in the VEP):
|
@kvaps are you aware of this: #12629?
|
Add KubeVirt.spec.configuration.vmStateVolumeMode (Filesystem|Block, default Filesystem). When Block, the vm-state backend PVC is rendered as a raw VolumeDevice and the EFI NVRAM is stored directly on the block device via libvirt <nvram type='block'> / QEMU pflash -- no filesystem, no mkfs/mount. This lets persistent-EFI VMs live-migrate on block-replicated storage (DRBD/LINSTOR), which provides RWX only at the block level: the OVMF VARS travel in QEMU's pflash migration stream like any pflash device, so the ReadWriteOnce-Filesystem pin that blocks live-migration today is avoided. Block-mode persistent EFI is routed through the explicit-loader firmware path rather than firmware auto-selection: auto-selection hides the OVMF VARS template path, but virt-launcher needs it to seed the blank raw device once (libvirt only auto-populates qcow2 nvram, not raw, since 10.8.0). Scope is EFI only: swtpm (vTPM) and CBT need a directory of files and stay Filesystem-only; Block vm-state is rejected for VMs that also need persistent TPM or CBT. The Filesystem /meta/migrated handshake is skipped for Block. Co-authored-by: Claude <[email protected]> Signed-off-by: Andrei Kvapil <[email protected]>
4b84769 to
eae35fa
Compare
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
f52ec25 to
d5ab74d
Compare
Extend Block-mode backend storage (KubeVirt.spec.configuration.vmStateVolumeMode
= Block) to persistent vTPM, complementing the existing Block-mode persistent
EFI. In Block mode each raw device backs exactly one fixed-size state blob, so a
persistent-TPM VM gets a SECOND backend-storage PVC alongside the EFI NVRAM one:
persistent-tpm-state-for-<vm> (label persistent-tpm-state-for: <vmName>), Block
mode, same access-mode negotiation as the EFI PVC, attached as a raw VolumeDevice
at /dev/vm-state-tpm.
swtpm backs its state file directly with the raw device via its file:// backend
(swtpm >= 0.7); libvirt (>= 10.9) exposes this through the TPM emulator backend
<source type='file' path='/dev/vm-state-tpm'/> element. No swtpm seeding is needed
-- libvirt/swtpm initialize the state file. The schema gains TPMBackend.Source
(*TPMSource{Type,Path}); the TPM converter emits it when virt-launcher detects the
device in the pod (mirroring the EFI NVRAMBlockDevice path, via a new
ConverterContext.TPMStateBlockDevice plumbed from manager.go's
isBlockDeviceFile(TPMBlockDevicePath) probe).
Migration relies on libvirt + swtpm persistent_state='yes' plus the swtpm
--migration stream handoff -- exactly as today's Filesystem-backed persistent TPM,
just on a block-backed file. No meta-marker handshake is added for the TPM device.
Filesystem-mode persistent TPM is unchanged: the vTPM state stays in the single
backend PVC's swtpm SubPath and no second PVC is created. The createPVC Block
guard now rejects only CBT (changed-block-tracking needs a directory of bitmaps);
the EFI+TPM combination is supported, each on its own raw device.
Co-authored-by: Claude <[email protected]>
Signed-off-by: Andrei Kvapil <[email protected]>
d5ab74d to
0d0e742
Compare
|
Update: this PR now also implements persistent vTPM on Block-mode backend storage. The branch is now two commits:
Validated end-to-end on a real DRBD/LINSTOR (RWX-Block) cluster, libvirt 11.10 / QEMU 10.1: both backend PVCs come up Block + RWX and bound, and the running domains carry Design note from runtime testing: in Block mode the varstore device is consumed directly as the OVMF pflash, which must fit the 8 MiB q35 firmware-flash window — so Block-mode backend PVCs are sized to 2Mi rather than the 10Mi Filesystem default. I'll update the VEP (kubevirt/enhancements#350) accordingly — vTPM moves into the initial implementation, with the two-device design. |
…r phase Reflect the kubevirt/kubevirt#18215 implementation: vTPM state lives on a second raw block device (/dev/vm-state-tpm) via swtpm's file:// backend, Block is rejected only for CBT (not TPM), and a Block varstore device must fit the 8 MiB q35 firmware-flash window. Move vTPM from Beta into the initial Alpha scope. Signed-off-by: Andrei Kvapil <[email protected]>
We already support RWO FS, including with migration, why do you say RWX is structurally required? |
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
@kvaps: The following test failed, say
DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/test pull-kubevirt-e2e-k8s-1.36-sig-network-smoke |
|
There has been no activity on this PR for 45 days. What you can do:
/lifecycle stale |
|
There has been no activity on this PR for 59 days and it has been stale for 14 days. What you can do:
/lifecycle rotten |
|
There has been no activity on this PR for 66 days and it has been rotten for 7 days. What you can do:
/close |
|
@kubevirt-bot: Closed this PR. DetailsIn response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
What this PR does / why we need it
KubeVirt stores persistent EFI/TPM state in a backend-storage PVC, today always
Filesystem. Such VMs are already live-migratable on ordinary RWO/RWX filesystem storage — on migration KubeVirt creates a fresh target PVC and copies the small state blob (#12629). So this is not a migration fix.This PR adds an optional
Blockvolume mode (KubeVirt.spec.configuration.vmStateVolumeMode: Filesystem|Block, default Filesystem) so persistent VM state can live natively on block storage (DRBD/LINSTOR, Ceph RBD) — on the same kind of raw volume the VM's disks already use, with no filesystem layer for the tens-of-MiB state blob. On RWX-Block the backend volume is shared by source and target during migration, so no per-migration target PVC or state copy is needed. It is additive and opt-in;Filesystem+ copy-on-migrate stays the default and is unchanged.Two commits:
<nvram type='block'><source dev=…>/ QEMU pflash.<tpm><backend persistent_state='yes'><source type='file' …>. Each raw device backs one blob, so a VM with both persistent EFI and TPM gets two backend PVCs.Blockis rejected only for CBT (multi-blob, needs a filesystem).Validated end-to-end on a real DRBD/LINSTOR (RWX-Block) cluster, libvirt 11.10 / QEMU 10.1: backend PVCs come up Block+RWX and bound; the running domains carry
<nvram type='block'>and the TPM<source type='file'>, with swtpm writing TPM state directly to the raw device.Design
createNamedPVChonorsvmStateVolumeMode; the PVC is rendered as a rawVolumeDevice(no SubPath). Block-mode PVCs are sized to fit the 8 MiB q35 firmware-flash window (a 10Mi varstore device makes QEMU refuse to start).<nvram>/<tpm>schema extended with the block/file source forms; the converter emits them.VEP: kubevirt/enhancements#349 (PR kubevirt/enhancements#350).
/kind feature
/area storage