Skip to content

backend-storage: persistent EFI NVRAM on a Block-mode (RWX) volume - #18215

Closed
kvaps wants to merge 2 commits into
kubevirt:mainfrom
kvaps:draft/block-nvram-efi
Closed

kvaps wants to merge 2 commits into
kubevirt:mainfrom
kvaps:draft/block-nvram-efi

Conversation

@kvaps

@kvaps kvaps commented Jun 22, 2026 •

Copy link
Copy Markdown
Member

What this PR does / why we need it

KubeVirt stores persistent EFI/TPM state in a backend-storage PVC, today always Filesystem. Such VMs are already live-migratable on ordinary RWO/RWX filesystem storage — on migration KubeVirt creates a fresh target PVC and copies the small state blob (#12629). So this is not a migration fix.

This PR adds an optional Block volume mode (KubeVirt.spec.configuration.vmStateVolumeMode: Filesystem|Block, default Filesystem) so persistent VM state can live natively on block storage (DRBD/LINSTOR, Ceph RBD) — on the same kind of raw volume the VM's disks already use, with no filesystem layer for the tens-of-MiB state blob. On RWX-Block the backend volume is shared by source and target during migration, so no per-migration target PVC or state copy is needed. It is additive and opt-in; Filesystem + copy-on-migrate stays the default and is unchanged.

Two commits:

  1. EFI NVRAM on Block — raw OVMF VARS pflash on the block device via libvirt <nvram type='block'><source dev=…> / QEMU pflash.
  2. vTPM on Block — swtpm's single-file backend on a separate raw block device via libvirt <tpm><backend persistent_state='yes'><source type='file' …>. Each raw device backs one blob, so a VM with both persistent EFI and TPM gets two backend PVCs. Block is rejected only for CBT (multi-blob, needs a filesystem).

Validated end-to-end on a real DRBD/LINSTOR (RWX-Block) cluster, libvirt 11.10 / QEMU 10.1: backend PVCs come up Block+RWX and bound; the running domains carry <nvram type='block'> and the TPM <source type='file'>, with swtpm writing TPM state directly to the raw device.

Design

  • backend-storage createNamedPVC honors vmStateVolumeMode; the PVC is rendered as a raw VolumeDevice (no SubPath). Block-mode PVCs are sized to fit the 8 MiB q35 firmware-flash window (a 10Mi varstore device makes QEMU refuse to start).
  • libvirt <nvram>/<tpm> schema extended with the block/file source forms; the converter emits them.
  • One-time template seed: libvirt auto-populates block nvram from a template only for qcow2 (≥10.8), not raw — so virt-launcher seeds the OVMF VARS template onto a blank device once (plain byte copy, no mkfs/mount/CAP_SYS_ADMIN). swtpm initializes its own state file.

VEP: kubevirt/enhancements#349 (PR kubevirt/enhancements#350).

/kind feature
/area storage

NVRAM/EFI and vTPM persistent state can optionally be stored on a Block-mode backend-storage volume (`vmStateVolumeMode: Block`), so VM state can live natively on block storage (e.g. DRBD/LINSTOR, Ceph RBD) alongside the VM's disks, without a filesystem layer.

@kubevirt-bot

Copy link
Copy Markdown
Contributor

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

@kubevirt-bot

Copy link
Copy Markdown
Contributor

@kvaps: The label(s) kind/feature, area/storage cannot be applied, because the repository doesn't have them.

Details

In response to this:

What this PR does / why we need it

Persistent EFI/TPM state is stored in a Filesystem-mode backend-storage PVC, which must be RWX-Filesystem to live-migrate. Block-replicated storage (DRBD/LINSTOR) provides RWX only at the block level, so persistent-EFI VMs there can't live-migrate.

This DRAFT/RFC lets the EFI NVRAM live directly on a raw Block backend volume — no filesystem — via libvirt <nvram type='block'> / QEMU pflash. The OVMF VARS then migrate through QEMU's normal pflash migration stream on a dual-primary RWX-Block volume, so persistent-EFI VMs can live-migrate on block-replicated storage without NFS/RWX-Filesystem. Mirrors long-standing block-device VM state in the OpenNebula LINSTOR driver.

Scope: EFI only. swtpm (vTPM) needs a directory of files and stays Filesystem-only; CBT likewise. vmStateVolumeMode: Block is rejected for VMs that also need persistent TPM/CBT.

Design

  • New KubeVirt.spec.configuration.vmStateVolumeMode (Filesystem|Block, default Filesystem).
  • backend-storage createPVC honors it; PVC rendered as a raw VolumeDevice (no SubPath).
  • libvirt <nvram> schema extended with type + <source dev=…>; converter emits the block form.
  • One-time template seed: libvirt auto-populates block nvram from template only for qcow2 (≥10.8), not raw — so the launcher seeds the OVMF VARS template onto a blank device once (plain byte copy, no mkfs/mount/CAP_SYS_ADMIN).
  • Migration: the Filesystem /meta/migrated marker handshake is skipped for Block.

Status / not yet validated (DRAFT/RFC for design feedback)

  • Non-cgo packages build + vet clean and unit tests pass; virt-launcher/virtwrap (manager.go) needs a Linux+libvirt+cgo build, not yet run.
  • Raw-block pflash migration-stream behavior on a dual-primary RWX-Block PVC is not yet validated on real KubeVirt+libvirt+DRBD.
  • make generate (openapi/CRD) still required.

Special notes for your reviewer: opening as draft to get design feedback before investing in e2e + migration validation.

/kind feature
/area storage

NVRAM/EFI persistent state can be stored on a Block-mode backend-storage volume (`vmStateVolumeMode: Block`), enabling live-migration of persistent-EFI VMs on block-replicated storage (e.g. DRBD/LINSTOR) without RWX-Filesystem storage.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubevirt-bot kubevirt-bot added release-note Denotes a PR that will be considered when it comes time to generate release notes. do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. dco-signoff: yes Indicates the PR's author has DCO signed all their commits. labels Jun 22, 2026

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, we are unable to review this pull request

The GitHub API does not allow us to fetch diffs exceeding 300 files, and this pull request has 328

@kubevirt-bot

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign rmohr for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kubevirt-bot kubevirt-bot added size/L and removed needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. size/XXL labels Jun 22, 2026
kvaps added a commit to cozystack/cozystack that referenced this pull request Jun 23, 2026
…migration (#3006)

## What this PR does

Temporary revert. v1.5.0 restored persistent EFI/TPM state for the
`windows.11` / `windows.2k22` / `windows.2k25` profiles. With
persistence on, KubeVirt creates a backend-storage PVC
(`persistent-state-for-<vm>`) which, on the default `replicated`
StorageClass, is **ReadWriteOnce** — it pins the VM to its node and
**blocks live-migration and node drains**. With `evictionStrategy:
LiveMigrate` this can stall cluster upgrades. See #3005.

This strips persistent EFI/TPM from the preferences again — but via `yq
del` instead of the original `sed -d`, so an emptied `preferredTPM`
stays `{}` (the v1.8 CRD rejects `null`, which is the bug the original
strip caused). **Secure Boot and the vTPM remain present, just
non-persistent** — Windows VMs still boot; they just don't persist
EFI/TPM across reboots.

⚠️ **Temporary stopgap** until vm-state can request an RWX backend —
either RWX-Block via the upstream KubeVirt change
(kubevirt/kubevirt#18215), or an RWX-Filesystem `vmStateStorageClass`.
Once that lands, persistence can be re-enabled with live-migration
intact.

### Screenshots

No UI changes.

### Release note

```release-note
fix(kubevirt-instancetypes): temporarily disable persistent EFI/TPM state to unblock live-migration of Windows VMs. Persistence created a ReadWriteOnce backend-storage PVC on the default storage class, pinning the VM to its node. Secure Boot and vTPM remain available (non-persistent); to be re-enabled once vm-state can use RWX storage.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Updated Windows virtual machine preferences to enable live migration
and drain operations
* Fixed compatibility issues with KubeVirt v1.8 virtual machine
preference configurations
* Removed persistent settings from TPM and EFI configurations to prevent
blocking behaviors on Windows 11 and Windows Server 2022/2025 variants

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
@kvaps

kvaps commented Jun 24, 2026

Copy link
Copy Markdown
Member Author

Thanks @mhenriks @jean-edouard! VEP opened: kubevirt/enhancements#349 (PR kubevirt/enhancements#350).

Quick answers to @jean-edouard's questions (full detail in the VEP):

  • Why Block, not RWO-FS: RWX is structurally required for live-migration — both launcher pods attach the backend during the migration window — so RWO-Filesystem doesn't help. The real choice is RWX-Block vs RWX-Filesystem. On DRBD/LINSTOR and Ceph RBD, RWX is native only in Block; RWX-Filesystem there means a per-volume NFS-Ganesha / CephFS layer.
  • TPM / do you plan to expand: yes. swtpm already has a single-file backend that targets a block device directly (--tpmstate backend-uri=file://<dev>), and libvirt exposes it via the TPM emulator <source type='file'> element (since 10.9), with state migrated through the swtpm migration stream (CMD_SET_STATEBLOB) + swtpm --migration release-lock-outgoing for the lock handoff. So vTPM-on-block is symmetric to EFI — a version-gated phase 2, not a redesign. EFI-only here is a temporary limitation, not permanent.

@mhenriks

Copy link
Copy Markdown
Member

Thanks @mhenriks @jean-edouard! VEP opened: kubevirt/enhancements#349 (PR kubevirt/enhancements#350).

Quick answers to @jean-edouard's questions (full detail in the VEP):

  • Why Block, not RWO-FS: RWX is structurally required for live-migration — both launcher pods attach the backend during the migration window — so RWO-Filesystem doesn't help. The real choice is RWX-Block vs RWX-Filesystem. On DRBD/LINSTOR and Ceph RBD, RWX is native only in Block; RWX-Filesystem there means a per-volume NFS-Ganesha / CephFS layer.

@kvaps are you aware of this: #12629?

  • TPM / do you plan to expand: yes. swtpm already has a single-file backend that targets a block device directly (--tpmstate backend-uri=file://<dev>), and libvirt exposes it via the TPM emulator <source type='file'> element (since 10.9), with state migrated through the swtpm migration stream (CMD_SET_STATEBLOB) + swtpm --migration release-lock-outgoing for the lock handoff. So vTPM-on-block is symmetric to EFI — a version-gated phase 2, not a redesign. EFI-only here is a temporary limitation, not permanent.

Add KubeVirt.spec.configuration.vmStateVolumeMode (Filesystem|Block, default
Filesystem). When Block, the vm-state backend PVC is rendered as a raw
VolumeDevice and the EFI NVRAM is stored directly on the block device via
libvirt <nvram type='block'> / QEMU pflash -- no filesystem, no mkfs/mount.

This lets persistent-EFI VMs live-migrate on block-replicated storage
(DRBD/LINSTOR), which provides RWX only at the block level: the OVMF VARS
travel in QEMU's pflash migration stream like any pflash device, so the
ReadWriteOnce-Filesystem pin that blocks live-migration today is avoided.

Block-mode persistent EFI is routed through the explicit-loader firmware path
rather than firmware auto-selection: auto-selection hides the OVMF VARS
template path, but virt-launcher needs it to seed the blank raw device once
(libvirt only auto-populates qcow2 nvram, not raw, since 10.8.0).

Scope is EFI only: swtpm (vTPM) and CBT need a directory of files and stay
Filesystem-only; Block vm-state is rejected for VMs that also need persistent
TPM or CBT. The Filesystem /meta/migrated handshake is skipped for Block.

Co-authored-by: Claude <[email protected]>
Signed-off-by: Andrei Kvapil <[email protected]>
@kvaps
kvaps force-pushed the draft/block-nvram-efi branch from 4b84769 to eae35fa Compare June 24, 2026 17:49
@kubevirt-prow kubevirt-prow Bot removed the size/L label Jun 24, 2026
@kubevirt-prow

kubevirt-prow Bot commented Jun 24, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign rmohr for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@kvaps
kvaps force-pushed the draft/block-nvram-efi branch 2 times, most recently from f52ec25 to d5ab74d Compare June 24, 2026 20:35
@kubevirt-prow kubevirt-prow Bot added size/XXL and removed size/XL labels Jun 24, 2026
Extend Block-mode backend storage (KubeVirt.spec.configuration.vmStateVolumeMode
= Block) to persistent vTPM, complementing the existing Block-mode persistent
EFI. In Block mode each raw device backs exactly one fixed-size state blob, so a
persistent-TPM VM gets a SECOND backend-storage PVC alongside the EFI NVRAM one:
persistent-tpm-state-for-<vm> (label persistent-tpm-state-for: <vmName>), Block
mode, same access-mode negotiation as the EFI PVC, attached as a raw VolumeDevice
at /dev/vm-state-tpm.

swtpm backs its state file directly with the raw device via its file:// backend
(swtpm >= 0.7); libvirt (>= 10.9) exposes this through the TPM emulator backend
<source type='file' path='/dev/vm-state-tpm'/> element. No swtpm seeding is needed
-- libvirt/swtpm initialize the state file. The schema gains TPMBackend.Source
(*TPMSource{Type,Path}); the TPM converter emits it when virt-launcher detects the
device in the pod (mirroring the EFI NVRAMBlockDevice path, via a new
ConverterContext.TPMStateBlockDevice plumbed from manager.go's
isBlockDeviceFile(TPMBlockDevicePath) probe).

Migration relies on libvirt + swtpm persistent_state='yes' plus the swtpm
--migration stream handoff -- exactly as today's Filesystem-backed persistent TPM,
just on a block-backed file. No meta-marker handshake is added for the TPM device.

Filesystem-mode persistent TPM is unchanged: the vTPM state stays in the single
backend PVC's swtpm SubPath and no second PVC is created. The createPVC Block
guard now rejects only CBT (changed-block-tracking needs a directory of bitmaps);
the EFI+TPM combination is supported, each on its own raw device.

Co-authored-by: Claude <[email protected]>
Signed-off-by: Andrei Kvapil <[email protected]>
@kvaps
kvaps force-pushed the draft/block-nvram-efi branch from d5ab74d to 0d0e742 Compare June 24, 2026 20:58
@kvaps

kvaps commented Jun 25, 2026

Copy link
Copy Markdown
Member Author

Update: this PR now also implements persistent vTPM on Block-mode backend storage.

The branch is now two commits:

  1. EFI NVRAM on Block (original).
  2. vTPM on Block — swtpm's single-file backend on a separate raw block device (/dev/vm-state-tpm), exposed to libvirt as <tpm><backend persistent_state='yes'><source type='file' .../></backend>. Each raw device backs exactly one state blob, so a VM with both persistent EFI and TPM gets two backend PVCs. Block is now rejected only for CBT (not TPM).

Validated end-to-end on a real DRBD/LINSTOR (RWX-Block) cluster, libvirt 11.10 / QEMU 10.1: both backend PVCs come up Block + RWX and bound, and the running domains carry <nvram type='block'><source dev='/dev/vm-state'/> and the TPM <source type='file' path='/dev/vm-state-tpm'/>, with swtpm writing TPM state directly to the raw device.

Design note from runtime testing: in Block mode the varstore device is consumed directly as the OVMF pflash, which must fit the 8 MiB q35 firmware-flash window — so Block-mode backend PVCs are sized to 2Mi rather than the 10Mi Filesystem default.

I'll update the VEP (kubevirt/enhancements#350) accordingly — vTPM moves into the initial implementation, with the two-device design.

kvaps added a commit to kvaps/enhancements that referenced this pull request Jun 25, 2026
…r phase

Reflect the kubevirt/kubevirt#18215 implementation: vTPM state lives on a
second raw block device (/dev/vm-state-tpm) via swtpm's file:// backend, Block
is rejected only for CBT (not TPM), and a Block varstore device must fit the
8 MiB q35 firmware-flash window. Move vTPM from Beta into the initial Alpha
scope.

Signed-off-by: Andrei Kvapil <[email protected]>
@jean-edouard

Copy link
Copy Markdown
Contributor
  • Why Block, not RWO-FS: RWX is structurally required for live-migration — both launcher pods attach the backend during the migration window — so RWO-Filesystem doesn't help. The real choice is RWX-Block vs RWX-Filesystem. On DRBD/LINSTOR and Ceph RBD, RWX is native only in Block; RWX-Filesystem there means a per-volume NFS-Ganesha / CephFS layer.

We already support RWO FS, including with migration, why do you say RWX is structurally required?

@kubevirt-bot kubevirt-bot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 2, 2026
@kubevirt-bot

Copy link
Copy Markdown
Contributor

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@kubevirt-prow

kubevirt-prow Bot commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

@kvaps: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
pull-kubevirt-e2e-kind-1.36-sig-compute-arm64 0d0e742 link true /test pull-kubevirt-e2e-kind-1.36-sig-compute-arm64
Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@kubevirt-bot

Copy link
Copy Markdown
Contributor

/test pull-kubevirt-e2e-k8s-1.36-sig-network-smoke

@kubevirt-bot

Copy link
Copy Markdown
Contributor

There has been no activity on this PR for 45 days.
To protect limited CI resources, it has been automatically labelled 'stale'.
This PR will automatically rot after an additional 14 days of inactivity, and will be closed shortly after that.

What you can do:

  • If the PR is waiting on you to respond to a question or feedback and/or update the PR, please do so.
  • You can mark the PR as fresh and remove the label with the following command: /remove-lifecycle stale
  • If this PR is safe to close now, please help the project by closing it with: /close
  • If you need attention on this PR from a reviewer, you can raise it on the agenda of the relevant SIG meeting or KubeVirt Community meeting, or ping the kubevirt-dev slack channel.

/lifecycle stale

@kubevirt-prow kubevirt-prow Bot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Oct 17, 2026
@kubevirt-bot

Copy link
Copy Markdown
Contributor

There has been no activity on this PR for 59 days and it has been stale for 14 days.
To protect limited CI resources, it has been automatically labelled 'rotten' and will be closed in 7 days.

What you can do:

  • If the PR is waiting on you to respond to a question or feedback and/or update the PR, please do so.
  • You can mark the PR as fresh and remove the label with the following command: /remove-lifecycle rotten
  • If this PR is safe to close now, please help the project by closing it with: /close
  • If you need attention on this PR from a reviewer, you can raise it on the agenda of the relevant SIG meeting or KubeVirt Community meeting, or ping the kubevirt-dev slack channel.

/lifecycle rotten

@kubevirt-prow kubevirt-prow Bot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Oct 31, 2026
@kubevirt-bot

Copy link
Copy Markdown
Contributor

There has been no activity on this PR for 66 days and it has been rotten for 7 days.
To protect limited CI resources, it has been automatically closed.

What you can do:

/close

@kubevirt-prow kubevirt-prow Bot closed this Nov 7, 2026
@kubevirt-prow

kubevirt-prow Bot commented Nov 7, 2026

Copy link
Copy Markdown
Contributor

@kubevirt-bot: Closed this PR.

Details

In response to this:

There has been no activity on this PR for 66 days and it has been rotten for 7 days.
To protect limited CI resources, it has been automatically closed.

What you can do:

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/controller area/handler area/launcher area/operator area/virtctl dco-signoff: yes Indicates the PR's author has DCO signed all their commits. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. release-note Denotes a PR that will be considered when it comes time to generate release notes. sig/buildsystem Denotes an issue or PR that relates to changes in the build system. sig/compute sig/control-plane sig/network sig/observability Denotes an issue or PR that relates to observability. sig/scale sig/storage size/XXL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants