Add an option to control nested virtualization #7408

russell-islam · 2025-10-09T20:04:36Z

No description provided.

liuw · 2025-10-09T22:35:59Z

I assume there isn't an option for KVM?

We should take into consideration we may want to provide very fine-grained configurations in the medium term.

Putting this into PlatformConfig looks wrong. This is a CPU feature, not a platform feature.

russell-islam · 2025-10-09T22:44:50Z

I assume there isn't an option for KVM?

We should take into consideration we may want to provide very fine-grained configurations in the medium term.

Putting this into PlatformConfig looks wrong. This is a CPU feature, not a platform feature.

We can directly add to the VM configuration as features or something?

liuw · 2025-10-10T18:31:53Z

I assume there isn't an option for KVM?
We should take into consideration we may want to provide very fine-grained configurations in the medium term.
Putting this into PlatformConfig looks wrong. This is a CPU feature, not a platform feature.

We can directly add to the VM configuration as features or something?

There is a workstream to expose fine-grained control over the CPU features. I think this belongs there.

KVM can benefit from the fine-grained control.

phip1611 · 2025-10-13T20:25:39Z

For the record: with current Cloud Hypervisor on Linux/KVM, nesting works out of the box including state save/resume and live-migration. It should work similar with MSHV.

russell-islam · 2025-10-13T20:45:34Z

For the record: with current Cloud Hypervisor on Linux/KVM, nesting works out of the box including state save/resume and live-migration. It should work similar with MSHV.
On MSHV without that CPU feature enablement it would not work.

phip1611 · 2025-10-14T08:41:46Z

We could add a runtime configuration for kvm similar to MSHV which disables/enables the support explicitly. Then we can have similar behavior for both backends - and disabling nesting is a valid use case in case infrastructure maintainers are concerned about people breaking out of the VMX emulation for example

anirudhrb · 2025-10-16T07:02:12Z

vmm/src/vm_config.rs

    #[cfg(target_arch = "x86_64")]
    #[serde(default)]
    pub amx: bool,
+    pub nested: bool,


How about nested_virt instead of just nested?

vmm/src/vm_config.rs

vmm/src/vm.rs

anirudhrb · 2025-10-16T08:52:51Z

nit: there are multiple typos in the commit messages.

russell-islam · 2025-10-17T20:04:49Z

@anirudhrb Please take another look. Thx

rbradford

We don't have a nested option for KVM - it's always enabled. Why do we need to make it an option for MSHV?

If you add the option to --cpu for MSHV then you need to do the same for KVM.

russell-islam · 2025-10-22T16:57:47Z

We don't have a nested option for KVM - it's always enabled. Why do we need to make it an option for MSHV?

If you add the option to --cpu for MSHV then you need to do the same for KVM.

With MSHV we have to pass the flag during VM creations. The customers want to control that flow with CLI, that's why I raised the PR.

Do you recommend adding feature guard around the nested feature, or recommend adding an extra argument like --mshv-cpu-feaures?

rbradford · 2025-10-22T17:51:46Z

We don't have a nested option for KVM - it's always enabled. Why do we need to make it an option for MSHV?
If you add the option to --cpu for MSHV then you need to do the same for KVM.

With MSHV we have to pss the flag during VM creations. THe customers wants to control that flow with CLI, that's why I raised the PR.

Do you recommend adding feature guard around the nested feature, or recommend adding an extra argument like --mshv-cpu-feaures?

You haven't explained why you can't have it turned on all the time.

russell-islam · 2025-10-22T21:54:46Z

We don't have a nested option for KVM - it's always enabled. Why do we need to make it an option for MSHV?
If you add the option to --cpu for MSHV then you need to do the same for KVM.

With MSHV we have to pss the flag during VM creations. THe customers wants to control that flow with CLI, that's why I raised the PR.
Do you recommend adding feature guard around the nested feature, or recommend adding an extra argument like --mshv-cpu-feaures?

You haven't explained why you can't have it turned on all the time.

Customers does not wnat to turn on by default.

We don't have a nested option for KVM - it's always enabled. Why do we need to make it an option for MSHV?
If you add the option to --cpu for MSHV then you need to do the same for KVM.

With MSHV we have to pss the flag during VM creations. THe customers wants to control that flow with CLI, that's why I raised the PR.
Do you recommend adding feature guard around the nested feature, or recommend adding an extra argument like --mshv-cpu-feaures?

You haven't explained why you can't have it turned on all the time.

We can turn it on by default with no issue but customer does not want it default rather it wants to control in run-time. Do you also refer to turn it on by default with the CpuFeatures default initialization? Also in future we will have more Cpu features on MSVH that might need to be turned on the CLI.

rbradford · 2025-10-23T05:49:47Z

Okay. If you want to make it controllable then I think you should make it controllable for KVM as well - with a default of on.

russell-islam · 2025-10-23T17:05:06Z

Okay. If you want to make it controllable then I think you should make it controllable for KVM as well - with a default of on.

Just curious, If we add some features that is only available for MSHV, How do we do it?

rbradford · 2025-10-24T09:20:43Z

Okay. If you want to make it controllable then I think you should make it controllable for KVM as well - with a default of on.

Just curious, If we add some features that is only available for MSHV, How do we do it?

Probably something like x-mshv- prefix

phip1611 · 2025-10-27T10:36:44Z

Why not add a parameter to --cpu, such as: --cpu nesting={on,off}? This is generic. I think in the KVM case we can also set the default to false, we probably just have to remove a few CPUid entries in that case.

with a default of on.

For the record: Some operators are concerned about the increased attack surface of the KVM VMX emulator code.

rbradford · 2025-10-27T11:49:33Z

Why not add a parameter to --cpu, such as: --cpu nesting={on,off}? This is generic. I think in the KVM case we can also set the default to false, we probably just have to remove a few CPUid entries in that case.

Exactly what I was getting at!

with a default of on.

For the record: Some operators are concerned about the increased attack surface of the KVM VMX emulator code.

We've already shipped with a default of on - so we can't change that now.

hypervisor/src/lib.rs

hypervisor/src/mshv/mod.rs

vmm/src/cpu.rs

rbradford · 2025-11-26T15:53:41Z

vmm/src/vm_config.rs

            self.cpus.max_vcpus
        }
    }
+    pub(crate) fn to_hypervisor_vm_config(&self) -> hypervisor::HypervisorVmConfig {


Would it be more Rust idiomatic to do it like this:

impl From<&VmConfig> for hypervisor::HypervisorVmConfig { fn from(value: &VmConfig) -> Self { hypervisor::HypervisorVmConfig { #[cfg(feature = "tdx")] tdx_enabled: value.platform.as_ref().map(|p| p.tdx).unwrap_or(false), #[cfg(feature = "sev_snp")] sev_snp_enabled: value.is_sev_snp_enabled(), #[cfg(feature = "sev_snp")] mem_size: self.memory.total_size(), nested: value.cpus.nested, } } }

and access it like

let hypervisor_vm_config = self .vm_config .as_ref() .unwrap() .lock() .unwrap() .deref() .into();

Moved to lib.rs to avoid the following error:
implementation is not supported in traits or impls
consider moving the implementation out to a nearby module scope rustc
#file://cloud-hypervisor/vmm/src/vm_config.rs)
hypervisor
pub struct HypervisorVmConfig {}
HypervisorVmConfig is defined in Hypervisor crate.

rbradford

Other than that one suggestion about Rust style I think this PR is ready to go.

rbradford

docs/cpu.md needs an update.

Create HypervisorVmConfig early and pass the struct to VM creation API in the vmm crate. Getting rid of multiple conditional parameter. Signed-off-by: Muminul Islam <[email protected]>

Use mshv-{ioctls, bindings) with the latest versions that have the nested MSHV support. Signed-off-by: Muminul Islam <[email protected]>

Add an option in the CLI to enable nested support. Adding an option --cpu nested=on|off to the CPU argument to enable nested support for Microsoft Hypervisor. By default it is enabled for both KVM and MSHV. Signed-off-by: Muminul Islam <[email protected]>

rbradford

Docs need clarifications.

docs/cpu.md

rbradford · 2025-12-01T21:48:13Z

docs/cpu.md

+running containerized workloads that require virtualization features within the
+guest OS.
+
+Note that nested virtualization may impact performance and should only be enabled


What evidence do you have that an Ln VM has worse performance on KVM when nested is turned on vs off? I know that Ln+1 is probably going to peform worse than Ln.

One layer of virtualization may impact performance, not? I can remove this statement at all.

But that's not what your saying in this paragraph - you're saying that the performance of this layer is affected. I acknowledge that higher layers will have worse performance but I don't have evidence that setting the CPUID fields to enable nesting on this layer will affect this layer.

I think this whole section is too editorialised. It can simply be "Enable nested virtualization (default on). Nested virtualization is needed to access hardware virtualization by this guest. This option can only be changed on x86-64."

Updated. Thank you

This patch connects the VMM and hypervisor to fully enable nested support for MSHV. Signed-off-by: Muminul Islam <[email protected]>

User can now disable nested virtualization for Intel and AMD if configured by the CLI. Signed-off-by: Muminul Islam <[email protected]>

This patch updates the documentation to reflect the newly added nested CPU feature option in the CLI. Signed-off-by: Muminul Islam <[email protected]>

rbradford

Thank you for your patience on this PR! We do need to be especially careful with adding APIs and things could affect our defaults.

russell-islam · 2025-12-05T00:01:39Z

Thank you for your patience on this PR! We do need to be especially careful with adding APIs and things could affect our defaults.

Thank you @rbradford for all your effort on this. Your thorough review was very helpful and some learning for me.

russell-islam requested a review from a team as a code owner October 9, 2025 20:04

russell-islam marked this pull request as draft October 9, 2025 20:20

russell-islam force-pushed the muislam/mshv-nested-feature branch from 3c26210 to f2847cb Compare October 9, 2025 20:30

russell-islam force-pushed the muislam/mshv-nested-feature branch from f2847cb to 1920b79 Compare October 10, 2025 20:14

russell-islam marked this pull request as ready for review October 13, 2025 20:45

anirudhrb reviewed Oct 16, 2025

View reviewed changes

russell-islam force-pushed the muislam/mshv-nested-feature branch from 1920b79 to 187ce24 Compare October 16, 2025 20:08

anirudhrb approved these changes Oct 20, 2025

View reviewed changes

rbradford requested changes Oct 22, 2025

View reviewed changes

russell-islam force-pushed the muislam/mshv-nested-feature branch 5 times, most recently from e9d670a to f9107db Compare October 30, 2025 20:54

sboeuf reviewed Nov 19, 2025

View reviewed changes

hypervisor/src/lib.rs Show resolved Hide resolved

sboeuf reviewed Nov 19, 2025

View reviewed changes

hypervisor/src/mshv/mod.rs Show resolved Hide resolved

sboeuf reviewed Nov 19, 2025

View reviewed changes

vmm/src/cpu.rs Show resolved Hide resolved

russell-islam force-pushed the muislam/mshv-nested-feature branch from f9a65b5 to ca35fa8 Compare November 19, 2025 20:48

rbradford reviewed Nov 26, 2025

View reviewed changes

russell-islam force-pushed the muislam/mshv-nested-feature branch 3 times, most recently from 140bb2f to 17fe8ed Compare November 26, 2025 22:04

rbradford reviewed Nov 27, 2025

View reviewed changes

liuw approved these changes Nov 28, 2025

View reviewed changes

russell-islam added 3 commits December 1, 2025 12:05

vmm: simplify VM creation API

24b7159

Create HypervisorVmConfig early and pass the struct to VM creation API in the vmm crate. Getting rid of multiple conditional parameter. Signed-off-by: Muminul Islam <[email protected]>

build: use latest 0.6.5 mshv crates

879bd31

Use mshv-{ioctls, bindings) with the latest versions that have the nested MSHV support. Signed-off-by: Muminul Islam <[email protected]>

russell-islam force-pushed the muislam/mshv-nested-feature branch from 17fe8ed to d84129a Compare December 1, 2025 20:05

rbradford requested changes Dec 1, 2025

View reviewed changes

russell-islam force-pushed the muislam/mshv-nested-feature branch from d84129a to a92eb90 Compare December 2, 2025 03:08

rbradford changed the title ~~Add an option to enable nested virtualization support on MSHV~~ Add an option to control nested virtualization Dec 2, 2025

russell-islam added 3 commits December 2, 2025 22:26

hypervisor: vmm: passing nested argument from VMM to hypervisor

316daef

This patch connects the VMM and hypervisor to fully enable nested support for MSHV. Signed-off-by: Muminul Islam <[email protected]>

arch: vmm: disable nested virtualization if needed

9fb7502

User can now disable nested virtualization for Intel and AMD if configured by the CLI. Signed-off-by: Muminul Islam <[email protected]>

docs: update guide for nested CPU feature

a5e7e7b

This patch updates the documentation to reflect the newly added nested CPU feature option in the CLI. Signed-off-by: Muminul Islam <[email protected]>

russell-islam force-pushed the muislam/mshv-nested-feature branch from a92eb90 to a5e7e7b Compare December 3, 2025 06:29

rbradford approved these changes Dec 3, 2025

View reviewed changes

rbradford added this pull request to the merge queue Dec 3, 2025

Merged via the queue into cloud-hypervisor:main with commit 5fb50ed Dec 3, 2025
44 of 45 checks passed

likebreath added this to Cloud Hypervisor Roadmap Dec 11, 2025

github-project-automation bot moved this to 🆕 New in Cloud Hypervisor Roadmap Dec 11, 2025

github-project-automation bot moved this from 🆕 New to ✅ Done in Cloud Hypervisor Roadmap Dec 11, 2025

Add an option to control nested virtualization #7408

Add an option to control nested virtualization #7408

Uh oh!

Conversation

russell-islam commented Oct 9, 2025

Uh oh!

liuw commented Oct 9, 2025

Uh oh!

russell-islam commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

liuw commented Oct 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

phip1611 commented Oct 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

russell-islam commented Oct 13, 2025

Uh oh!

phip1611 commented Oct 14, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

anirudhrb Oct 16, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

anirudhrb commented Oct 16, 2025

Uh oh!

russell-islam commented Oct 17, 2025

Uh oh!

rbradford left a comment

Choose a reason for hiding this comment

Uh oh!

russell-islam commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rbradford commented Oct 22, 2025

Uh oh!

russell-islam commented Oct 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rbradford commented Oct 23, 2025

Uh oh!

russell-islam commented Oct 23, 2025

Uh oh!

rbradford commented Oct 24, 2025

Uh oh!

phip1611 commented Oct 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rbradford commented Oct 27, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rbradford Nov 26, 2025

Choose a reason for hiding this comment

Uh oh!

russell-islam Nov 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rbradford left a comment

Choose a reason for hiding this comment

Uh oh!

rbradford left a comment

Choose a reason for hiding this comment

Uh oh!

rbradford left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

rbradford Dec 1, 2025

Choose a reason for hiding this comment

Uh oh!

russell-islam Dec 2, 2025

Choose a reason for hiding this comment

Uh oh!

rbradford Dec 2, 2025

russell-islam commented Oct 9, 2025 •

edited

Loading

liuw commented Oct 10, 2025 •

edited

Loading

phip1611 commented Oct 13, 2025 •

edited

Loading

phip1611 commented Oct 14, 2025 •

edited

Loading

russell-islam commented Oct 22, 2025 •

edited

Loading

russell-islam commented Oct 22, 2025 •

edited

Loading

phip1611 commented Oct 27, 2025 •

edited

Loading

russell-islam Nov 26, 2025 •

edited

Loading

russell-islam Dec 3, 2025 •

edited

Loading