devices: Add fw_cfg device #7117

AlexOrozco1256 · 2025-06-04T17:36:12Z

We are working with the project Oak team to enable running Oak Containers on cloud hypervisor. These containers are SEV-SNP enlightened and can be used alongside with the rest of the oak suite for attestation and measurement of the cvm.

Here is the functionality we are adding to CHV.

Implmenting the sev-snp APIs previously defined (sev_snp_init, import_isolated_pages, complete_isolated_import).
Expanded the igvm loader to support booting KVM + SEV-SNP enabled VMs. (Note that we can also use the igvm loader to boot VMs with standard cloud images with/without sev-snp enabled as seen here (https://github.com/AlexOrozco1256/chv-demo)
Added a fw_cfg device with DMA enabled. Oak's Stage0 firmware uses fw_cfg for loading various parameters into the VM.

Additional design/implementation details can be found here: https://tinyurl.com/chv-kvm-sev-snp

See: #6653

AlexOrozco1256 · 2025-06-04T17:36:19Z

Here is how I tested the fw_cfg device. These are the same steps as the direct kernel boot instructions in the README for the chv repo.

The only difference is that we must enable CONFIG_FW_CFG_SYSFS in the cloud hypervisor linux kernel.

Here is how to build and run the vm with fw_cfg device.

git clone --depth 1 https://github.com/cloud-hypervisor/linux.git -b ch-6.12.8 linux-cloud-hypervisor
### ENABLE CONFIG_FW_CFG_SYSFS in arch/x86/configs/ch_defconfig
pushd linux-cloud-hypervisor
make ch_defconfig
KCFLAGS="-Wa,-mx86-used-note=no" make bzImage -j `nproc`
popd

wget https://cloud-images.ubuntu.com/oracular/current/oracular-server-cloudimg-amd64.img
qemu-img convert -p -f qcow2 -O raw oracular-server-cloudimg-amd64.img oracular-server-cloudimg-amd64.raw

echo "Hello from fw_cfg!" > /tmp/fw_cfg_test_item

git clone https://github.com/AlexOrozco1256/chv-g.git -b pr-fw_cfg
pushd chv-g
cargo build --features fw_cfg
./scripts/create-cloud-init.sh
popd

./chv-g/target/debug/cloud-hypervisor \
	--kernel linux-cloud-hypervisor/arch/x86/boot/compressed/vmlinux.bin \
	--disk path=oracular-server-cloudimg-amd64.raw path=/tmp/ubuntu-cloudinit.img\
	--cmdline "console=hvc0 root=/dev/vda1 rw" \
	--cpus boot=4 \
	--memory size=4G \
        --fw-cfg-config initramfs=off,items=[name=opt/org.test/fw_cfg_test_item,file=/tmp/fw_cfg_test_item] \
	--net "tap=,mac=,ip=,mask="

Now log into the vm with:
User: cloud
Password:cloud123

sudo cat /sys/firmware/qemu_fw_cfg/by_name/opt/org.test/fw_cfg_test_item/raw
Hello from fw_cfg!

rbradford · 2025-06-10T13:11:56Z

@AlexOrozco1256 Please can you rebase to get rid of the merge commit?

AlexOrozco1256 · 2025-06-10T17:58:33Z

@AlexOrozco1256 Please can you rebase to get rid of the merge commit?

Done.

rbradford

I'm happy to go ahead with this (with those logging nits fixed) - i've mainly focused my review time on the generic changes.

Thank you so much for your patience!

Let's also think about the route to making this non-feature flag so that we can use it for booting a (less modified) edk2.

vmm/src/config.rs

vmm/src/acpi.rs

vmm/src/device_manager.rs

rbradford

Still lgtm! Thanks.

AlexOrozco1256 · 2025-06-10T20:21:33Z

I'm happy to go ahead with this (with those logging nits fixed) - i've mainly focused my review time on the generic changes.

Thank you so much for your patience!

Let's also think about the route to making this non-feature flag so that we can use it for booting a (less modified) edk2.

Thanks for the review! I will look into edk2 to see what else besides fw_cfg is missing.

likebreath

Thank you for the good work. Besides comments below, couple other thoughts:

Please extend our GH workflows (mostly quality.yaml and build.yaml) to cover the fw_cfg feature flag;
Please add a documentation to explain what it does and how to use it;

Also, looks like adding CONFIG_FW_CFG_SYSFS to the guest kernel config is the only missing bit to add an integration test on fw_cfg, as what you described here.

If that's the case, It'd be great to have that integration test added, and I can help with adding the missing kernel config to reference kernel being used by our CI. (This is not required, since we don't require integration test for features behind feature flags).

devices/src/legacy/mod.rs

vmm/src/device_manager.rs

vmm/src/vm.rs

devices/src/legacy/fw_cfg.rs

likebreath · 2025-06-10T23:20:18Z

devices/src/legacy/fw_cfg.rs

+        #[cfg(target_arch = "x86_64")]
+        let mut mem_regions = vec![
+            (GuestAddress(0), EBDA_START.0 as usize, RegionType::Ram),
+            (
+                MEM_32BIT_DEVICES_START,
+                MEM_32BIT_DEVICES_SIZE as usize,
+                RegionType::Reserved,
+            ),
+            (
+                PCI_MMCONFIG_START,
+                PCI_MMCONFIG_SIZE as usize,
+                RegionType::Reserved,
+            ),
+            (STAGE0_START_ADDRESS, STAGE0_SIZE, RegionType::Reserved),
+        ];
+        #[cfg(target_arch = "aarch64")]
+        let mut mem_regions = arch::aarch64::arch_memory_regions();
+        if mem_size < MEM_32BIT_DEVICES_START.0 as usize {
+            mem_regions.push((
+                HIGH_RAM_START,
+                mem_size - HIGH_RAM_START.0 as usize,
+                RegionType::Ram,
+            ));
+        } else {
+            mem_regions.push((
+                HIGH_RAM_START,
+                MEM_32BIT_RESERVED_START.0 as usize - HIGH_RAM_START.0 as usize,
+                RegionType::Ram,
+            ));
+            mem_regions.push((
+                RAM_64BIT_START,
+                mem_size - (MEM_32BIT_DEVICES_START.0 as usize),
+                RegionType::Ram,
+            ));
+        }


We also calculate the e820 maps from couple of other places, say from configure_pvh and configure_32bit_entry on x86 (not sure what's done on aarch64). I wonder if it would be beneficial to refactor related code so that they can be reused?

@rbradford You know this area much better than me.

Good point but might be tricky as there are lots of very similar but subtly different formats for conveying the same information about e.g. the memory holes.

@likebreath

@likebreath found some good gaps in my review.

AlexOrozco1256 · 2025-06-12T20:59:27Z

Thank you for the good work. Besides comments below, couple other thoughts:

Please extend our GH workflows (mostly quality.yaml and build.yaml) to cover the fw_cfg feature flag;

Please add a documentation to explain what it does and how to use it;

Also, looks like adding CONFIG_FW_CFG_SYSFS to the guest kernel config is the only missing bit to add an integration test on fw_cfg, as what you described here.

If that's the case, It'd be great to have that integration test added, and I can help with adding the missing kernel config to reference kernel being used by our CI. (This is not required, since we don't require integration test for features behind feature flags).

I added an integration test that does exactly what my comment here was doing. I use direct kernel boot to boot a vm and verify that the item I added to the fw_cfg device is accessible from within the guest and has the correct contents.

I would just need help from @likebreath to enable the CONFIG_FW_CFG_SYSFS option in the reference kernel.

likebreath · 2025-06-12T22:22:35Z

I would just need help from @likebreath to enable the CONFIG_FW_CFG_SYSFS option in the reference kernel.

Of course. I will try to get to it tomorrow. Thanks for the quick turn-around.

This enables integration test of the `fw_cfg` device [1]. [1] cloud-hypervisor/cloud-hypervisor#7117 Signed-off-by: Bo Chen <[email protected]>

likebreath · 2025-06-13T23:33:04Z

I would just need help from @likebreath to enable the CONFIG_FW_CFG_SYSFS option in the reference kernel.

Of course. I will try to get to it tomorrow. Thanks for the quick turn-around.

@AlexOrozco1256 Okay. New kernel release was out, plus changes to consume it #7137. I also pushed few commits to kick-off the added fw_cfg tests here. Let's rebase and squash the couple commits into the integration test commit later.

FWIW, I ran the test locally on a aarch64 machine, and found the test failed. I looked into the guest and found the "/sys/firmware/qemu_fw_cfg/by_name/opt/org.test/fw_cfg_test_item/raw" is missing while "/sys/firmware/qemu_fw_cfg/". I believe the arm64 worker will fail for the same reason. PTAL. Thanks.

likebreath

Some comments below around CLI, which should all be relatively easy to resolve.

I believe we are very close to land this large PR, once #7226 is landed (and you can rebase). Thanks again for the persistency.

vmm/src/config.rs

vmm/src/vm.rs

vmm/src/vm_config.rs

vmm/src/vm.rs

likebreath · 2025-07-30T22:46:57Z

vmm/src/vm.rs

+                .map(|p| p.kernel.as_ref().map(File::open))
+                .unwrap_or_default()
+                .transpose()
+                .map_err(Error::MissingFwCfgKernelFile)?;


It should be handled in a central place for the validation of CLI options for the combination of payload. See my comments from #7166.

I just want to call it out here and let's leave it for future cleanup once #7166 is landed when @phip1611 is back.

vmm/src/config.rs

AlexOrozco1256 · 2025-08-07T15:33:37Z

last force push:

fixed clippy (changed !.is_some() -> .is_none())
added fw_cfg to vmm::feature_list()

likebreath

LGTM. Only two minor comments added. Can you please rebase? I think we are ready to merge.

I was hoping to get the new "linux-loader" release, but that's alright to not include that.

vmm/src/acpi.rs

AlexOrozco1256 · 2025-08-08T23:50:32Z

I was hoping to get the new "linux-loader" release, but that's alright to not include that.

I can fix this in a follow-up PR. Thanks for the review! I really appreciate it

Here we add the fw_cfg device as a legacy device to the device manager. It is guarded behind a fw_cfg flag in vmm at creation of the DeviceManager. In this cl we implement the fw_cfg device with one function (signature). Signed-off-by: Alex Orozco <[email protected]>

The kernel and initramfs are passed to the fw_cfg device as file references. The cmdline is passed directly. Signed-off-by: Alex Orozco <[email protected]>

We build the memory map in the fw_cfg device based on the memory size. Signed-off-by: Alex Orozco <[email protected]>

The acpi tables are created in the same place the acpi tables would be created for the regular bootflow, except here we add them to the fw_cfg device to be measured by the fw and then the fw will put the acpi tables into memory. Signed-off-by: Alex Orozco <[email protected]>

We pass a reference to the guest memory when we create the device in DeviceManager. This allows us to access the guest memory for DMA. Signed-off-by: Alex Orozco <[email protected]>

This allows the fw_cfg device to be recognized by the guest linux kernel. This becomes more relavnt in the following cl where I add the option to load files into the guest via fw_cfg. The Linux kernel already has a fw_cfg driver that will automatically load these files under /sys when CONFIG_FW_CFG_SYSFS is enabled in the kernel config For arm we must add fw_cfg to the devices tree Signed-off-by: Alex Orozco <[email protected]>

This allows us to enable/disable the fw_cfg device via the cli We can also now upload files into the guest vm using fw_cfg_items via the cli Signed-off-by: Alex Orozco <[email protected]>

This test verifies that we can see custom items added to the fw_cfg device from inside the guest Signed-off-by: Alex Orozco <[email protected]>

AlexOrozco1256 · 2025-08-10T20:33:47Z

last push I:

rebased to get the lychee lint fix ci: Improve link checker conditions #7247

likebreath

Thanks again for the good work and persistency. We are now ready to land it. :)

AlexOrozco1256 requested a review from a team as a code owner June 4, 2025 17:36

AlexOrozco1256 mentioned this pull request Jun 4, 2025

devices: Add fw_cfg device #7109

Closed

AlexOrozco1256 force-pushed the pr-fw_cfg branch from 2196905 to 56caeef Compare June 10, 2025 17:57

rbradford approved these changes Jun 10, 2025

View reviewed changes

vmm/src/config.rs Outdated Show resolved Hide resolved

vmm/src/acpi.rs Outdated Show resolved Hide resolved

vmm/src/device_manager.rs Outdated Show resolved Hide resolved

vmm/src/device_manager.rs Outdated Show resolved Hide resolved

AlexOrozco1256 force-pushed the pr-fw_cfg branch from 56caeef to eae2db8 Compare June 10, 2025 18:50

rbradford previously approved these changes Jun 10, 2025

View reviewed changes

likebreath self-requested a review June 10, 2025 19:23

AlexOrozco1256 force-pushed the pr-fw_cfg branch from eae2db8 to e94054b Compare June 10, 2025 20:34

likebreath reviewed Jun 10, 2025

View reviewed changes

AlexOrozco1256 force-pushed the pr-fw_cfg branch 5 times, most recently from c7a4b63 to 263743f Compare June 12, 2025 18:11

likebreath mentioned this pull request Jun 13, 2025

build: Enable CONFIG_FW_CFG_SYSFS_CMDLINE as default configs cloud-hypervisor/linux#23

Merged

likebreath mentioned this pull request Jun 13, 2025

build: Enable CONFIG_FW_CFG_SYSFS_CMDLINE as default configs cloud-hypervisor/linux#24

Merged

likebreath force-pushed the pr-fw_cfg branch from a46b6da to 49ef346 Compare June 13, 2025 23:27

AlexOrozco1256 force-pushed the pr-fw_cfg branch 3 times, most recently from 6069a50 to 393271b Compare June 16, 2025 18:03

likebreath mentioned this pull request Jul 28, 2025

Refactor 'create_acpi_tables' #7226

Merged

likebreath reviewed Jul 30, 2025

View reviewed changes

AlexOrozco1256 force-pushed the pr-fw_cfg branch from 665eb4e to 8fde3f6 Compare July 30, 2025 23:48

likebreath reviewed Aug 1, 2025

View reviewed changes

vmm/src/config.rs Outdated Show resolved Hide resolved

AlexOrozco1256 force-pushed the pr-fw_cfg branch 3 times, most recently from 232cbcc to 2156cfe Compare August 7, 2025 15:32

likebreath reviewed Aug 8, 2025

View reviewed changes

vmm/src/acpi.rs Outdated Show resolved Hide resolved

vmm/src/acpi.rs Outdated Show resolved Hide resolved

AlexOrozco1256 force-pushed the pr-fw_cfg branch from 2156cfe to 2bb6ad1 Compare August 8, 2025 23:48

AlexOrozco1256 added 8 commits August 10, 2025 20:32

devices: Add kernel cmdline, kernel, and initramfs to fw_cfg device

f7c37a3

The kernel and initramfs are passed to the fw_cfg device as file references. The cmdline is passed directly. Signed-off-by: Alex Orozco <[email protected]>

devices: Add e820/memory_map to fw_cfg device

24fc941

We build the memory map in the fw_cfg device based on the memory size. Signed-off-by: Alex Orozco <[email protected]>

devices: Implement DMA for fw_cfg device

ec18636

We pass a reference to the guest memory when we create the device in DeviceManager. This allows us to access the guest memory for DMA. Signed-off-by: Alex Orozco <[email protected]>

devices: Add fw_cfg cli options

a75a39f

This allows us to enable/disable the fw_cfg device via the cli We can also now upload files into the guest vm using fw_cfg_items via the cli Signed-off-by: Alex Orozco <[email protected]>

tests: Add fw_cfg device integration test

7bea7a3

This test verifies that we can see custom items added to the fw_cfg device from inside the guest Signed-off-by: Alex Orozco <[email protected]>

AlexOrozco1256 force-pushed the pr-fw_cfg branch from 2bb6ad1 to 7bea7a3 Compare August 10, 2025 20:33

likebreath approved these changes Aug 11, 2025

View reviewed changes

likebreath added this pull request to the merge queue Aug 11, 2025

Merged via the queue into cloud-hypervisor:main with commit 5d478c5 Aug 11, 2025
40 of 41 checks passed

likebreath mentioned this pull request Aug 11, 2025

vmm, devices: introduce ivshmem device #6703

Merged

AlexOrozco1256 deleted the pr-fw_cfg branch August 13, 2025 15:59

likebreath moved this to ✅ Done in Cloud Hypervisor Roadmap Aug 14, 2025

likebreath added this to Cloud Hypervisor Roadmap Aug 14, 2025

likebreath mentioned this pull request Aug 18, 2025

vm-migration: Add support for downtime limits #7033

Closed

devices: Add fw_cfg device #7117

devices: Add fw_cfg device #7117

Uh oh!

Conversation

AlexOrozco1256 commented Jun 4, 2025

Uh oh!

AlexOrozco1256 commented Jun 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rbradford commented Jun 10, 2025

Uh oh!

AlexOrozco1256 commented Jun 10, 2025

Uh oh!

rbradford left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rbradford left a comment

Choose a reason for hiding this comment

Uh oh!

AlexOrozco1256 commented Jun 10, 2025

Uh oh!

likebreath left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

likebreath Jun 10, 2025

Choose a reason for hiding this comment

Uh oh!

rbradford Jun 11, 2025

Choose a reason for hiding this comment

Uh oh!

AlexOrozco1256 commented Jun 12, 2025

Uh oh!

likebreath commented Jun 12, 2025

Uh oh!

likebreath commented Jun 13, 2025

Uh oh!

likebreath left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

likebreath Jul 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AlexOrozco1256 commented Aug 7, 2025

Uh oh!

likebreath left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

AlexOrozco1256 commented Aug 8, 2025

Uh oh!

AlexOrozco1256 commented Aug 10, 2025

Uh oh!

likebreath left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

AlexOrozco1256 commented Jun 4, 2025 •

edited

Loading