VFIO device BAR reprogramming doesn't move VFIO IOMMU mappings #7027
Description
Describe the bug
When hotplugging a VFIO device, the guest may reprogram the bar. CLH, however, doesn't unmap the old VFIO mappings and map the new ones. Specifically, move_bar in pci/src/vfio.rs only moves KVM TDP mappings.
To Reproduce
Steps to reproduce the behaviour:
Start a standard CLH
/root/cloud-hypervisor-upstream/target/release/cloud-hypervisor \
-v \
--kernel /root/guest-kernel-6.1.103/arch/x86/boot/bzImage \
--disk path=focal-server-cloudimg-amd64.raw \
--cmdline "nokaslr root=/dev/vda1 rw" \
--cpus boot=1 \
--memory size=3G \
--seccomp false \
--api-socket=/tmp/ch-socket
Attach a GPU named 01:00:0 to the VM
sudo ./target/release/ch-remote --api-socket=/tmp/ch-socket add-device path=/sys/bus/pci/devices/0000:01:00.0/
Shutdown VMM
sudo ./target/release/ch-remote --api-socket=/tmp/ch-socket shutdown-vmm
Version
Output of cloud-hypervisor --version:
cloud-hypervisor v45.0
Did you build from source, if so build command line (e.g. features):
Yes. RUSTUP_TOOLCHAIN=nightly-2025-02-07 cargo build --release
VM configuration
What command line did you run (or JSON config data):
See above.
Guest OS version details:
Linux 6.1.103 (Upstream)
Host OS version details:
Linux 5.10.112 (Upstream)
Logs
Output of cloud-hypervisor -v from either standard error or via --log-file:
cloud-hypervisor: 458.831000µs: <vmm> INFO:vmm/src/api/mod.rs:839 -- API request event: VmCreate VmConfig { cpus: CpusConfig { boot_vcpus: 1, max_vcpus: 1, topology: None, kvm_hyperv: false, max_phys_bits: 46, affinity: None, features: CpuFeatures { amx: false } }, memor}
cloud-hypervisor: 2.367927ms: <vmm> INFO:vmm/src/api/mod.rs:730 -- API request event: VmBoot
cloud-hypervisor: 2.433829ms: <vmm> INFO:vmm/src/lib.rs:1498 -- Booting VM
cloud-hypervisor: 2.997141ms: <vmm> INFO:vmm/src/memory_manager.rs:590 -- create ram region for zone mem0, region_start: 0x0, region_size: 0xc0000000
cloud-hypervisor: 3.072413ms: <vmm> INFO:vmm/src/memory_manager.rs:1517 -- Anonymous mapping at 0x7fae78000000 (size = 0xc0000000)
cloud-hypervisor: 3.135476ms: <vmm> INFO:vmm/src/vm.rs:514 -- Booting VM from config: Mutex { data: VmConfig { cpus: CpusConfig { boot_vcpus: 1, max_vcpus: 1, topology: None, kvm_hyperv: false, max_phys_bits: 46, affinity: None, features: CpuFeatures { amx: false } }, me}
cloud-hypervisor: 3.165946ms: <payload_loader> INFO:vmm/src/vm.rs:1096 -- Loading kernel
cloud-hypervisor: 3.454428ms: <vmm> INFO:arch/src/x86_64/mod.rs:619 -- Generating guest CPUID for with physical address size: 39
cloud-hypervisor: 3.527559ms: <vmm> INFO:vmm/src/cpu.rs:890 -- Request to create new vCPUs: desired = 1, max = 1, allocated = 0, present = 0
cloud-hypervisor: 3.542981ms: <vmm> INFO:vmm/src/cpu.rs:797 -- Creating vCPU: cpu_id = 0
cloud-hypervisor: 3.627582ms: <vmm> INFO:vmm/src/pci_segment.rs:106 -- Adding PCI segment: id=0, PCI MMIO config address: 0xe8000000, mem32 area [0xc0000000-0xe7ffffff, mem64 area [0x100000000-0x7effffffff
cloud-hypervisor: 3.678003ms: <vmm> INFO:vmm/src/device_manager.rs:2377 -- Creating virtio-block device: DiskConfig { path: Some("focal-server-cloudimg-amd64.raw"), readonly: false, direct: false, iommu: false, num_queues: 1, queue_size: 128, vhost_user: false, vhost_soc}
cloud-hypervisor: 3.805618ms: <vmm> INFO:vmm/src/device_manager.rs:2463 -- Using asynchronous RAW disk file (io_uring)
cloud-hypervisor: 3.818501ms: <vmm> INFO:virtio-devices/src/block.rs:642 -- Disk topology: DiskTopology { logical_block_size: 512, physical_block_size: 512, minimum_io_size: 512, optimal_io_size: 0 }
cloud-hypervisor: 3.847749ms: <vmm> INFO:vmm/src/device_manager.rs:2610 -- Creating virtio-net device: NetConfig { tap: None, ip: 192.168.249.1, mask: 255.255.255.0, mac: MacAddr { bytes: [194, 103, 79, 83, 41, 204] }, host_mac: None, mtu: None, iommu: false, num_queues:}
cloud-hypervisor: 3.948355ms: <vmm> INFO:vmm/src/device_manager.rs:2776 -- Creating virtio-rng device: RngConfig { src: "/dev/urandom", iommu: false }
cloud-hypervisor: 4.347205ms: <vmm> INFO:vmm/src/acpi.rs:833 -- Generated ACPI tables: took 306µs size = 5165
cloud-hypervisor: 4.359343ms: <vmm> INFO:vmm/src/vm.rs:2153 -- Created ACPI tables: rsdp_addr = 0xa0000
cloud-hypervisor: 9.465954ms: <payload_loader> INFO:vmm/src/vm.rs:1135 -- bzImage kernel loaded: entry_addr = 0x100000
cloud-hypervisor: 9.523673ms: <vmm> INFO:vmm/src/cpu.rs:396 -- Configuring vCPU: cpu_id = 0
cloud-hypervisor: 9.550491ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000000 index = 0x00000000 eax = 0x0000001b ebx = 0x756e6547 ecx = 0x6c65746e edx = 0x49656e69 flags = 0x00000000
cloud-hypervisor: 9.597589ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000001 index = 0x00000000 eax = 0x000806c1 ebx = 0x00100800 ecx = 0xf7fab223 edx = 0x1f8bfbff flags = 0x00000000
cloud-hypervisor: 9.638778ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000002 index = 0x00000000 eax = 0x00feff01 ebx = 0x000000f0 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000000
cloud-hypervisor: 9.689954ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000003 index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000000
cloud-hypervisor: 9.744801ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000004 index = 0x00000000 eax = 0x1c004121 ebx = 0x02c0003f ecx = 0x0000003f edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 9.792651ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000004 index = 0x00000001 eax = 0x1c004122 ebx = 0x01c0003f ecx = 0x0000003f edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 9.839254ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000004 index = 0x00000002 eax = 0x1c004143 ebx = 0x04c0003f ecx = 0x000003ff edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 9.889827ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000004 index = 0x00000003 eax = 0x1c03c163 ebx = 0x02c0003f ecx = 0x00003fff edx = 0x00000004 flags = 0x00000001
cloud-hypervisor: 9.933411ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000004 index = 0x00000004 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 9.991415ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000005 index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000000
cloud-hypervisor: 10.047232ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000006 index = 0x00000000 eax = 0x00000004 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000000
cloud-hypervisor: 10.102553ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000007 index = 0x00000000 eax = 0x00000001 ebx = 0xf1bf07ab ecx = 0x18405f4e edx = 0xac000510 flags = 0x00000001
cloud-hypervisor: 10.141974ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000007 index = 0x00000001 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 10.196494ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000008 index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000000
cloud-hypervisor: 10.247045ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000009 index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000000
cloud-hypervisor: 10.304206ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x0000000a index = 0x00000000 eax = 0x08300802 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00008603 flags = 0x00000000
cloud-hypervisor: 10.355375ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x0000000b index = 0x00000000 eax = 0x00000000 ebx = 0x00000001 ecx = 0x00000100 edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 10.410684ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x0000000b index = 0x00000001 eax = 0x00000000 ebx = 0x00000001 ecx = 0x00000200 edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 10.466704ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x0000000b index = 0x00000002 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000002 edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 10.523021ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x0000000c index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000000
cloud-hypervisor: 10.578065ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x0000000d index = 0x00000000 eax = 0x000002e7 ebx = 0x00000a88 ecx = 0x00000a88 edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 10.629972ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x0000000d index = 0x00000001 eax = 0x0000000f ebx = 0x00000988 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 10.684690ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x0000000d index = 0x00000002 eax = 0x00000100 ebx = 0x00000240 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 10.738212ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x0000000d index = 0x00000005 eax = 0x00000040 ebx = 0x00000440 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 10.793855ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x0000000d index = 0x00000006 eax = 0x00000200 ebx = 0x00000480 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 10.845749ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x0000000d index = 0x00000007 eax = 0x00000400 ebx = 0x00000680 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 10.902407ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x0000000d index = 0x00000009 eax = 0x00000008 ebx = 0x00000a80 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 10.956364ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x0000000e index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000000
cloud-hypervisor: 11.013207ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x0000000f index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 11.067195ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000010 index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 11.122387ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000011 index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000000
cloud-hypervisor: 11.178325ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000012 index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 11.234130ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000013 index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000000
cloud-hypervisor: 11.292595ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000014 index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 11.347714ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000015 index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000000
cloud-hypervisor: 11.402851ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000016 index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000000
cloud-hypervisor: 11.459605ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000017 index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 11.514027ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000018 index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 11.568245ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x00000019 index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000000
cloud-hypervisor: 11.623475ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x0000001a index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000000
cloud-hypervisor: 11.675914ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x0000001b index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000000
cloud-hypervisor: 11.729330ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x80000000 index = 0x00000000 eax = 0x80000008 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000000
cloud-hypervisor: 11.774098ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x80000001 index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000121 edx = 0x2c100800 flags = 0x00000000
cloud-hypervisor: 11.817113ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x80000005 index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000000
cloud-hypervisor: 11.867269ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x80000006 index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x01007040 edx = 0x00000000 flags = 0x00000000
cloud-hypervisor: 11.913588ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x80000007 index = 0x00000000 eax = 0x00000000 ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000100 flags = 0x00000000
cloud-hypervisor: 11.965094ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x80000008 index = 0x00000000 eax = 0x00003027 ebx = 0x0100d000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000000
cloud-hypervisor: 12.010432ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x40000000 index = 0x00000000 eax = 0x40000010 ebx = 0x4b4d564b ecx = 0x564b4d56 edx = 0x0000004d flags = 0x00000000
cloud-hypervisor: 12.045104ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x40000001 index = 0x00000000 eax = 0x01007efb ebx = 0x00000000 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000000
cloud-hypervisor: 12.092754ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x80000002 index = 0x00000000 eax = 0x68743131 ebx = 0x6e654720 ecx = 0x746e4920 edx = 0x52286c65 flags = 0x00000000
cloud-hypervisor: 12.124274ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x80000003 index = 0x00000000 eax = 0x6f432029 ebx = 0x54286572 ecx = 0x6920294d edx = 0x31312d37 flags = 0x00000000
cloud-hypervisor: 12.154232ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x80000004 index = 0x00000000 eax = 0x37473536 ebx = 0x32204020 ecx = 0x4730382e edx = 0x00007a48 flags = 0x00000000
cloud-hypervisor: 12.189145ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x0000001f index = 0x00000000 eax = 0x00000000 ebx = 0x00000001 ecx = 0x00000100 edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 12.244414ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x0000001f index = 0x00000001 eax = 0x00000000 ebx = 0x00000001 ecx = 0x00000200 edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 12.300920ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x0000001f index = 0x00000002 eax = 0x00000000 ebx = 0x00000001 ecx = 0x00000500 edx = 0x00000000 flags = 0x00000001
cloud-hypervisor: 12.354493ms: <vmm> INFO:arch/src/x86_64/mod.rs:891 -- function = 0x40000010 index = 0x00000000 eax = 0x002ac600 ebx = 0x000f4240 ecx = 0x00000000 edx = 0x00000000 flags = 0x00000000
cloud-hypervisor: 12.422679ms: <vmm> INFO:vmm/src/vm.rs:1248 -- Configuring system
cloud-hypervisor: 12.440387ms: <vmm> INFO:vmm/src/memory_manager.rs:1763 -- Creating userspace mapping: 0 -> 7fae78000000 c0000000, slot 0
cloud-hypervisor: 12.991077ms: <vmm> INFO:vmm/src/memory_manager.rs:1811 -- Created userspace mapping: 0 -> 7fae78000000 c0000000
cloud-hypervisor: 13.005898ms: <vmm> INFO:vmm/src/cpu.rs:1217 -- Starting vCPUs: desired = 1, allocated = 1, present = 0, paused = false
cloud-hypervisor: 13.021697ms: <vmm> INFO:vmm/src/cpu.rs:987 -- Starting vCPU: cpu_id = 0
cloud-hypervisor: 103.384223ms: <vcpu0> INFO:vmm/src/vm.rs:436 -- Guest PIO write to unregistered address 0x43
cloud-hypervisor: 103.407536ms: <vcpu0> INFO:vmm/src/vm.rs:436 -- Guest PIO write to unregistered address 0x42
cloud-hypervisor: 103.419258ms: <vcpu0> INFO:vmm/src/vm.rs:436 -- Guest PIO write to unregistered address 0x42
cloud-hypervisor: 103.432698ms: <vcpu0> INFO:vmm/src/vm.rs:427 -- Guest PIO read to unregistered address 0x42
cloud-hypervisor: 103.444280ms: <vcpu0> INFO:vmm/src/vm.rs:427 -- Guest PIO read to unregistered address 0x42
cloud-hypervisor: 103.454674ms: <vcpu0> INFO:vmm/src/vm.rs:427 -- Guest PIO read to unregistered address 0x42
cloud-hypervisor: 103.465820ms: <vcpu0> INFO:vmm/src/vm.rs:427 -- Guest PIO read to unregistered address 0x42
cloud-hypervisor: 195.308491ms: <vcpu0> INFO:virtio-devices/src/transport/pci_device.rs:1218 -- _virtio-pci-__console: Needs activation; writing to activate event fd
cloud-hypervisor: 195.329507ms: <vcpu0> INFO:virtio-devices/src/transport/pci_device.rs:1223 -- _virtio-pci-__console: Needs activation; returning barrier
cloud-hypervisor: 195.340049ms: <vcpu0> INFO:vmm/src/vm.rs:415 -- Waiting for barrier
cloud-hypervisor: 195.438647ms: <vmm> INFO:vmm/src/lib.rs:1406 -- Trying to activate pending virtio devices: count = 1
cloud-hypervisor: 195.521437ms: <vmm> INFO:virtio-devices/src/transport/pci_device.rs:302 -- _virtio-pci-__console: Waiting for barrier
cloud-hypervisor: 195.547884ms: <vmm> INFO:virtio-devices/src/transport/pci_device.rs:304 -- _virtio-pci-__console: Barrier released
cloud-hypervisor: 195.551173ms: <vcpu0> INFO:vmm/src/vm.rs:417 -- Barrier released
cloud-hypervisor: 197.236291ms: <vcpu0> INFO:virtio-devices/src/transport/pci_device.rs:1218 -- _virtio-pci-__rng: Needs activation; writing to activate event fd
cloud-hypervisor: 197.246444ms: <vcpu0> INFO:virtio-devices/src/transport/pci_device.rs:1223 -- _virtio-pci-__rng: Needs activation; returning barrier
cloud-hypervisor: 197.253047ms: <vcpu0> INFO:vmm/src/vm.rs:415 -- Waiting for barrier
cloud-hypervisor: 197.362577ms: <vmm> INFO:vmm/src/lib.rs:1406 -- Trying to activate pending virtio devices: count = 1
cloud-hypervisor: 197.407430ms: <vmm> INFO:virtio-devices/src/transport/pci_device.rs:302 -- _virtio-pci-__rng: Waiting for barrier
cloud-hypervisor: 197.419495ms: <vmm> INFO:virtio-devices/src/transport/pci_device.rs:304 -- _virtio-pci-__rng: Barrier released
cloud-hypervisor: 197.422152ms: <vcpu0> INFO:vmm/src/vm.rs:417 -- Barrier released
cloud-hypervisor: 199.982966ms: <vcpu0> INFO:virtio-devices/src/transport/pci_device.rs:1218 -- _virtio-pci-_disk0: Needs activation; writing to activate event fd
cloud-hypervisor: 199.993986ms: <vcpu0> INFO:virtio-devices/src/transport/pci_device.rs:1223 -- _virtio-pci-_disk0: Needs activation; returning barrier
cloud-hypervisor: 200.002187ms: <vcpu0> INFO:vmm/src/vm.rs:415 -- Waiting for barrier
cloud-hypervisor: 200.101300ms: <vmm> INFO:vmm/src/lib.rs:1406 -- Trying to activate pending virtio devices: count = 1
cloud-hypervisor: 200.113181ms: <vmm> INFO:virtio-devices/src/block.rs:728 -- Changing cache mode to writeback
cloud-hypervisor: 218.075177ms: <vmm> INFO:virtio-devices/src/transport/pci_device.rs:302 -- _virtio-pci-_disk0: Waiting for barrier
cloud-hypervisor: 218.097526ms: <vmm> INFO:virtio-devices/src/transport/pci_device.rs:304 -- _virtio-pci-_disk0: Barrier released
cloud-hypervisor: 218.107366ms: <vcpu0> INFO:vmm/src/vm.rs:417 -- Barrier released
cloud-hypervisor: 220.587570ms: <vcpu0> INFO:virtio-devices/src/transport/pci_device.rs:1218 -- _virtio-pci-_net1: Needs activation; writing to activate event fd
cloud-hypervisor: 220.600980ms: <vcpu0> INFO:virtio-devices/src/transport/pci_device.rs:1223 -- _virtio-pci-_net1: Needs activation; returning barrier
cloud-hypervisor: 220.608763ms: <vcpu0> INFO:vmm/src/vm.rs:415 -- Waiting for barrier
cloud-hypervisor: 220.703051ms: <vmm> INFO:vmm/src/lib.rs:1406 -- Trying to activate pending virtio devices: count = 1
cloud-hypervisor: 220.781731ms: <vmm> INFO:virtio-devices/src/transport/pci_device.rs:302 -- _virtio-pci-_net1: Waiting for barrier
cloud-hypervisor: 220.798040ms: <vmm> INFO:virtio-devices/src/transport/pci_device.rs:304 -- _virtio-pci-_net1: Barrier released
cloud-hypervisor: 220.801057ms: <vcpu0> INFO:vmm/src/vm.rs:417 -- Barrier released
cloud-hypervisor: 220.889221ms: <_net1_ctrl> INFO:net_util/src/ctrl_queue.rs:100 -- Number of MQ pairs requested: 1
Ubuntu 20.04.6 LTS cloud hvc0
cloud login: cloud-hypervisor: 22.510027s: <vmm> INFO:vmm/src/api/mod.rs:438 -- API request event: VmAddDevice DeviceConfig { path: "/sys/bus/pci/devices/0000:01:00.0/", iommu: false, id: None, pci_segment: 0, x_nv_gpudirect_clique: None }
cloud-hypervisor: 23.754417s: <vcpu0> INFO:pci/src/configuration.rs:1008 -- Detected BAR reprogramming: (BAR 6) 0x7e->0x1
cloud-hypervisor: 23.755954s: <vcpu0> INFO:pci/src/configuration.rs:1008 -- Detected BAR reprogramming: (BAR 8) 0x7e->0x1
cloud-hypervisor: 23.756053s: <vcpu0> INFO:pci/src/configuration.rs:982 -- Detected BAR reprogramming: (BAR 4) 0xe6000000->0xc0000000
cloud-hypervisor: 23.756161s: <vcpu0> INFO:pci/src/configuration.rs:982 -- Detected BAR reprogramming: (BAR 9) 0xff81->0x1001
cloud-hypervisor: 33.810590s: <vmm> INFO:vmm/src/api/mod.rs:1404 -- API request event: VmmShutdown
cloud-hypervisor: 33.810611s: <vmm> INFO:virtio-devices/src/device.rs:335 -- Resuming virtio-console
cloud-hypervisor: 33.810622s: <vmm> INFO:virtio-devices/src/device.rs:335 -- Resuming virtio-rng
cloud-hypervisor: 33.810647s: <vmm> INFO:virtio-devices/src/device.rs:335 -- Resuming virtio-block
cloud-hypervisor: 33.810659s: <vmm> INFO:virtio-devices/src/device.rs:335 -- Resuming virtio-net
cloud-hypervisor: 33.811756s: <vmm> INFO:virtio-devices/src/device.rs:335 -- Resuming virtio-console
cloud-hypervisor: 33.811763s: <vmm> INFO:virtio-devices/src/device.rs:335 -- Resuming virtio-rng
cloud-hypervisor: 33.811769s: <vmm> INFO:virtio-devices/src/device.rs:335 -- Resuming virtio-block
cloud-hypervisor: 33.811777s: <vmm> INFO:virtio-devices/src/device.rs:335 -- Resuming virtio-net
cloud-hypervisor: 33.811801s: <__console> INFO:virtio-devices/src/epoll_helper.rs:217 -- KILL_EVENT received, stopping epoll loop
cloud-hypervisor: 33.811885s: <_disk0_q0> INFO:virtio-devices/src/epoll_helper.rs:217 -- KILL_EVENT received, stopping epoll loop
cloud-hypervisor: 33.811994s: <__rng> INFO:virtio-devices/src/epoll_helper.rs:217 -- KILL_EVENT received, stopping epoll loop
cloud-hypervisor: 34.166378s: <_net1_qp0> INFO:virtio-devices/src/epoll_helper.rs:217 -- KILL_EVENT received, stopping epoll loop
cloud-hypervisor: 34.166402s: <_net1_ctrl> INFO:virtio-devices/src/epoll_helper.rs:217 -- KILL_EVENT received, stopping epoll loop
Host Linux kernel output (with customized logs):
[ 7422.047174] vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258
[ 7422.047205] vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0x900
[ 7422.049786] vfio_dma_do_map: iova 0x0 size 0xc0000000
[ 7423.095375] vfio_dma_do_map: iova 0xe6000000 size 0x1000000
[ 7423.098837] vfio_dma_do_map: iova 0x7ee0000000 size 0x10000000
[ 7423.114275] vfio_dma_do_map: iova 0x7efc000000 size 0x2000000
[ 7434.560540] vfio_dma_do_unmap: iova 0xc0000000 size 0x1000000
[ 7434.567574] vfio_dma_do_unmap: iova 0x100000000 size 0x10000000
[ 7434.567662] vfio_dma_do_unmap: iova 0x110000000 size 0x2000000
Guest lspci:
00:05.0 VGA compatible controller: NVIDIA Corporation TU106 [GeForce RTX 2060] (rev a1) (prog-if 00 [VGA controller])
Subsystem: Intel Corporation Device 2090
Physical Slot: 5
Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
Latency: 0, Cache Line Size: 64 bytes
Interrupt: pin A routed to IRQ 36
Region 0: Memory at c0000000 (32-bit, non-prefetchable) [size=16M]
Region 1: Memory at 100000000 (64-bit, prefetchable) [size=256M]
Region 3: Memory at 110000000 (64-bit, prefetchable) [size=32M]
Guest kernel dmesg:
[ 9.539546] ACPI: \_SB_.PC00.S005: Device check in hotplug_event()
[ 9.539788] pci 0000:00:05.0: [10de:1f15] type 00 class 0x030000
[ 9.541456] pci 0000:00:05.0: reg 0x10: [mem 0xe6000000-0xe6ffffff]
[ 9.541543] pci 0000:00:05.0: reg 0x14: [mem 0x7ee0000000-0x7eefffffff 64bit pref]
[ 9.541628] pci 0000:00:05.0: reg 0x1c: [mem 0x7efc000000-0x7efdffffff 64bit pref]
[ 9.541692] pci 0000:00:05.0: reg 0x24: [io 0xff80-0xffff]
[ 9.541757] pci 0000:00:05.0: reg 0x30: [mem 0xe7f00000-0xe7f7ffff pref]
[ 9.541940] pci 0000:00:05.0: Video device with shadowed ROM at [mem 0x000c0000-0x000dffff]
[ 9.542466] pci 0000:00:05.0: PME# supported from D0 D3hot D3cold
[ 9.542491] pci 0000:00:05.0: PME# disabled
[ 9.542758] pci 0000:00:05.0: 8.000 Gb/s available PCIe bandwidth, limited by 2.5 GT/s PCIe x4 link at 0000:00:05.0 (capable of 126.016 Gb/s with 8.0 GT/s PCIe x16 link)
[ 9.543296] pci 0000:00:05.0: vgaarb: pci_notify
[ 9.543312] pci 0000:00:05.0: vgaarb: setting as boot VGA device
[ 9.543314] pci 0000:00:05.0: vgaarb: bridge control possible
[ 9.543315] pci 0000:00:05.0: vgaarb: VGA device added: decodes=io+mem,owns=io+mem,locks=none
[ 9.543349] pci 0000:00:05.0: BAR 1: assigned [mem 0x100000000-0x10fffffff 64bit pref]
[ 9.545000] pci 0000:00:05.0: BAR 3: assigned [mem 0x110000000-0x111ffffff 64bit pref]
[ 9.545124] pci 0000:00:05.0: BAR 0: assigned [mem 0xc0000000-0xc0ffffff]
[ 9.545222] pci 0000:00:05.0: BAR 5: assigned [io 0x1000-0x107f]
When hotplugging the device, the guest reprograms a bar 0xe6000000->0xc0000000. But CLH doesn't unmap 0xe6000000 and map 0xc0000000. Not sure whether it's a bug or a feature. Any information is appreciated.