devices: gracefully close preserved FDs on hot device remove #7371
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
phip1611
force-pushed
the
productize-network_fd_livemig_prep-1
branch
from
014ccc5 to
99e72c1
Compare
phip1611
commented
Sep 22, 2025
vmm/src/config.rs
Outdated
| fds.append(&mut preserved_fds.clone()); | ||
| for fd in preserved_fds { | ||
| if fds.contains(fd) { | ||
| log::warn!("fd {fd} is already preserved, skipping"); |
Member
Author
There was a problem hiding this comment.
actually, this should be an error I think. Otherwise, we have non-unique resources which is undefined behavior anyway
8 tasks
rbradford
reviewed
Sep 22, 2025
Member
rbradford
left a comment
rbradford
left a comment
There was a problem hiding this comment.
We don't use the terminology of attached/detached - we use added/removed. Please update the commit message, comment, etc to be consistent with that (otherwise it might be confusing!.)
rbradford
reviewed
Sep 22, 2025
vmm/src/device_manager.rs
Outdated
| .unwrap() | ||
| .device_type(), | ||
| ); | ||
| // When the device is hot-detached, we close all file descriptors |
Member
There was a problem hiding this comment.
(We also use "hotplugged" as well as added.)
phip1611
force-pushed
the
productize-network_fd_livemig_prep-1
branch
3 times, most recently
from
f415134 to
82070e1
Compare
phip1611
changed the title
phip1611
force-pushed
the
productize-network_fd_livemig_prep-1
branch
from
82070e1 to
f9d6055
Compare
For a graceful resource management of externally provided FDs in Cloud Hypervisor, corresponding FDs need to be closed on a device removal. This is the case for virtio-net devices using external FDs, for example. With the fix introduced in this commit, we allow management software to properly clean up resources, e.g., libvirt can clean up tap devices. PS: CHV uses "added" and "removed", which has the same meaning as hot device attach/hotplug and hot device detach/unplug. Signed-off-by: Philipp Schuster <[email protected]> On-behalf-of: SAP [email protected]
phip1611
force-pushed
the
productize-network_fd_livemig_prep-1
branch
from
f9d6055 to
e261ecf
Compare
rbradford
approved these changes
Sep 25, 2025
Merged
via the queue into
cloud-hypervisor:main
with commit Sep 25, 2025
a6426e3
38 of 40 checks passed
2 tasks
phip1611
added a commit
to phip1611/cloud-hypervisor
that referenced
this pull request
Oct 23, 2025
Allowing the same file descriptor (FD) to be preserved more than once can lead to severe issues. Consider the case where management software adds two virtio-net devices at runtime, both backed by the same externally provided FD. When the first device is dropped, the FD for the second device becomes invalid [0]. To avoid this misconfiguration, we now prevent multiple preservation attempts of the same FD. [0] cloud-hypervisor#7371
phip1611
added a commit
to phip1611/cloud-hypervisor
that referenced
this pull request
Oct 23, 2025
Allowing the same file descriptor (FD) to be preserved more than once can lead to severe issues. Consider the case where management software adds two virtio-net devices at runtime, both backed by the same externally provided FD. When the first device is removed during runtime, the FD for the second device becomes invalid [0]. To avoid misconfigurations, we now prevent multiple preservation attempts of the same FD. [0] cloud-hypervisor#7371 Signed-off-by: Philipp Schuster <[email protected]> On-behalf-of: SAP [email protected]
phip1611
added a commit
to phip1611/cloud-hypervisor
that referenced
this pull request
Oct 23, 2025
Allowing the same file descriptor (FD) to be preserved more than once can lead to severe issues. Consider the case where management software adds two virtio-net devices at runtime, both backed by the same externally provided FD. When the first device is removed during runtime, the FD for the second device becomes invalid [0]. To avoid misconfigurations, we now prevent multiple preservation attempts of the same FD. [0] cloud-hypervisor#7371 Signed-off-by: Philipp Schuster <[email protected]> On-behalf-of: SAP [email protected]
phip1611
added a commit
to phip1611/cloud-hypervisor
that referenced
this pull request
Oct 23, 2025
Allowing the same file descriptor (FD) to be preserved more than once can lead to severe issues. Consider the case where management software adds two virtio-net devices at runtime, both backed by the same externally provided FD. When the first device is removed during runtime, the FD for the second device becomes invalid [0]. To avoid misconfigurations, we now prevent multiple preservation attempts of the same FD. [0] cloud-hypervisor#7371 Signed-off-by: Philipp Schuster <[email protected]> On-behalf-of: SAP [email protected]
phip1611
added a commit
to phip1611/cloud-hypervisor
that referenced
this pull request
Oct 23, 2025
Allowing the same file descriptor (FD) to be preserved more than once can lead to severe issues. Consider the case where management software adds two virtio-net devices at runtime, both backed by the same externally provided FD. When the first device is removed during runtime, the FD for the second device becomes invalid [0]. To avoid misconfigurations, we now prevent multiple preservation attempts of the same FD. This however has some caveats: In the current model, external FDs are added to the list of preserved FDs after the corresponding device has been created successfully. Now assume the following: - VM boots - VM config has no preserved FDs - We add a device to the VM that uses an external FD for its Tap dev - The device is successfully initialized - VM config is extended with the preserved FDs - VM reboots - VM config already has preserved FDs - Same as above - CRASH Therefore, we need to know for every preserved FD whether it is "cold", i.e., only has been part of a former VmConfig without associated device, or "hot", i.e., it is currently also actively in use. This allows state transitions from cold->hot in which case we won't throw an error for a reused FD. Otherwise, we throw errors. This requires that on shutdown, all preserved FDs are marked as cold. [0] cloud-hypervisor#7371 Signed-off-by: Philipp Schuster <[email protected]> On-behalf-of: SAP [email protected]
phip1611
added a commit
to phip1611/cloud-hypervisor
that referenced
this pull request
Oct 23, 2025
Allowing the same file descriptor (FD) to be preserved more than once can lead to severe issues. Consider the case where management software adds two virtio-net devices at runtime, both backed by the same externally provided FD. When the first device is removed during runtime, the FD for the second device becomes invalid [0]. To avoid misconfigurations, we now prevent multiple preservation attempts of the same FD. This however has some caveats: In the current model, external FDs are added to the list of preserved FDs after the corresponding device has been created successfully. Now assume the following: - VM boots - VM config has no preserved FDs - We add a device to the VM that uses an external FD for its Tap dev - The device is successfully initialized - VM config is extended with the preserved FDs - VM reboots - VM config already has preserved FDs - Same as above - CRASH Therefore, we need to know for every preserved FD whether it is "cold", i.e., only has been part of a former VmConfig without associated device, or "hot", i.e., it is currently also actively in use. This allows state transitions from cold->hot in which case we won't throw an error for a reused FD. Otherwise, we throw errors. This requires that on shutdown, all preserved FDs are marked as cold. [0] cloud-hypervisor#7371 Signed-off-by: Philipp Schuster <[email protected]> On-behalf-of: SAP [email protected]
phip1611
added a commit
to phip1611/cloud-hypervisor
that referenced
this pull request
Oct 23, 2025
Allowing the same file descriptor (FD) to be preserved more than once can lead to severe issues. Consider the case where management software adds two virtio-net devices at runtime, both backed by the same externally provided FD. When the first device is removed during runtime, the FD for the second device becomes invalid [0]. To avoid misconfigurations, we now prevent multiple preservation attempts of the same FD. This however has some caveats: In the current model, external FDs are added to the list of preserved FDs after the corresponding device has been created successfully. Now assume the following: - VM boots - VM config has no preserved FDs - We add a device to the VM that uses an external FD for its Tap dev - The device is successfully initialized - VM config is extended with the preserved FDs - VM reboots - VM config already has preserved FDs - Same as above - CRASH Therefore, we need to know for every preserved FD whether it is "cold", i.e., only has been part of a former VmConfig without associated device, or "hot", i.e., it is currently also actively in use. This allows state transitions from cold->hot in which case we won't throw an error for a reused FD. Otherwise, we throw errors. This requires that on shutdown, all preserved FDs are marked as cold. [0] cloud-hypervisor#7371 Signed-off-by: Philipp Schuster <[email protected]> On-behalf-of: SAP [email protected]
phip1611
added a commit
to phip1611/cloud-hypervisor
that referenced
this pull request
Oct 27, 2025
Allowing the same file descriptor (FD) to be preserved more than once can lead to severe issues. Consider the case where management software adds two virtio-net devices at runtime, both backed by the same externally provided FD. When the first device is removed during runtime, the FD for the second device becomes invalid [0]. To avoid misconfigurations, we now prevent multiple preservation attempts of the same FD. This however has some caveats: In the current model, external FDs are added to the list of preserved FDs after the corresponding device has been created successfully. Now assume the following: - VM boots - VM config has no preserved FDs - We add a device to the VM that uses an external FD for its Tap dev - The device is successfully initialized - VM config is extended with the preserved FDs - VM reboots - VM config already has preserved FDs - Same as above - CRASH Therefore, we need to know for every preserved FD whether it is "cold", i.e., only has been part of a former VmConfig without associated device, or "hot", i.e., it is currently also actively in use. This allows state transitions from cold->hot in which case we won't throw an error for a reused FD. Otherwise, we throw errors. This requires that on shutdown, all preserved FDs are marked as cold. [0] cloud-hypervisor#7371 Signed-off-by: Philipp Schuster <[email protected]> On-behalf-of: SAP [email protected]
phip1611
added a commit
to phip1611/cloud-hypervisor
that referenced
this pull request
Oct 27, 2025
Allowing the same file descriptor (FD) to be preserved more than once can lead to severe issues. Consider the case where management software adds two virtio-net devices at runtime, both backed by the same externally provided FD. When the first device is removed during runtime, the FD for the second device becomes invalid [0]. To avoid misconfigurations, we now prevent multiple preservation attempts of the same FD. This however has some caveats: In the current model, external FDs are added to the list of preserved FDs after the corresponding device has been created successfully. Now assume the following: - VM boots - VM config has no preserved FDs - We add a device to the VM that uses an external FD for its Tap dev - The device is successfully initialized - VM config is extended with the preserved FDs - VM reboots - VM config already has preserved FDs - Same as above - CRASH Therefore, we need to know for every preserved FD whether it is "cold", i.e., only has been part of a former VmConfig without associated device, or "hot", i.e., it is currently also actively in use. This allows state transitions from cold->hot in which case we won't throw an error for a reused FD. Otherwise, we throw errors. This requires that on shutdown, all preserved FDs are marked as cold. [0] cloud-hypervisor#7371 Signed-off-by: Philipp Schuster <[email protected]> On-behalf-of: SAP [email protected]
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
For a graceful resource management of externally provided FDs in Cloud
Hypervisor, corresponding FDs need to be closed on a device removal.
This is the case for virtio-net devices using external FDs, for example.
With the fix introduced in this commit, we allow management software to
properly clean up resources, e.g., libvirt can clean up tap devices.
PS: CHV uses "added" and "removed", which has the same meaning as
hot device attach/hotplug and hot device detach/unplug.
Part of #7291.