This issue is created to track the development and discussion of live migration features and related improvements.

Feel free to comment below, claim features you are interested in, and suggest updates to the checklist as needed.

• Devices
  • Generic mechanism for handling external FDs (@posk-io)
  • Network devices with external FDs (@cyberus-technology)
    • [WIP] migration: Persist Nested Virtualisation State #7074
  • VFIO devices
• Transport / Wire
  • TLS encryption of the migration (@cyberus-technology)
  • postcopy migration (@cyberus-technology)
  • Bandwidth limit
  • Downtime limit (taken over by @cyberus-technology)
    • originally started by Jinrong
    • vm-migration: Add support for downtime limits #7033
  • Multiple parallel TCP connections (@cloud-technology)
    • Working prototype: Multiple TCP Connections for Migration cyberus-technology/cloud-hypervisor#39
  • Smarter downtime calculation: take VM stop and final VM state transfer (devices, vcpus) into account
  • Gracefully handle lost or spurious network
  • optimize zero-page transmission (shortcut)
  • RDMA transport (Live-migration: RDMA support #7395):
    • Firs task: evaluate value-add once TCP/IP migration is production-ready
    • Was RDMA requested for short downtimes? If yes, we probably don't need it once we have postcopy.
• Network
  • send "hello" network packages after hand-over to populate ARP caches
• Guest Lifecycle
  • Allow guest-induced shutdown during migration
  • Allow guest-induced reboot during migration
• Other / Implementation Tweaks
  • AMX
    • hypervisor: AMX state snapshot and restore support #7534
  • vCPU throttling / auto-converge
    • Production-ready solution exists by @cyberus-technology
    • gardenlinux: PoC: vCPU throttling/auto-converge cyberus-technology/cloud-hypervisor#13
    • We will not upstream this. Postcopy is clearly superior to vCPU throttling
  • "Asynchronization" / Unblock VMM lifecycle thread (@cyberus-technology)
    • Original attempt: Add support for live migration asynchronization #7039
    • Working prototype: vmm: live migration asynchronization (dedicated thread) cyberus-technology/cloud-hypervisor#24
  • Migration cancellation
  • Migration information query (@cyberus-technology)
  • Selectable CPU profiles (@cyberus-technology)
    • Working prototype: CPU Profiles (Part 1: CPUID) cyberus-technology/cloud-hypervisor#25
  • Support nested virtualization (Philipp)
    • [WIP] migration: Persist Nested Virtualisation State #7074
  • Implement abort-migration API call
  • "Migration in progress": Block VMResize, AddNet, etc with "VM in migration" error
    • Working prototype: vmm: live migration asynchronization (dedicated thread) cyberus-technology/cloud-hypervisor#24
• Documentation
  • Add overview of supported functionality, caveats, and known limitations
• Production-readyness
  • Add feature / migration protocol negiation
  • Announce in README that live-migration can't be breaking any longer between releases

Collaboration process:

• Comment below to discuss implementation details, share ideas, or suggest changes.
• Assign yourself to features you want to work on.
• If you have suggestions for updating the checklist or description, please leave a comment with your proposed changes, or send the revised text to me directly.
• We will update the issue accordingly as progress is made.