gardenlinux: PoC: vCPU throttling/auto-converge #13

phip1611 · 2025-08-18T14:50:58Z

This adds the PoC for vCPU-throttling/auto-converge for pre-copy migration.

vm-migration: add vCPU throttling (auto-converge) for pre-copy

About

auto-converge (vCPU throttling) is a crucial technique to migrate
VMs with a high dirty rate (high working set with intense usage).
It is an alternative to postcopy migration, which is not yet
implemented in Cloud Hypervisor.

Implementation

vCPU throttling was implemented with a different thread and a
manager for that thread. It is possible to abort vCPU throttling
in case one aborts a live-migration, for example.

The rather complex thread state management is covered in unit
tests ensuring liveliness throughout all possible scenarios.

The throttling itself is implemented by utilizing the CpuManager's
pause() and resume() functions.

Effective Behaviour

auto-converging starts always after four memory delta transfer
iterations (not configurable yet)
every two iterations, it is increased again
step size is 10%
maximum is 99%
time window for throttling is 100ms. So with 99% throttling,
the vCPU is paused for 99ms and runs for 1ms, and again.

Signed-off-by: Philipp Schuster [email protected]
On-behalf-of: SAP [email protected]

Hints for Reviewers

Review this commit by commit
We did this to enable SAP to migrate their memory intensive workloads somehow in the Pilot.

Steps to Undraft

Test that the vCPU throttling has the desired effect onto guest workloads
libvirt-tests succeeds (ran locally)
merge and rebase onto cpu-manager: accelerate pause() #16
merge and rebase onto pr #7033 squashed 2025-08-18: downtime limits #17
merge and rebase onto vmm: fix kicking vCPU out of KVM_RUN vCPU from signal handler #20

vmm/src/lib.rs

docs/live_migration.md

vmm/src/api/openapi/cloud-hypervisor.yaml

vmm/src/vcpu_throttling.rs

tpressure · 2025-08-19T11:57:22Z

I just stress tested this and ran into an issue after 174 successful migrations. I'm now in a state where the receiver side is on 100% cpu whereas the sender side is idling, and the migration does not continue

vmm/src/lib.rs

vmm/src/vm.rs

phip1611 · 2025-08-22T12:22:45Z

vmm/src/vm.rs

            VmState::Created
        };

+        let vcpu_throttle_thread_handle = ThrottleThreadHandle::new_from_cpu_manager(&cpu_manager);


Suggested change

let vcpu_throttle_thread_handle = ThrottleThreadHandle::new_from_cpu_manager(&cpu_manager);

let vcpu_throttler = ThrottleThreadHandle::new_from_cpu_manager(&cpu_manager);

olivereanderson

Thank you for the changes and especially for the awesome documentation that made it relatively easy to follow what the code in this commit does!

I have added a few comments below which you can either decide to consider already now or wait until you start preparing this feature for production. See also the list on this issue.

Moreover once we get to making this production ready then if we decide to instead spawn the throttling thread per (full) migration attempt (potentially with seccomp setup) instead, then as we already mentioned in direct messages it should be possible to simplify the code a bit by only having a single loop in the throttling thread.

olivereanderson · 2025-08-22T15:01:09Z

vmm/src/vcpu_throttling.rs

+    fn test_vcpu_throttling_thread_lifecycle() {
+        for _ in 0..5 {
+            // State transitions: Waiting -> Exit
+            {
+                let mut handler = ThrottleThreadHandle::new(Box::new(|| {}), Box::new(|| {}));
+
+                // The test is successful if it does not get stuck.
+                handler.shutdown();
+            }
+
+            // Dummy CpuManager
+            let cpus_throttled = Arc::new(AtomicBool::new(false));
+            let callback_pause_vcpus = {
+                let cpus_running = cpus_throttled.clone();
+                Box::new(move || {
+                    let old = cpus_running.swap(true, Ordering::SeqCst);
+                    assert!(!old);
+                })
+            };
+            let callback_resume_vcpus = {
+                let cpus_running = cpus_throttled.clone();
+                Box::new(move || {
+                    let old = cpus_running.swap(false, Ordering::SeqCst);
+                    assert!(old);
+                })
+            };
+
+            // State transitions: Waiting -> Throttle -> Waiting -> Throttle -> Exit
+            {
+                let mut handler =
+                    ThrottleThreadHandle::new(callback_pause_vcpus, callback_resume_vcpus);
+                handler.set_throttle_percent(5);
+                sleep(Duration::from_millis(ThrottleWorker::TIMESLICE_MS));
+                handler.set_throttle_percent(10);
+                sleep(Duration::from_millis(ThrottleWorker::TIMESLICE_MS));
+
+                // Assume we aborted vCPU throttling (or the live-migration at all).
+                handler.set_throttle_percent(0 /* reset to waiting */);
+                handler.set_throttle_percent(5);
+                sleep(Duration::from_millis(ThrottleWorker::TIMESLICE_MS));
+                handler.set_throttle_percent(10);
+                sleep(Duration::from_millis(ThrottleWorker::TIMESLICE_MS));
+
+                // The test is successful if we don't have a panic here due to a
+                // closed channel.
+                for _ in 0..10 {
+                    handler.shutdown();
+                    sleep(Duration::from_millis(1));
+                }
+
+                // The test is successful if it does not get stuck.
+                drop(handler);
+            }
+        }
+    }


This will suffice for the PoC, but when you start preparing this for production then it would be cool if you would instead insert callbacks that record certain events (e.g. the timestamps when they get called, how many times they are called, etc) and check that this (roughly) corresponds to what one would expect.

olivereanderson · 2025-08-22T15:03:18Z

vmm/src/vcpu_throttling.rs

+    ///   function must not perform any artificial delay itself.
+    /// - `callback_resume_vcpus`: Function putting all vCPUs back into running
+    ///   state. The function must not perform any artificial delay itself.
+    fn new(


Nit:

Suggested change

fn new(

fn spawn(

olivereanderson · 2025-08-22T15:04:49Z

vmm/src/vcpu_throttling.rs

+    ///
+    /// # Parameters
+    /// - `cpu_manager`: CPU manager to pause and resume vCPUs
+    pub fn new_from_cpu_manager(cpu_manager: &Arc<Mutex<CpuManager>>) -> Self {


Nit:

Suggested change

pub fn new_from_cpu_manager(cpu_manager: &Arc<Mutex<CpuManager>>) -> Self {

pub fn spawn_with_cpu_manager(cpu_manager: Arc<Mutex<CpuManager>>) -> Self {

vmm/src/vcpu_throttling.rs

auto-converge (vCPU throttling) is a technique combined with precopy live-migration flows to migrate VMs with a high dirty rate (high working set with many writes). It is an alternative to postcopy migration, which is not yet implemented in Cloud Hypervisor. By throttling the vCPUs incrementally, the dirty rate drops and the VM migrates (converges) eventually. More specifically, the reduced dirty rate ensures that the configured downtime can be reached. The implementation is inspired by QEMU, but adapted to Cloud Hypervisor. Various discussions, intermediate steps, and experiments lead to this final result. vCPU throttling was implemented with a dedicated thread and a manager for that thread. This thread utilizes the CpuManager's pause() and resume() in conjunction with (interruptible) sleeps to apply the current throttling percentage onto the vCPUs, thus the VM. The implementation is designed to not block or delay normal operation any longer than necessary. The proposed design relies on the recent improvements and fixes for CpuManager's pause() and resume(). For correctness, on each pause/resume cycle, the time for these actions is measured. This way, a dynamic timeslice can be used, guaranteeing the VM is indeed throttled at the indented percentage. Although not supported yet by Cloud Hypervisor, this thread will support throttling cancellation when live-migrations are cancelled. This was intensively tested in an automated setup with thousands of live-migrations with VMs under load. - auto-converging starts always after two memory delta transfer iterations - every two iterations, it is increased (step size is 10%) - maximum throttling is 99% - the VM will get slower. At 99% throttling, it will be unsurprisingly barely usable. This is something users have to accept if they want to migrate their VMs running heavy workloads. Signed-off-by: Philipp Schuster <[email protected]> Reviewed-by: Stefan Kober <[email protected]> Reviewed-by: Oliver Anderson <[email protected]> Reviewed-by: Thomas Prescher <[email protected]> On-behalf-of: SAP [email protected]

phip1611 self-assigned this Aug 18, 2025

phip1611 force-pushed the cyberus-fork-poc-precopy-autoconverge branch from 86db5eb to f2978b1 Compare August 18, 2025 15:03

phip1611 requested review from hertrste, olivereanderson and tpressure August 18, 2025 15:04

phip1611 force-pushed the cyberus-fork-poc-precopy-autoconverge branch from f2978b1 to d7003ea Compare August 18, 2025 15:10

phip1611 marked this pull request as draft August 18, 2025 15:11

phip1611 changed the title ~~PoC: vCPU throttling/auto-converge~~ gardenlinux: PoC: vCPU throttling/auto-converge Aug 18, 2025

phip1611 commented Aug 18, 2025

View reviewed changes

vmm/src/lib.rs Show resolved Hide resolved

hertrste reviewed Aug 19, 2025

View reviewed changes

docs/live_migration.md Show resolved Hide resolved

hertrste reviewed Aug 19, 2025

View reviewed changes

vmm/src/api/openapi/cloud-hypervisor.yaml Show resolved Hide resolved

phip1611 commented Aug 19, 2025

View reviewed changes

vmm/src/vcpu_throttling.rs Outdated Show resolved Hide resolved

olivereanderson reviewed Aug 19, 2025

View reviewed changes

vmm/src/lib.rs Outdated Show resolved Hide resolved

olivereanderson reviewed Aug 19, 2025

View reviewed changes

vmm/src/lib.rs Outdated Show resolved Hide resolved

phip1611 force-pushed the cyberus-fork-poc-precopy-autoconverge branch 2 times, most recently from 5554499 to 36985cb Compare August 19, 2025 13:46

This comment was marked as outdated.

Sign in to view

phip1611 force-pushed the cyberus-fork-poc-precopy-autoconverge branch from 36985cb to 7b8e12a Compare August 20, 2025 06:21

phip1611 requested a review from olivereanderson August 20, 2025 06:22

This comment was marked as outdated.

Sign in to view

phip1611 force-pushed the cyberus-fork-poc-precopy-autoconverge branch 6 times, most recently from b291803 to 464bc74 Compare August 22, 2025 11:14

phip1611 marked this pull request as ready for review August 22, 2025 11:46

phip1611 force-pushed the cyberus-fork-poc-precopy-autoconverge branch from 464bc74 to fc98bdb Compare August 22, 2025 11:49

phip1611 commented Aug 22, 2025

View reviewed changes

vmm/src/vm.rs Outdated Show resolved Hide resolved

phip1611 commented Aug 22, 2025

View reviewed changes

phip1611 force-pushed the cyberus-fork-poc-precopy-autoconverge branch 3 times, most recently from 502eaff to f4bd5b4 Compare August 22, 2025 13:04

olivereanderson approved these changes Aug 22, 2025

View reviewed changes

olivereanderson reviewed Aug 25, 2025

View reviewed changes

vmm/src/vcpu_throttling.rs Show resolved Hide resolved

phip1611 force-pushed the cyberus-fork-poc-precopy-autoconverge branch from f4bd5b4 to 55a1549 Compare August 25, 2025 09:16

phip1611 marked this pull request as draft August 27, 2025 14:24

phip1611 force-pushed the cyberus-fork-poc-precopy-autoconverge branch 4 times, most recently from c2bc313 to ff9d41e Compare September 3, 2025 10:55

phip1611 force-pushed the cyberus-fork-poc-precopy-autoconverge branch 3 times, most recently from 3dd8f15 to 0b82a8f Compare September 12, 2025 11:55

phip1611 force-pushed the cyberus-fork-poc-precopy-autoconverge branch from 0b82a8f to 1b1cabb Compare September 12, 2025 12:52

tpressure approved these changes Sep 15, 2025

View reviewed changes

phip1611 marked this pull request as ready for review September 15, 2025 07:58

phip1611 merged commit 5c96d5f into cyberus-technology:gardenlinux Sep 15, 2025
20 of 21 checks passed

phip1611 deleted the cyberus-fork-poc-precopy-autoconverge branch September 15, 2025 07:59

phip1611 mentioned this pull request Nov 14, 2025

Live Migration Feature Tracking and Collaboration cloud-hypervisor/cloud-hypervisor#7111

Open

26 tasks

	let vcpu_throttle_thread_handle = ThrottleThreadHandle::new_from_cpu_manager(&cpu_manager);
	let vcpu_throttler = ThrottleThreadHandle::new_from_cpu_manager(&cpu_manager);

	pub fn new_from_cpu_manager(cpu_manager: &Arc<Mutex<CpuManager>>) -> Self {
	pub fn spawn_with_cpu_manager(cpu_manager: Arc<Mutex<CpuManager>>) -> Self {

gardenlinux: PoC: vCPU throttling/auto-converge #13

gardenlinux: PoC: vCPU throttling/auto-converge #13

Uh oh!

Conversation

phip1611 commented Aug 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

About

Implementation

Effective Behaviour

Hints for Reviewers

Steps to Undraft

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

tpressure commented Aug 19, 2025

Uh oh!

Uh oh!

Uh oh!

This comment was marked as outdated.

Uh oh!

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

This comment was marked as outdated.

Uh oh!

phip1611 Aug 22, 2025

Choose a reason for hiding this comment

Uh oh!

olivereanderson left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

olivereanderson Aug 22, 2025

Choose a reason for hiding this comment

Uh oh!

olivereanderson Aug 22, 2025

Choose a reason for hiding this comment

Uh oh!

olivereanderson Aug 22, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

phip1611 commented Aug 18, 2025 •

edited

Loading

olivereanderson left a comment •

edited

Loading