Resuming a microVM actor on the same node no longer reads its rootfs upper back from disk - #3485

I'm going ahead and pursuing the density-aware-write-behind to scope out if we can reasonably include that as well, just to bring the performance back inline to before OOMkiller on workers that aren't packed to the gills, but for now doing that in another branch as a POC.

I'm going ahead and pursuing the density-aware-write-behind to scope out if we can reasonably include that as well, just to bring the performance back inline to before OOMkiller on workers that aren't packed to the gills, but for now doing that in another branch as a POC.

this is not too bad actually, and it has a similarly bad regression without.

EDIT: added

GKE burn-in: rc.1 vs rc.1 + #3485 + #3476

Tested this PR together with #3476 on top of v1.0.0-rc.1. This PR's change (keeping snapshot tars in the page cache and skipping the write-behind) targets the microVM upper_join regression measured below.

Result: with both PRs applied, the #3353 suspend/resume regression is gone on microVM, and the gVisor suspend tail shrinks. microVM upper_join drops from p50 310 → 29 ms and p90 796 → 40 ms, back to the pre-#3353 level. No new regressions turned up.

Setup

• Same clusters, control plane, settings and load for both runs. The control plane was the published v1.0.0-rc.1 images, byte-identical in both runs. Only the worker image changed.
• GKE 1.37.1, 5 × c3-standard-8 per cluster, one microVM cluster and one gVisor cluster.
• deploy_locust.sh --worker-count 4 --actor-memory 1Gi with unsized workers (no --worker-memory), agentsession user class.
• Ladder 6:2h,19:1h,40:20m,0:45m,13:90m,2:25m, chaos off.
• Window: the first ~4.5 minutes of the 6-user step for each build.
• PATCHED = v1.0.0-rc.1 + the 4 commits of Resuming a microVM actor on the same node no longer reads its rootfs upper back from disk #3485 + the 2 commits of leave page cache out of the memory guard's overshoot check #3476, cherry-picked cleanly. I checked the running worker pods' digests and the commit embedded in the worker binaries before starting load.

microVM

gVisor

Samples per build: microVM ~215 → ~260, gVisor ~270 → ~325 restores and checkpoints. The patched window was 23 s longer; the rest of the increase comes from faster cycles.

Regression checks (patched): 0 locust failures (baseline 0). 0 ERROR-level lines in atelet or worker logs. 0 worker or ate-system restarts. No OOMKilled or Evicted events. No actors stuck in RESUMING/SUSPENDING. No new warning types: the only WARN in either run is the same pre-existing OpenTelemetry SDK error, at the same rate. vm_restore and gVisor wake/resume are unchanged, as expected.

Caveats

• Both PRs were applied together, so this doesn't attribute the gains to one PR or the other.
• This covers only the early 6-user window. The 2 h step, the T+3 h 40-user burst and the full 6–12 h ladder weren't run, and neither was the sized-worker variant (--worker-memory).
• The pre-Suspending actors no longer OOM-kills tight worker pods #3353 upper_join figure (~30 ms) comes from the original burn-in, not a local 45ab64df build.
• microVM SuspendActorAsync max stayed ~6.3 s even though median/avg dropped ~72%, so a rare slow suspend remains.

Correction: the micro-VM checkpoint tail is not back to pre-break yet

My comment above used only the first ~4.5 minutes of the 6-user step, and that overstated the micro-VM result. Over a longer window (same patched build, same clusters, still 6 users), the micro-VM checkpoint tail grows with time. gVisor stays fixed.

Patched, first 22 minutes of the 6-user step (exact percentiles, ms; 0 failures)

micro-VM worker_checkpoint per 5 minutes, constant 6 users:

Compared with the original burn-in: Round 1 = 28520c0d, before #3353; Round 2 = 34cce1e9, after it. Those were full runs including 40-user bursts.

• Round 1: micro-VM worker_checkpoint p50 / p90 / p99 = 250 / 355 / 2,200 ms.
• Round 2: 3,100 / 10,900 / 19,900 ms.
• Patched (6 users only): 239 / 1,266 / 5,624 ms. The median is back to Round 1 levels, but the tail is still well above Round 1, even though Round 1's tail includes bursts.
• gVisor: suspend p90 / p99 is 280 / 470 ms patched, against Round 1 680 ms / 16 s and Round 2 10.1 s / 30 s.

Where the slow micro-VM checkpoints spend their time (the worker's own "Checkpoint timing breakdown"):

So the remaining tail is in the guest-memory snapshot phase (plus occasional teardown spikes), not in rootfs_upper. I can't yet tell whether this is a part of the regression these PRs don't cover, or something they introduce (for example, page cache kept by #3485 slowing guest-memory writes over time). To find out, I'll run the rc.1 workers over the same 22+ minute window, and I'm keeping the patched ladder running through the 40-user burst at T+3 h. I'll follow up with both.

Second GKE burn-in confirms Grant McCloskey (@MushuEE)'s results, including a node-disk regression to fix before merge

Grant McCloskey (@MushuEE)'s runs above (v1.0.0-rc.1 + #3485 + #3476) and an independent run on our side (main @ 34cce1e + this PR's 4 commits; #3476 doesn't apply on that base) agree. Setup for ours: GKE 1.37, 5 × c3-standard-8, microVM, 4 unsized benchmark workers (no pod memory limit), agentsession load with 1Gi actors, 30-minute high-churn ladder.

Latency medians are fixed 👍 upper_join p50 ~300–840 ms → ~20–30 ms, and microVM worker_checkpoint p50 ~0.7–3 s → ~200–250 ms. Grant McCloskey (@MushuEE) also saw the gVisor suspend tail fixed.

Regression: node disk fills on microVM nodes.

• Grant McCloskey (@MushuEE) saw it at higher load and over a longer run: one node went 22 → 93.5 of 94 GiB in ~10 min. That caused DiskPressure, one ephemeral-storage eviction and container-runtime errors. Then disk and page cache were freed together. rc.1 alone stayed ≤ 12.9 GiB under the same load, and gVisor nodes stayed flat.
• Our run is still in progress, so we don't have peak numbers yet. So far, nodes with a worker show the same pattern in a milder form: disk jumps by several GiB within a minute and then frees, with 2–4 GiB of dirty pages per node. The node without a worker stays flat.
• The microVM checkpoint tail grows with load: worker_checkpoint p90 8.3 s / p99 17 s at 19 users. That matches Grant McCloskey (@MushuEE)'s correction above.

Cause (confirmed on a node in our run): /var had 21 GiB used, while everything visible under /var/lib/ate was 2 GiB. ateworker-microvm held ~54 GiB (apparent size) of deleted memory-ranges files open (…/local-checkpoint/<uuid>/memory-ranges (deleted), …/checkpoint-state/memory-ranges (deleted)).

• snapshotcache.StartWriteback opens every file up front and closes each only after its own fdatasync.
• atelet uploads and deletes the checkpoint in the meantime.
• This PR lifts memory.high in pods without a limit (and adds a writeback per tar in archiveDir). So each sync has much more dirty data to flush, and deleted checkpoints stay pinned for minutes.
• The writeback call itself dates from Suspending actors no longer OOM-kills tight worker pods #3353. With the cap in place its syncs were short, which fits rc.1 looking clean.

Suggestions:

1. In StartWriteback (and Avoid synchronous fdatasync in snapshotcache writeback and eviction #3477's polling variant), open, sync and close one file at a time. If fstat shows Nlink == 0, close immediately and return the loan, since a deleted file's dirty pages don't need writing.
2. Cap what a pod without a memory limit can lend, so dirty pages stay bounded per node.

Happy to re-run either change on the same setups.

Drafted and posted by an AI assistant on behalf of Anna Pendleton (@annapendleton).

Candidate fix under test: StartWriteback pinning deleted snapshot files

What we saw. In the burn-in with v1.0.0-rc.1 + #3485 + #3476 (unsized worker pods, micro-VM), node disk usage on some worker nodes climbed by tens of GiB with no visible file accounting for it. On one node it went from 22 to 93.5 of 94 GiB in ~10 minutes while /var/lib/ate stayed at 3–5 GiB, then disk and page cache were freed together. Another node evicted its worker for low ephemeral storage. rc.1 and v0.6.0 workers under the same load stayed flat (~10–13 GiB).

Suspected mechanism (not yet confirmed). snapshotcache.StartWriteback opens every file up front and syncs them one after another, closing each only after its fdatasync. This PR adds a call per snapshot tar (archiveDir, always taken in unsized pods). A file queued behind a slow sync stays open after atelet uploads and deletes it, so its blocks stay allocated. And fdatasync still forces out pages the kernel would otherwise drop. #3477 changes the same function (it closes unlinked files while polling), so this may overlap with it.

What we're testing. The patch below, on top of rc.1 + #3485 + #3476:

• open each file only when its turn comes;
• skip it if it has been unlinked or its path names another file;
• cap concurrent fdatasync at 2 per process.

done still runs once every file is on disk or gone. We'll report whether node disk stays flat and whether the micro-VM checkpoint tail moves closer to v0.6.0's (micro-VM worker_checkpoint p50/p90/p99 = 225/253/366 ms on v0.6.0, vs 239/1,266/5,624 ms with #3485 + #3476 over the same 22-minute window).

diff --git a/internal/snapshotcache/snapshotcache.go b/internal/snapshotcache/snapshotcache.go
index c3812a96..7302ad07 100644
--- a/internal/snapshotcache/snapshotcache.go
+++ b/internal/snapshotcache/snapshotcache.go
@@ -46,6 +46,8 @@ var (
 	// tmpfs, overlayfs) or kernel version.
 	cachestatFn = unix.Cachestat
 	fadviseFn   = unix.Fadvise
+	// fdatasyncFn is a seam for tests of StartWriteback.
+	fdatasyncFn = unix.Fdatasync
 
 	// RetryDelays are the backoff intervals between post-restore eviction
 	// passes when dirty pages are waiting for background kernel writeback.
@@ -111,25 +113,39 @@ func DropFiles(files []File) bool {
 	return dropped
 }
 
+// writebackSlots bounds how many StartWriteback fdatasync calls run at once in
+// this process. Concurrent fdatasync calls contend on the filesystem journal,
+// so each takes longer, and a file pinned by a slow sync can't be freed.
+var writebackSlots = make(chan struct{}, 2)
+
+// pendingWriteback is a file StartWriteback will sync, pinned to the inode the
+// path named when the write finished.
+type pendingWriteback struct {
+	path     string
+	dev, ino uint64
+}
+
 // StartWriteback writes each path's dirty pages to disk in the background and
 // then calls done, if set. Writing can block on a busy disk, and callers don't
 // wait.
+//
+// A file is opened only when its turn comes, and skipped if by then it has been
+// unlinked or its path names another file: the kernel drops a deleted file's
+// dirty pages without writing them, so syncing it would only pin its blocks and
+// slow the other writes. Either way its pages no longer count against the pod,
+// so done still means every file is on disk or gone.
 func StartWriteback(paths []string, done func()) error {
-	var files []*os.File
+	pending := make([]pendingWriteback, 0, len(paths))
 	for _, p := range paths {
-		f, err := os.Open(p)
-		if err != nil {
-			for _, f := range files {
-				_ = f.Close()
-			}
+		var st unix.Stat_t
+		if err := unix.Stat(p, &st); err != nil {
 			return err
 		}
-		files = append(files, f)
+		pending = append(pending, pendingWriteback{path: p, dev: uint64(st.Dev), ino: st.Ino})
 	}
 	go func() {
-		for _, f := range files {
-			_ = unix.Fdatasync(int(f.Fd()))
-			_ = f.Close()
+		for _, w := range pending {
+			writeback(w)
 		}
 		if done != nil {
 			done()
@@ -138,6 +154,23 @@ func StartWriteback(paths []string, done func()) error {
 	return nil
 }
 
+// writeback syncs w if the path still names the same, linked file.
+func writeback(w pendingWriteback) {
+	writebackSlots <- struct{}{}
+	defer func() { <-writebackSlots }()
+	f, err := os.Open(w.path)
+	if err != nil {
+		return // removed: nothing left to write back
+	}
+	defer f.Close()
+	var st unix.Stat_t
+	if err := unix.Fstat(int(f.Fd()), &st); err != nil || st.Nlink == 0 ||
+		uint64(st.Dev) != w.dev || st.Ino != w.ino {
+		return
+	}
+	_ = fdatasyncFn(int(f.Fd()))
+}
+
 // SyncAndEvict waits for targets to reach disk and then evicts them, so pages
 // still dirty from a recent snapshot get dropped too.
 func SyncAndEvict(targets []File) {
diff --git a/internal/snapshotcache/writeback_test.go b/internal/snapshotcache/writeback_test.go
new file mode 100644
index 00000000..234c706a
--- /dev/null
+++ b/internal/snapshotcache/writeback_test.go
@@ -0,0 +1,192 @@
+//go:build linux
+
+// Copyright 2026 Google LLC
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package snapshotcache
+
+import (
+	"os"
+	"path/filepath"
+	"strconv"
+	"sync"
+	"testing"
+	"time"
+)
+
+// syncRecorder replaces fdatasyncFn and records which files were synced.
+type syncRecorder struct {
+	mu    sync.Mutex
+	paths []string
+}
+
+func (r *syncRecorder) install(t *testing.T) {
+	t.Helper()
+	prev := fdatasyncFn
+	fdatasyncFn = func(fd int) error {
+		p, err := os.Readlink("/proc/self/fd/" + strconv.Itoa(fd))
+		if err != nil {
+			p = "?"
+		}
+		r.mu.Lock()
+		r.paths = append(r.paths, p)
+		r.mu.Unlock()
+		return nil
+	}
+	t.Cleanup(func() { fdatasyncFn = prev })
+}
+
+func (r *syncRecorder) synced() []string {
+	r.mu.Lock()
+	defer r.mu.Unlock()
+	return append([]string(nil), r.paths...)
+}
+
+// holdSlots occupies every writeback slot until the returned func runs, so a
+// StartWriteback started meanwhile waits before opening anything.
+func holdSlots(t *testing.T) func() {
+	t.Helper()
+	for range cap(writebackSlots) {
+		writebackSlots <- struct{}{}
+	}
+	var once sync.Once
+	release := func() {
+		once.Do(func() {
+			for range cap(writebackSlots) {
+				<-writebackSlots
+			}
+		})
+	}
+	t.Cleanup(release)
+	return release
+}
+
+func writeFile(t *testing.T, path, data string) {
+	t.Helper()
+	if err := os.WriteFile(path, []byte(data), 0o600); err != nil {
+		t.Fatal(err)
+	}
+}
+
+func waitDone(t *testing.T, done <-chan struct{}) {
+	t.Helper()
+	select {
+	case <-done:
+	case <-time.After(10 * time.Second):
+		t.Fatal("done was not called")
+	}
+}
+
+func TestStartWritebackSyncsLinkedFile(t *testing.T) {
+	var rec syncRecorder
+	rec.install(t)
+	p := filepath.Join(t.TempDir(), "memory-ranges")
+	writeFile(t, p, "pages")
+
+	done := make(chan struct{})
+	if err := StartWriteback([]string{p}, func() { close(done) }); err != nil {
+		t.Fatal(err)
+	}
+	waitDone(t, done)
+	if got := rec.synced(); len(got) != 1 || got[0] != p {
+		t.Errorf("synced %v, want [%s]", got, p)
+	}
+}
+
+// A file unlinked while its writeback is queued is never synced (its dirty
+// pages are dropped, not written), and done still runs.
+func TestStartWritebackSkipsFileUnlinkedWhileQueued(t *testing.T) {
+	var rec syncRecorder
+	rec.install(t)
+	dir := t.TempDir()
+	gone, kept := filepath.Join(dir, "rootfs-upper.tar"), filepath.Join(dir, "state.json")
+	writeFile(t, gone, "tar")
+	writeFile(t, kept, "state")
+
+	release := holdSlots(t)
+	done := make(chan struct{})
+	if err := StartWriteback([]string{gone, kept}, func() { close(done) }); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.Remove(gone); err != nil {
+		t.Fatal(err)
+	}
+	release()
+	waitDone(t, done)
+	if got := rec.synced(); len(got) != 1 || got[0] != kept {
+		t.Errorf("synced %v, want only [%s]", got, kept)
+	}
+}
+
+// A path that names a different file by its turn is skipped: the file that was
+// written is gone, and the new one isn't StartWriteback's to sync.
+func TestStartWritebackSkipsReplacedFile(t *testing.T) {
+	var rec syncRecorder
+	rec.install(t)
+	p := filepath.Join(t.TempDir(), "memory-ranges")
+	writeFile(t, p, "old")
+
+	release := holdSlots(t)
+	done := make(chan struct{})
+	if err := StartWriteback([]string{p}, func() { close(done) }); err != nil {
+		t.Fatal(err)
+	}
+	if err := os.Remove(p); err != nil {
+		t.Fatal(err)
+	}
+	writeFile(t, p, "new")
+	release()
+	waitDone(t, done)
+	if got := rec.synced(); len(got) != 0 {
+		t.Errorf("synced %v, want nothing", got)
+	}
+}
+
+// Queued writebacks hold no open file: an open descriptor on a deleted file
+// pins its blocks, which is how unsized micro-VM workers filled node disks.
+func TestStartWritebackOpensNothingWhileQueued(t *testing.T) {
+	var rec syncRecorder
+	rec.install(t)
+	p := filepath.Join(t.TempDir(), "memory-ranges")
+	writeFile(t, p, "pages")
+
+	release := holdSlots(t)
+	done := make(chan struct{})
+	if err := StartWriteback([]string{p}, func() { close(done) }); err != nil {
+		t.Fatal(err)
+	}
+	time.Sleep(50 * time.Millisecond)
+	ents, err := os.ReadDir("/proc/self/fd")
+	if err != nil {
+		t.Fatal(err)
+	}
+	for _, e := range ents {
+		if target, err := os.Readlink(filepath.Join("/proc/self/fd", e.Name())); err == nil && target == p {
+			t.Errorf("%s is open while its writeback is queued", p)
+		}
+	}
+	release()
+	waitDone(t, done)
+}
+
+func TestStartWritebackMissingPathFailsUpFront(t *testing.T) {
+	called := false
+	err := StartWriteback([]string{filepath.Join(t.TempDir(), "missing")}, func() { called = true })
+	if err == nil {
+		t.Fatal("StartWriteback succeeded for a missing path")
+	}
+	if called {
+		t.Error("done was called for a writeback that never started")
+	}
+}

Anna Pendleton (@annapendleton) Thanks, great find, that's a big miss.

Pulled #3477's writeback rework in (Chenyi Wang (@chw120)): it closes each file as soon as it's clean or unlinked instead of holding all of them through fdatasync.

On a 15 round soak of 3 filled actors in an unsized pool, /var stayed flat and deleted files held open went to 0 within 30s. Could you rerun your burn-in on the new head?

Tim Hockin (@thockin) fixed the should-fix items in separate commits. Agreed this needs a holistic redesign after 1.0.0, these are stopgaps for the regression, and frankly the whole PR is less curated than I'd generally aim for given the time constraints.

I also had an epoll cleanup / optimization for all this polling logic I'm ~happy with prior to this PR, but I'm keeping it out of this PR to focus on the regressions.

Similarly #3431 is something we should take a hard look at, but it's not a regression and can wait.

Results: #3485 + #3476 + the StartWriteback fix (micro-VM, first 22 minutes)

The fix is the patch in my previous comment (#3485 (comment)). It changes only snapshotcache.StartWriteback:

• open each file only when its turn comes;
• skip it if unlinked or replaced;
• at most 2 concurrent fdatasync per process.

Same setup as the other builds:

• v1.0.0-rc.1 control plane (identical image digests);
• GKE 1.37, 5 × c3-standard-8, unsized worker pods, 1Gi actors;
• agentsession ladder, first 22 minutes of the 6-user step;
• the cluster was cleaned and its nodes replaced before this run.

Only the worker image differs. v0.6.0 is the whole v0.6.0 stack, pre-#3353; its benchmark client needed #3089 to reach the router.

micro-VM, p50 / p90 / p99 (ms)

0 request failures in all four.

What changed

• The guest-memory snapshot tail is nearly back to v0.6.0 (p99 349 vs 5,197 ms).
• The checkpoint p90 no longer grows over time at constant load.
• The remaining tail is mostly teardown (p99 ~1.9 s vs 178 ms on v0.6.0).

Disk / page cache

• Node disk stayed at 9.1–15.2 GiB with no evictions. With Resuming a microVM actor on the same node no longer reads its rootfs upper back from disk #3485 + leave page cache out of the memory guard's overshoot check #3476 it reached 93.5 of 94 GiB on one node and a worker was evicted.
• Dirty page cache peaked at 1.9 GiB, against ~4 GiB on 3 of 5 nodes with Resuming a microVM actor on the same node no longer reads its rootfs upper back from disk #3485 + leave page cache out of the memory guard's overshoot check #3476.
• The deleted-file holder is confirmed: a sampler listing deleted-but-open files shows the micro-VM worker process holding deleted checkpoint files (/var/lib/ate/snapshots/<actor>/local-checkpoint/…/memory-ranges).

There's a second path the fix doesn't cover. At one point a single worker held 7 deleted memory-ranges files (~5 GiB of real disk, released shortly after). StartWriteback now holds at most 2, so these must come from the post-restore snapshotcache.SyncAndEvict. It runs once per restore in its own goroutine and does a blocking fdatasync on each file, with no concurrency limit and even when the file has already been unlinked. #3477 changes this path too. The equivalent change here would be to skip unlinked or replaced targets (t.Info is already pinned) and share the same slots.

Caveats

• One 22-minute window per build. The 6-user step is where Resuming a microVM actor on the same node no longer reads its rootfs upper back from disk #3485 + leave page cache out of the memory guard's overshoot check #3476 started to diverge. The disk growth with Resuming a microVM actor on the same node no longer reads its rootfs upper back from disk #3485 + leave page cache out of the memory guard's overshoot check #3476 showed up 1–2 h in (mostly at 19 users), so the leak check on this build continues through that step.
• micro-VM only.

Thanks for the results

There's a second path the fix doesn't cover. At one point a single worker held 7 deleted memory-ranges files (~5 GiB of real disk, released shortly after). StartWriteback now holds at most 2, so these must come from the post-restore snapshotcache.SyncAndEvict. It runs once per restore in its own goroutine and does a blocking fdatasync on each file, with no concurrency limit and even when the file has already been unlinked. #3477 changes this path too. The equivalent change here would be to skip unlinked or replaced targets (t.Info is already pinned) and share the same slots.

That should be in HEAD already, unless I'm missing something, no fdatasync left anywhere (133dcf5 + #3476)

Burn-in update: #3485 @ 133dcf52 (includes #3477) + #3476 on rc.1, first 90 minutes

gVisor looks healthy: worker checkpoint p99 is 314 ms and node disk is flat.

Micro-VM still leaks disk. One node climbed to 79 GiB, and the kubelet evicted its worker about an hour in. The worker was holding 15–30 open handles to deleted 1 GiB checkpoint-state/memory-ranges files.

Micro-VM median latency is back to v0.6.0 levels (239 ms), but the tail is worse than the earlier #3485 run: p90 2,863 ms, p99 6,133 ms, with nearly all of it in snapshot.

The new teardown timing shows the teardown tail is almost all rootfs upper removal (p99 2.2 s), which fits disk contention. The full ladder follows tonight.

I can't repro the leak so far, fc9ece2 may be the fix though. Pushed some fixes based on Dmitry Berkovich (@dberkov) AI scans + more adversarial review locally.

1183627 fixes the leak and the checkpoint tail.

The writeback goroutine was blocking in sync_file_range while holding deleted snapshot files, see https://github.com/torvalds/linux/blob/v6.12/fs/sync.c#L315-L317.

It now starts write-out once, 64MiB at a time, and closes a file as soon as it's unlinked.

A/B on 4 identical clusters matching the burn-in setup from Grant McCloskey (@MushuEE) (5x c3-standard-8, 4 unsized workers, 1Gi actors, agentsession at 6 users for 45m, started together), 32432e1 vs 1183627:

• microVM checkpoint p50/p90/p99: 247/3052/6551ms -> 246/312/1356ms, 19% -> 1% over 1s
• microVM node disk peak: 50.8GiB -> 17.7GiB, deleted files held with no links: 19.0GiB -> 2.0GiB
• gVisor checkpoint p50/p90/p99: 222/2570/4471ms -> 213/259/2052ms, 22% -> 2% over 1s
• gVisor deleted files held with no links: 8.0GiB -> 1.5GiB
• no failures, evictions or restarts

The base matches the 9f90899 burn-in on both runtimes, including the slow busiest node, and with the fix the busiest node looks like the others.

Second run on the same clusters and builds. The base didn't fall behind this time, load spread more evenly across the workers, so checkpoint latency came out about the same on both. The base still held ~6x more deleted micro-VM data though.

• microVM over 1s: 1% on both, deleted files held with no links: 14.2GiB -> 2.3GiB, node disk peak 19.5GiB -> 16.9GiB
• gVisor over 1s: 0% on both, deleted files held with no links: 1.1GiB -> 0.4GiB
• 3 failures on the base, all "another operation is in progress for this actor" from one session collision, none on the fix

So the slowdown needs one node to fall behind. The base did that in one of two runs, the fix in neither.

Actor placement is uneven in every run, often one worker gets no actors at all. That's separate from this PR, I'll file an issue after some further debugging.

I also added a cleanup for the global test seam kludge introduced by this PR.
We inject a local private interface the way I'd write it by hand.
It has no functional changes so I'm not rushing to burn-in test that one.

I resisted the urge to do this for nearby code we didn't touch, but we really need a cleanup pass over all of this.

Squashed into 6 commits and rebased on main for easier review, one change per commit.

Same code as d6d3b1c earlier with the burn-in results, the only diff in the PR's files is the comments now stating pre- and post-conditions.

git diff d6d3b1ca7 68a88d63a -- $(git diff --name-only 0ddbf78d5 68a88d63a)

Given we need N=2 reviews anyhow and it is late, we might as well stash the branch as-is, fix those, and qualify overnight in this case. nobody else has reviewed the current code.

If it qualifies, I will push the update. Otherwise I will leave it as-is, because I know this code is thoroughly vetted to work and none of these are blocking in my opinion but some form of the fix is needed for GA.

Realistically, this code is so subtle that nobody can read it and fully comprehend. The proof is in the testing.

Dima previously approved, so I will enqueue this,