You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit a6442f9
Browse filesBrowse the repository at this point in the historyBrowse files
Resuming a microVM actor on the same node no longer reads its rootfs upper back from disk (#3485)
Why: In the microVM burn-in (GKE downstream testing), wake p50 went from
~350ms to ~1.1s (!), almost all of it in upper_join. Reported to me by
@annapendleton though I'm not sure who originally found it.
#3353 started writing out snapshot tars as they're written, so their
dirty pages can't run a tight worker pod out of memory. It also dropped
the written pages from the page cache, so a resume on the same node read
the rootfs upper tar back from disk ... which is slow. It also made
every suspend write the tars at disk speed.
Fixes by commit order:
1. Pull summing memory.stat keys out of memoryUsed, no behavior change.
2. Clean pages are reclaimable, so dropping them is irrelevant for OOM
safety.
- The tars now stay cached. Write-out waits on the previous chunk
instead of the one just written, so dirty pages stay at about two chunks
without stalling on each one.
4. Start snapshot writeback after the OnDemand delta merge, which
replaces memory-ranges, so it tracks the merged file instead of the
unlinked delta.
5. Write snapshot files back in the background instead of with a
blocking fdatasync.
- Starting write-out can block for a long time on a busy disk
(https://github.com/torvalds/linux/blob/v6.12/fs/sync.c#L315-L317), and
holding deleted snapshot files open meanwhile leaked disk under load, so
it starts write-out once, 64MiB at a time, and closes a file as soon as
it's unlinked, which drops its dirty pages.
7. What snapshot writes can borrow leaves clean page cache out, so
cached tars don't make a pod look "full".
- A worker pod without a memory limit (e.g. the deploy_locust.sh
default) has nothing to protect, so it lends all of it instead of
capping every suspend as if the pod were full.
9. When the pod can lend memory for the whole archive, the tar skips the
write-behind and goes to the page cache at full speed, and the loan
comes back once it's on disk. Otherwise the write-behind still bounds
its dirty pages.
That said, with respect to workers without limits: We should probably
consider recommending memory limits on worker pods. I will think about
it overnight and propose docs updates tomorrow.
Tested with locust burn-in tests (see comments in the PR).
AI use: AI-assisted, I reviewed and tested it.
- [x] Tests pass
- [x] Docs updated
---------
Signed-off-by: Benjamin Elder <[email protected]>
0 commit comments