Repository navigation
A microVM actor using more host memory than it booked no longer takes down its worker - #3429
Conversation
Signed-off-by: Benjamin Elder <[email protected]>
Signed-off-by: Benjamin Elder <[email protected]>
Benjamin Elder (BenTheElder)
added
area/microVM
breaking-change
| } | ||
| // the page cache the write adds isn't the actor running away | ||
| if ra != nil { | ||
| defer ra.memory.Resume(ra.memory.Pause()) |
There was a problem hiding this comment.
This re-arms the guard as soon as the write returns, while the snapshot's writeback pages are still charged to the leaf. Restoring memory.high doesn't reclaim them, so the overshoot check (memwatch.go:74) can kill the paused VM on the next tick, and a checkpoint that fails after the write gets reported as a memory kill. Could the guard stay paused through teardown, and on error re-arm only once that cache is gone (or have the overshoot check exclude file cache)?
There was a problem hiding this comment.
Hi Claude :-)
I'll follow up on this when I'm at a keyboard. Also looking for results from downstream scale / density testing.
| func (s *AteWorkerService) leafMemoryHigh(memoryBytes int64) int64 { | ||
| if !s.throttleMemory || memoryBytes <= 0 { | ||
| return 0 | ||
| } |
There was a problem hiding this comment.
With memory.high at the booking, the 4MiB + 8MiB/GiB overhead becomes a kill threshold for a guest that touches all its RAM. Was a 100% fill tried? The test in the description stopped at 925MiB.
There was a problem hiding this comment.
MicroVM guests can't directly touch ram outside, they would be triggering the interior kernel oomkiller as the VMM already configured a memory limit which is below the cost+ limit.
This is a backstop against some kind of leak on the outside of the VMM.
VMM leak aside, if they do a lot of file system reads then we fill page cache with virtiofsd.
…strate#3476) Follow-up to agent-substrate#3429, from @aramase's review: agent-substrate#3429 (comment) When the pod can lend a microVM snapshot write all the memory it needs, the actor's cgroup has no `memory.high` during the write, so it can end with a lot of dirty page cache. Putting `memory.high` back reclaims what it can, but gives up on pages still dirty or under writeback: https://github.com/torvalds/linux/blob/v7.0/mm/memcontrol.c#L4409-L4429 The memory guard resumes as soon as the write returns, before teardown, and its overshoot check counted that page cache. So it could kill the paused VM and report the actor crashed for exceeding its memory limit. The overshoot check now leaves reclaimable page cache out too, and both checks take what's left from one `memory.stat` read. The kernel only flushes `memory.stat` past a threshold of updates, so it can trail `memory.current`, and subtracting one from the other could count fresh page cache as used: https://github.com/torvalds/linux/blob/v7.0/mm/memcontrol.c#L561-L564 AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Benjamin Elder <[email protected]>
…uard's overshoot check (#3513) Cherry-pick of #3476 onto `release-1.0` (`git cherry-pick -x` of the squash commit), it applied cleanly with no changes. The memory guard's overshoot check leaves reclaimable page cache out, so it no longer kills a paused microVM and reports the actor crashed when a snapshot write leaves dirty page cache behind. This fixes a bug in #3429, which is in 1.0. AI use: AI-assisted. Clean pick. - [x] Tests pass - [x] Docs updated Signed-off-by: Benjamin Elder <[email protected]>
A microVM actor's cloud-hypervisor and virtiofsd run in the worker pod with no memory limit of their own.
If they grow past what the actor booked, they take the pod to its limit, and the OOM kill takes out the worker and every actor on it.
#3106 fixed this for gVisor, this PR fixes it for microVM.
Part of #3092.
memory.highat what it books (its memory limit plus the runtime overhead from MicroVM guests get their whole memory limit, and placement books each actor's runtime overhead #3104). The guest can't use more than its memory limit, so anything past that is cloud-hypervisor or virtiofsd.memory.highwithcgroup.kill, using the same rule as A gVisor actor going over its memory limit no longer takes down every actor on its worker #3106, and reports the actor crashed with "killed for exceeding its memory limit". Other actors keep running.memory.highis lifted for the write and restored after, same as Suspending actors no longer OOM-kills tight worker pods #3353.cgroup.MemoryGuardthat both workers use, gVisor's behavior is unchangedTested on GKE with a fresh install:
memory.highof 1036MiB, and survived 3/3 suspend/resume cycles with no killsAI use: AI-assisted, I reviewed and tested it.