Repository navigation
MicroVM guests get their whole memory limit, and placement books each actor's runtime overhead - #3104
Conversation
Benjamin Elder (BenTheElder)
added
area/api
|
Doing some more measurements for the overhead constants on both gVisor and microVM, but I think the mechanics here are good to go along with the API shape. |
Eitan Yarmush (EItanya)
left a comment
Eitan Yarmush (EItanya)
left a comment
There was a problem hiding this comment.
Love the direction looks great overall, just some smaller things
| // memoryReservedBytes is how much of the worker pod's memory limit we keep from | ||
| // actors: the worker process itself, and the page cache of checkpoints in | ||
| // flight. An actor's sentry and gofers are charged to its own cgroup. | ||
| const memoryReservedBytes = 64 << 20 |
There was a problem hiding this comment.
Should this be configurable? We spoke offline/in the issue about reserving capacity for snapshot operations. Totally fine for a follow-up but wondering about the plan there
There was a problem hiding this comment.
I'm not sure yet, I bias towards not when we aren't sure the model is ready yet, we can evolve it more easily if it's not an extension point. I would like to iterate on substrate getting it right.
|
BenTheElder/substrate@1a26fe5...snapshot-write-cap is one of the draft follow-ups, trying to keep the PR manageable. |
7695cb7 to
bbd9c64
Compare
Walter Fender (cheftako)
left a comment
Walter Fender (cheftako)
left a comment
There was a problem hiding this comment.
AI flagged a couple of decent points/questions.
14f42ac to
6b32695
Compare
|
[rebase only] |
| - **gVisor (`ateworker-gvisor`)** — applied to the container OCI spec: `limits.cpuMillicores` sets the cgroup v2 CPU quota (`cpu.max`) and the Sentry vCPU count (`--cpu-num-from-quota`); `limits.memoryBytes` sets the cgroup v2 memory limit (`memory.max`) and bounds the virtual total memory the sandbox reports (so JVM/Go do not over-allocate from host RAM). | ||
| - **Micro-VM (`ateworker-microvm`)** — `limits.cpuMillicores` sets Cloud Hypervisor `BootVcpus` / `MaxVcpus` (rounded up to whole vCPUs); `limits.memoryBytes` sets guest RAM, reserving a small configurable margin (default 256 MiB, `--vmm-mem-reserve-mib`) for the VMM and virtiofsd so the pod cgroup does not OOM. | ||
| 2. **Gate scheduling.** An actor is only placed on a `WorkerPool` whose [worker capacity](#worker-capacity-spectemplateresources) is `>=` these limits. | ||
| - **Micro-VM (`ateworker-microvm`)** — `limits.cpuMillicores` sets Cloud Hypervisor `BootVcpus` / `MaxVcpus` (rounded up to whole vCPUs); `limits.memoryBytes` sets guest RAM, all of it. The smallest micro-VM actor is 128 MiB. What the actor costs outside the guest comes out of the worker's capacity instead (see [Worker Capacity](#worker-capacity-spectemplateresources)). |
There was a problem hiding this comment.
WRT 128Mi min -- where do we enforce that? What happens if I ask for less?
There was a problem hiding this comment.
Only the microVM worker enforces it, at cold boot.
A smaller limit fails the resume with 'actor memory limit NMiB is too small for a micro-VM, the minimum is 128MiB'.
The API doesn't validate it because the template doesn't know which runtime it'll land on. The docs now say where it's enforced
NOTE: This is something I want to change ~immediately, hence the seperate commit, but it needs part of the fix for 2) in #3092, which is going to be a itself a decent sized follow-up.
There was a problem hiding this comment.
Yeah, I don't think we should fail to resume, probably, but it means we need to report a status which is the ACTUAL resources granted. e.g. I ask for 64Mi, I get 128Mi, status is how I learn that.
There was a problem hiding this comment.
Yeah, I don't think we should fail to resume, probably [...]
How would we not?
[...] but it means we need to report a status which is the ACTUAL resources granted. e.g. I ask for 64Mi, I get 128Mi, status is how I learn that.
ACK.
There was a problem hiding this comment.
Yeah, I don't think we should fail to resume, probably [...]
How would we not?
If we are now in a cost-plus model, an actor that asks for 1MiB should be feasible, even if the overhead is wildly out of scale. Failing to resume is about the worst outcome -- we allowed it into the system without errors and then LATER decide "nevermind, it's no good".
We should strive to do better than that.
There was a problem hiding this comment.
Let me think more about this.
ae74880 to
562bf3f
Compare
|
I have draft follow-up branches for:
With those I have a refined uVM overhead down to 4MiB + 8MiB/GiB, but it's particularly unsafe to do that now without at least 1 (versus keeping extra overhead to at least not regress vs main until then), and they're decently large diffs. Should do some more testing for the numbers, but we can also send tiny follow-up PRs to tweak those. |
each runtime a worker reports can carry what an actor costs on top of its limits, a fixed amount plus an amount per GiB of its memory limit, and the scheduler books it with the actor. workers can also keep part of their memory limit out of the capacity they report. nothing sets either yet. Signed-off-by: Benjamin Elder <[email protected]>
instead of holding back 128MiB of every guest, microVM workers report 4MiB plus 8MiB per GiB of an actor's memory limit as its overhead, and both workers keep 64MiB of their memory limit for themselves. Signed-off-by: Benjamin Elder <[email protected]>
the store prices an assignment from the worker it locks, before writing it, so the recorded booking, the admission check and the allocation agree. the price follows the snapshot's runtime even when placement relaxes the match, which is the runtime the assignment stamps, and a booking that overflows fits nowhere. Signed-off-by: Benjamin Elder <[email protected]>
…bound the overhead fields Signed-off-by: Benjamin Elder <[email protected]>
Signed-off-by: Benjamin Elder <[email protected]>
we'll tune this back to 4MiB when the snapshot write cap lands Signed-off-by: Benjamin Elder <[email protected]>
…py the actor overhead per runtime Signed-off-by: Benjamin Elder <[email protected]>
Signed-off-by: Benjamin Elder <[email protected]>
…d scales, explain the book callback Signed-off-by: Benjamin Elder <[email protected]>
…rop the book callback Signed-off-by: Benjamin Elder <[email protected]>
Signed-off-by: Benjamin Elder <[email protected]>
Signed-off-by: Benjamin Elder <[email protected]>
562bf3f to
e56219c
Compare
|
[ammended commits for DCO] |
|
The next two after this are here: BenTheElder:actor-memory-overhead...BenTheElder:snapshot-write-cap BenTheElder/substrate@snapshot-write-cap...BenTheElder:microvm-oom-kill It's too bad stacked PRs don't do what we want. But you can see the idea anyhow. |
Tim Hockin (thockin)
added
prio/P0
Bowei Du (bowei)
added
area/node
area/scheduling
kind/feature
…ate#3353) A suspend writes about as much page cache as the actor has in memory, and it's charged to the worker pod. In a pod sized close to its actors that runs the pod out of memory, and the kernel OOM kills the worker along with every actor on it. Part of agent-substrate#3092. - While a microVM snapshot or gVisor checkpoint is written, its cgroup gets `memory.high`, so the page cache can only grow into what the actor booked and isn't using, plus what the pod has spare to lend. If the pod can lend all of it there's no limit, so pods with room still suspend at full speed. Tight pods slow down to disk speed instead. - A worker writes at most 2 snapshots at once, and a newly placed actor waits for lent memory to be written back before it boots - The rootfs upper and durable-dir tars write out and drop their page cache every 8MiB, so they can't pile up dirty pages either. This waits on the disk, so it gives back some of the speed agent-substrate#3026 got by dropping the fsync. Happy to tune it. - microVM actor overhead goes from 128MiB back to 4MiB + 8MiB/GiB, the 128MiB in agent-substrate#3104 was headroom for these writes. Both workers keep 32MiB plus 16MiB per write slot (64MiB) for themselves. Tested on GKE with a fresh install: - A 1GiB microVM actor filled to 893MiB in a 1132Mi pod survived 4/4 suspend/resume cycles - four 256Mi gVisor actors filled to 225MiB in a 1088Mi pod survived 6/6 concurrent suspend waves. - Without the cap that gVisor case OOM-killed the worker. Neither worker restarted. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Benjamin Elder <[email protected]>
… down its worker (agent-substrate#3429) A microVM actor's cloud-hypervisor and virtiofsd run in the worker pod with no memory limit of their own. If they grow past what the actor booked, they take the pod to its limit, and the OOM kill takes out the worker and every actor on it. agent-substrate#3106 fixed this for gVisor, this PR fixes it for microVM. Part of agent-substrate#3092. - each actor's cgroup gets `memory.high` at what it books (its memory limit plus the runtime overhead from agent-substrate#3104). The guest can't use more than its memory limit, so anything past that is cloud-hypervisor or virtiofsd. - the worker kills a cgroup stuck at its `memory.high` with `cgroup.kill`, using the same rule as agent-substrate#3106, and reports the actor crashed with "killed for exceeding its memory limit". Other actors keep running. - the watch pauses during snapshot writes. When the pod can lend the whole write, the cgroup's `memory.high` is lifted for the write and restored after, same as agent-substrate#3353. - without PSI the worker sets no limit and warns at startup, like gVisor - the gVisor watch from agent-substrate#3106 moves into a shared `cgroup.MemoryGuard` that both workers use, gVisor's behavior is unchanged Tested on GKE with a fresh install: - a 1GiB actor filled to 925MiB peaked at 1007MiB against a `memory.high` of 1036MiB, and survived 3/3 suspend/resume cycles with no kills - a 400MiB process added to one 512Mi actor's cgroup got that actor killed and reported crashed. Its neighbour kept answering and the worker didn't restart. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Benjamin Elder <[email protected]>
Today the microVM worker takes a flat 128MiB out of each actor's memory limit for the VMM, which is too much for small guests and too little for large ones, and nothing accounts for what an actor costs outside its sandbox when placing it.
WorkerSandboxRuntime.actor_overhead: a fixed amount plus an amount per GiB of the actor's memory limit), and placement books it with the actor, so it shows in the worker's allocated resources#3091 proposed wrapping
SandboxRuntime, but #3022 already split the worker-side runtime (WorkerSandboxRuntime) from the stamp on snapshots and assignments (ActorSandboxRuntime), so the overhead is a field on the former. Same intent, and it stays out of compatibility matching.Fixes #3091
TODO: For microVM, actors still lose some memory inside the guest limit to various things, including the kernel and kata-agent. Similarly, in gVisor they lose memory to the sentry, which is inside the actor's cgroup. As such, we currently leave gVisor overhead accounting at zero.
AI use: AI-assisted, I reviewed and tested it. Tested with a fresh install on GKE (gVisor and microVM e2e lanes).
Breaking change: worker pools hold a little less. A worker's memory capacity is now its memory limit minus 64MiB, and each microVM actor books its limit plus 128MiB + 8MiB/GiB. A pool sized to exactly fit its actors (e.g. pod memory limit equal to one actor's limit) needs its
limits.memoryraised, roughly 64MiB per worker plus the microVM overhead per actor. MicroVM guests also now see their whole memory limit, 128MiB more than before.These values are subject to tuning in the future.