Repository navigation
atelet, ateapi: report worker runtimes per sandbox class - #3022
Merged
Zoe Zhao (zoez7) merged 4 commits intoOct 21, 2026
Merged
Zoe Zhao (zoez7) merged 4 commits into
Zoe Zhao (zoez7) merged 4 commits into
Conversation
Shrutiya Mohan (shrutiyam-glitch)
force-pushed
the
worker-sandbox-classes
branch
from
October 20, 2026 23:32
34ad693 to
87ed664
Compare
Shrutiya Mohan (shrutiyam-glitch)
marked this pull request as ready for review
October 20, 2026 23:33
Zoe Zhao (zoez7)
previously approved these changes
Oct 20, 2026
Shrutiya Mohan (shrutiyam-glitch)
force-pushed
the
worker-sandbox-classes
branch
from
October 21, 2026 00:06
87ed664 to
d5a11a4
Compare
Zoe Zhao (zoez7)
previously approved these changes
Oct 21, 2026
Zoe Zhao (zoez7)
enabled auto-merge
October 21, 2026 00:27
github-merge-queue
Bot
removed this pull request from the merge queue due to a conflict with the base branch
Oct 21, 2026
Shrutiya Mohan (shrutiyam-glitch)
force-pushed
the
worker-sandbox-classes
branch
from
October 21, 2026 01:26
070e357 to
e6b2998
Compare
The message is the runtime one actor runs under: recorded on its worker assignment and on the snapshots taken from it, and named on atelet's Run and Restore. The name now says so, and leaves the plain name free for what a Worker reports per sandbox class, which carries more than an actor's stamp does. Field numbers and the sandbox_runtime field names are unchanged, so stored assignments and snapshots stay readable.
Replace WorkerStatus.default_runtime and restorable_runtimes with sandbox_classes, a list of WorkerSandboxClass keyed by sandbox_class. Each entry carries that class's default_runtime and restorable_runtimes as WorkerSandboxRuntime, a runtime the worker runs with its compatibility version. RegisterWorkerRequest and the internal ateletpb.RegisterWorkerRequest take the same list. ActorSandboxRuntime stays as the stamp on assignments and snapshots. Two flat fields can only ever describe versions of one sandbox class, while WorkerPoolSpec.sandboxClasses is already a list. Turning default_runtime into a repeated field after GA would be a breaking change; doing it now, while stored-data breaks are still allowed, is cheap. A worker-side runtime is its own message so that what a runtime can also restore, and its name, can be added to it later without re-typing the lists or reaching an actor's stamp. The lists allow one entry, matching WorkerPoolSpec. This is a stored-data break: Worker rows are binary protos, so existing installs need a fresh database. Callers move to the new shape in the two commits that follow; the tree builds again at the last of them.
…lass The node file atelet writes for a worker pod lists sandboxClasses, one entry per class the pool declares, with that class's defaultRuntime and restorableRuntimes, each a runtime with its version, as the registration request has them; the flat layout is no longer read. buildFile emits one entry for the pool's class, and the written-versions bookkeeping and write log go through the list. ateom registers the entry for the class it implements, its SandboxClass, and waits when the file has no such entry or no default for it, as it did for a missing default. atelet forwards sandbox_classes to the control plane field for field instead of converting one default and a list of restorables.
internal/hardware looks up the worker's entry for the snapshot's class. It matches the snapshot's compat against that entry's default_runtime, then its restorable_runtimes in order, and builds the ActorSandboxRuntime stamp from the match. RuntimeFor falls back to ColdBootRuntime, the default of the worker's class. SandboxClass is that class alone, for the readers that need nothing else: the scheduler, which skips a worker that has registered no class yet, the worker metrics, and kubectl ate's worker filter and CLASS column. WorkerService.RegisterWorker checks each entry's sandbox_class against the class the Worker's first registration recorded and stores the list as reported. fake-workersync reports one entry for the pool's class. Fixtures build the worker side from the same ActorSandboxRuntime a test expects on the stamp, so both sides of a match derive from one value.
Shrutiya Mohan (shrutiyam-glitch)
force-pushed
the
worker-sandbox-classes
branch
from
October 21, 2026 02:02
e6b2998 to
b2e8e59
Compare
Zoe Zhao (zoez7)
enabled auto-merge
October 21, 2026 04:28
Zoe Zhao (zoez7)
approved these changes
Oct 21, 2026
2 tasks done
igooch
pushed a commit
to igooch/substrate
that referenced
this pull request
Oct 27, 2026
… actor's runtime overhead (agent-substrate#3104) Today the microVM worker takes a flat 128MiB out of each actor's memory limit for the VMM, which is too much for small guests and too little for large ones, and nothing accounts for what an actor costs outside its sandbox when placing it. - Workers report what an actor costs on top of its limits on each runtime (`WorkerSandboxRuntime.actor_overhead`: a fixed amount plus an amount per GiB of the actor's memory limit), and placement books it with the actor, so it shows in the worker's allocated resources - MicroVM guests get their whole memory limit, and the worker reports 128MiB + 8MiB/GiB per actor as overhead (cloud-hypervisor, virtiofsd, host page tables, and the page cache of snapshot writes). Once a follow-up caps snapshot writes this drops to 4MiB + 8MiB/GiB - Both workers keep 64MiB of their memory limit out of the capacity they report, for the worker process and in-flight snapshot writes agent-substrate#3091 proposed wrapping `SandboxRuntime`, but agent-substrate#3022 already split the worker-side runtime (`WorkerSandboxRuntime`) from the stamp on snapshots and assignments (`ActorSandboxRuntime`), so the overhead is a field on the former. Same intent, and it stays out of compatibility matching. Fixes agent-substrate#3091 TODO: For microVM, actors still lose some memory inside the guest limit to various things, including the kernel and kata-agent. Similarly, in gVisor they lose memory to the sentry, which is inside the actor's cgroup. As such, we currently leave gVisor overhead accounting at zero. AI use: AI-assisted, I reviewed and tested it. Tested with a fresh install on GKE (gVisor and microVM e2e lanes). Breaking change: worker pools hold a little less. A worker's memory capacity is now its memory limit minus 64MiB, and each microVM actor books its limit plus 128MiB + 8MiB/GiB. A pool sized to exactly fit its actors (e.g. pod memory limit equal to one actor's limit) needs its `limits.memory` raised, roughly 64MiB per worker plus the microVM overhead per actor. MicroVM guests also now see their whole memory limit, 128MiB more than before. **These values are subject to tuning in the future**. - [x] Tests pass - [x] Docs update --------- Signed-off-by: Benjamin Elder <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #3008 .
WorkerStatus.default_runtimeandrestorable_runtimesare two flat fields, so they can only ever describe versions of one sandbox class, whileWorkerPoolSpec.sandboxClassesis already a list. Replacing a singulardefault_runtimewith a repeated field after GA would be a breaking change, and sharingSandboxRuntimebetween worker reports and actor checkpoints leaves nowhere to add worker-only runtime fields (such as additional compatible versions or a runtime name) without reaching an actor's stamp.ateapipb&ateletpb: Replacedefault_runtimeandrestorable_runtimesonWorkerStatusandRegisterWorkerRequestwithrepeated WorkerSandboxClass sandbox_classes(keyed bysandbox_class,maxItems=1today). Each entry carriesdefault_runtimeandrestorable_runtimesasWorkerSandboxRuntime{version}. RenameSandboxRuntimetoActorSandboxRuntime{sandbox_class, version}onWorkerAssignment,Snapshot, andatelet'sRunRequestandRestoreRequest.atelet&ateworker: Writesandbox-runtimes.jsonwith asandboxClasseslist of per-class entries (defaultRuntimeandrestorableRuntimesas{version: attrs}).ateomselects the entry for its configuredSandboxClassand forwardssandbox_classesthroughWorkerSupport.RegisterWorker.ateapi(internal/hardware,workerservice,scheduling,controlapi): Look up the snapshot's class instatus.sandbox_classes, match againstdefault_runtime.versionthenrestorable_runtimes[].version, and stampActorSandboxRuntimefrom the matched entry (orColdBootRuntimeon cold boot andSNAPSHOT_FIDELITY_ROOTFSfallback).RegisterWorkerchecks each entry'ssandbox_classagainst theWorker'ssandbox_classand stores the list as reported.This is a stored-data break for existing
Workerrows in Postgres.