Skip to content

atelet, ateapi: report worker runtimes per sandbox class - #3022

Merged
Zoe Zhao (zoez7) merged 4 commits into
agent-substrate:mainfrom
shrutiyam-glitch:worker-sandbox-classes
Oct 21, 2026
Merged

Zoe Zhao (zoez7) merged 4 commits into
agent-substrate:mainfrom
shrutiyam-glitch:worker-sandbox-classes

Conversation

@shrutiyam-glitch

Copy link
Copy Markdown
Collaborator

Fixes #3008 .

WorkerStatus.default_runtime and restorable_runtimes are two flat fields, so they can only ever describe versions of one sandbox class, while WorkerPoolSpec.sandboxClasses is already a list. Replacing a singular default_runtime with a repeated field after GA would be a breaking change, and sharing SandboxRuntime between worker reports and actor checkpoints leaves nowhere to add worker-only runtime fields (such as additional compatible versions or a runtime name) without reaching an actor's stamp.

  • ateapipb & ateletpb: Replace default_runtime and restorable_runtimes on WorkerStatus and RegisterWorkerRequest with repeated WorkerSandboxClass sandbox_classes (keyed by sandbox_class, maxItems=1 today). Each entry carries default_runtime and restorable_runtimes as WorkerSandboxRuntime{version}. Rename SandboxRuntime to ActorSandboxRuntime{sandbox_class, version} on WorkerAssignment, Snapshot, and atelet's RunRequest and RestoreRequest.
  • atelet & ateworker: Write sandbox-runtimes.json with a sandboxClasses list of per-class entries (defaultRuntime and restorableRuntimes as {version: attrs}). ateom selects the entry for its configured SandboxClass and forwards sandbox_classes through WorkerSupport.RegisterWorker.
  • ateapi (internal/hardware, workerservice, scheduling, controlapi): Look up the snapshot's class in status.sandbox_classes, match against default_runtime.version then restorable_runtimes[].version, and stamp ActorSandboxRuntime from the matched entry (or ColdBootRuntime on cold boot and SNAPSHOT_FIDELITY_ROOTFS fallback). RegisterWorker checks each entry's sandbox_class against the Worker's sandbox_class and stores the list as reported.
    This is a stored-data break for existing Worker rows in Postgres.
  • Tests pass
  • Docs updated

Comment thread pkg/proto/ateapipb/ateapi.proto Resolved
@zoez7
Zoe Zhao (zoez7) added this pull request to the merge queue Oct 21, 2026
@github-merge-queue
github-merge-queue Bot removed this pull request from the merge queue due to a conflict with the base branch Oct 21, 2026
The message is the runtime one actor runs under: recorded on its worker
assignment and on the snapshots taken from it, and named on atelet's Run
and Restore. The name now says so, and leaves the plain name free for
what a Worker reports per sandbox class, which carries more than an
actor's stamp does.

Field numbers and the sandbox_runtime field names are unchanged, so
stored assignments and snapshots stay readable.
Replace WorkerStatus.default_runtime and restorable_runtimes with
sandbox_classes, a list of WorkerSandboxClass keyed by sandbox_class. Each
entry carries that class's default_runtime and restorable_runtimes as
WorkerSandboxRuntime, a runtime the worker runs with its compatibility
version. RegisterWorkerRequest and the internal
ateletpb.RegisterWorkerRequest take the same list. ActorSandboxRuntime
stays as the stamp on assignments and snapshots.

Two flat fields can only ever describe versions of one sandbox class, while
WorkerPoolSpec.sandboxClasses is already a list. Turning default_runtime
into a repeated field after GA would be a breaking change; doing it now,
while stored-data breaks are still allowed, is cheap. A worker-side runtime
is its own message so that what a runtime can also restore, and its name,
can be added to it later without re-typing the lists or reaching an actor's
stamp. The lists allow one entry, matching WorkerPoolSpec.

This is a stored-data break: Worker rows are binary protos, so existing
installs need a fresh database. Callers move to the new shape in the two
commits that follow; the tree builds again at the last of them.
…lass

The node file atelet writes for a worker pod lists sandboxClasses, one entry
per class the pool declares, with that class's defaultRuntime and
restorableRuntimes, each a runtime with its version, as the registration
request has them; the flat layout is no longer read. buildFile emits one
entry for the pool's class, and the written-versions bookkeeping and write
log go through the list.

ateom registers the entry for the class it implements, its SandboxClass,
and waits when the file has no such entry or no default for it, as it did
for a missing default. atelet forwards sandbox_classes to the control
plane field for field instead of converting one default and a list of
restorables.
internal/hardware looks up the worker's entry for the snapshot's class. It
matches the snapshot's compat against that entry's default_runtime, then
its restorable_runtimes in order, and builds the ActorSandboxRuntime stamp
from the match. RuntimeFor falls back to ColdBootRuntime, the default of
the worker's class. SandboxClass is that class alone, for the readers that
need nothing else: the scheduler, which skips a worker that has registered
no class yet, the worker metrics, and kubectl ate's worker filter and
CLASS column.

WorkerService.RegisterWorker checks each entry's sandbox_class against the
class the Worker's first registration recorded and stores the list as
reported. fake-workersync reports one entry for the pool's class.

Fixtures build the worker side from the same ActorSandboxRuntime a test
expects on the stamp, so both sides of a match derive from one value.
@zoez7
Zoe Zhao (zoez7) added this pull request to the merge queue Oct 21, 2026
Merged via the queue into agent-substrate:main with commit 3b779a8 Oct 21, 2026
15 checks passed
igooch pushed a commit to igooch/substrate that referenced this pull request Oct 27, 2026
… actor's runtime overhead (agent-substrate#3104)

Today the microVM worker takes a flat 128MiB out of each actor's memory
limit for the VMM, which is too much for small guests and too little for
large ones, and nothing accounts for what an actor costs outside its
sandbox when placing it.

- Workers report what an actor costs on top of its limits on each
runtime (`WorkerSandboxRuntime.actor_overhead`: a fixed amount plus an
amount per GiB of the actor's memory limit), and placement books it with
the actor, so it shows in the worker's allocated resources
- MicroVM guests get their whole memory limit, and the worker reports
128MiB + 8MiB/GiB per actor as overhead (cloud-hypervisor, virtiofsd,
host page tables, and the page cache of snapshot writes). Once a
follow-up caps snapshot writes this drops to 4MiB + 8MiB/GiB
- Both workers keep 64MiB of their memory limit out of the capacity they
report, for the worker process and in-flight snapshot writes

agent-substrate#3091 proposed wrapping `SandboxRuntime`, but agent-substrate#3022 already split the
worker-side runtime (`WorkerSandboxRuntime`) from the stamp on snapshots
and assignments (`ActorSandboxRuntime`), so the overhead is a field on
the former. Same intent, and it stays out of compatibility matching.

Fixes agent-substrate#3091

TODO: For microVM, actors still lose some memory inside the guest limit
to various things, including the kernel and kata-agent. Similarly, in
gVisor they lose memory to the sentry, which is inside the actor's
cgroup. As such, we currently leave gVisor overhead accounting at zero.

AI use: AI-assisted, I reviewed and tested it. Tested with a fresh
install on GKE (gVisor and microVM e2e lanes).

Breaking change: worker pools hold a little less. A worker's memory
capacity is now its memory limit minus 64MiB, and each microVM actor
books its limit plus 128MiB + 8MiB/GiB. A pool sized to exactly fit its
actors (e.g. pod memory limit equal to one actor's limit) needs its
`limits.memory` raised, roughly 64MiB per worker plus the microVM
overhead per actor. MicroVM guests also now see their whole memory
limit, 128MiB more than before.

**These values are subject to tuning in the future**.

- [x] Tests pass
- [x] Docs update

---------

Signed-off-by: Benjamin Elder <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

api: report worker runtimes per sandbox class (WorkerSandboxClass, WorkerSandboxRuntime, ActorSandboxRuntime)

2 participants