Skip to content

ROOTFS snapshot fidelity: micro-VM capture and restore, API opened for micro-VM templates - #2448

Merged
Zoe Zhao (zoez7) merged 5 commits into
agent-substrate:mainfrom
dberkov:rootfs-fidelity
Oct 11, 2026
Merged

Zoe Zhao (zoez7) merged 5 commits into
agent-substrate:mainfrom
dberkov:rootfs-fidelity

Conversation

@dberkov

Copy link
Copy Markdown
Collaborator

Step III of #2102, tracked in #2447. Step I (#2309) removed on_pause; step II (#2364) introduced the SnapshotFidelity ladder with ROOTFS defined but rejected everywhere.

ROOTFS becomes real on micro-VM. A ROOTFS snapshot holds the durable volumes plus each container's root-filesystem changes, without guest memory: the actor cold-boots on resume but keeps what it wrote to its rootfs. gVisor keeps rootfs changes inside its memory checkpoint and cannot serve it yet; it now declares that explicitly and a gVisor template that asks for ROOTFS is rejected at admission.

Four commits, meant to be read in order. Each builds and passes tests on its own, and ROOTFS stays unreachable from the public API until the third.

1. atelet: forward ROOTFS and stop narrowing paused uploads

Atelet accepts every level of the ladder and forwards it; which levels a runtime serves is the runtime's knowledge. The paused-upload path still converted a MEMORY pause into a VOLUMES upload, a leftover from the days of separate pause and commit scopes. Pause and suspend now capture one fidelity, a paused actor's template cannot change, and golden actors are never paused, so captured and requested fidelity are always equal; the conversion is replaced by an equality check that refuses any mismatch as drift.

2. ateom-microvm: capture and restore ROOTFS

Every piece already existed: MEMORY checkpoints tar the host-side rootfs upper per container alongside the Cloud Hypervisor snapshot, and MEMORY restores untar those uppers before assembling the overlays. ROOTFS is the same checkpoint without the VM snapshot, and the upper restore followed by the VOLUMES cold-boot path with a new keepRootfsUpper boot parameter so the cold boot does not wipe the restored upper (safe across its retry). Only a MEMORY restore resumes the guest's CPU counters; ROOTFS counts from zero like VOLUMES.

resources.ValidateSnapshotFidelity now takes the runtime's supported set: micro-VM passes all three, gVisor passes VOLUMES and MEMORY and rejects ROOTFS before taking the actor lock, naming the levels it serves.

3. ateapi: allow ROOTFS on micro-VM templates

The field-level rejection moves to the template-level cross-field hook: ROOTFS requires sandbox_config.sandbox_class: SANDBOX_CLASS_MICROVM. An actor can only be repointed to a template with an identical sandbox config, so a ROOTFS template never ends up under a gVisor actor. The control plane needs no logic change: a ROOTFS snapshot restores at ROOTFS, and a repointed actor drops to VOLUMES as it does for MEMORY, since a rootfs upper is only valid over the image it was written on.

4. e2e: prove it on the counter demo

The counter app gains a third counter in a file on its root filesystem, printed in the response, so a ROOTFS snapshot is distinguishable from VOLUMES. All lifecycle rows state their rootfs expectation (MEMORY and ROOTFS keep it, VOLUMES resets it). New micro-VM-only ROOTFS rows cover pause, suspend, suspend from PAUSED, and the template repoint and revert. On the gVisor CI lane the ROOTFS rows skip; the new rootfs assertions on MEMORY and VOLUMES rows run there as a side check.

Behavior changes

  • API: SNAPSHOT_FIDELITY_ROOTFS is accepted on micro-VM templates and rejected on gVisor ones with a message naming the requirement. No field or wire changes; the preferred_fidelity field-level custom validation tag is removed.
  • Atelet: a paused-upload whose desired fidelity differs from the captured one is now FailedPrecondition; previously a MEMORY capture could be downgraded to VOLUMES. Unreachable in practice, as explained in commit 1.
  • Unchanged: MEMORY and VOLUMES capture and restore, the golden-actor MEMORY override, metric labels (rootfs was already a member).
  • No backward compatibility is kept, per the pre-v1.0 policy.

Not in this PR

required_fidelity, ResumeActorResponse.resumed_fidelity, the resume fallback rule, gVisor ROOTFS (pending the gVisor team's separable rootfs capture), and restoring a MEMORY snapshot as a ROOTFS cold boot (the artifacts already allow it; atelet would need the rootfs file subset in the manifest to skip the memory image).

Verification

  • Unit tests for atelet, ateom-microvm, ateom-gvisor, resources, ateattr, ateomphaselog, ateapi (validation, controlapi, functional), kubectl-ate, and apitool pass; the ateom tests are linux-tagged and were type-checked for linux locally.

  • golangci-lint, kube-api-linter, python-protos, boilerplate pass; codegen shows no drift.

  • The micro-VM counter demo was run manually on a GKE cluster with a nested-virtualization node pool and behaved as expected. The ROOTFS e2e rows need a micro-VM lane; the gVisor lane runs the rest.

  • Breaking change

  • Not backward compatible

🤖 Generated with Claude Code

Atelet refused ROOTFS in its checkpoint, restore, and paused-upload
validation while the micro-VM runtime gains the ability to capture it.
Which fidelities a sandbox runtime can serve is the runtime's knowledge,
not atelet's: atelet now accepts every level of the ladder and forwards it,
and each ateom rejects what it cannot capture.

The paused-upload path still carried a conversion from the days when a
template had separate pause and commit scopes: a MEMORY pause could be
uploaded as a VOLUMES snapshot by dropping to the durable-volume tars.
Pause and suspend now capture the template's one preferred fidelity, a
paused actor's template cannot change, and golden actors are never paused,
so the fidelity a pause recorded always equals the one the suspend asks for.
Replace the conversion with that equality check and refuse any mismatch as
drift, rather than guessing. The manifest still records the durable-volume
subset ateom reported; nothing in atelet acts on it anymore.
A ROOTFS snapshot holds the durable volumes plus each container's root
filesystem changes, without guest memory: the actor cold-boots on restore
but keeps what it wrote to its rootfs. The micro-VM runtime already had
every piece. A MEMORY checkpoint tars the host-side rootfs upper of each
container alongside the Cloud Hypervisor snapshot, and a MEMORY restore
untars those uppers before assembling the overlays. ROOTFS is the same
checkpoint without the VM snapshot, and the same upper restore followed by
the VOLUMES cold-boot path instead of a VMM relaunch.

Checkpoint takes the upper tars for ROOTFS as well as MEMORY and needs no
durable volume to have something to capture. Restore gains a ROOTFS arm
that re-materializes the uppers and cold-boots on top of them; the cold
boot learns to keep a pre-populated upper instead of wiping it, which holds
across its retry too since a guest that never reached its agent wrote
nothing. Only a MEMORY restore resumes the guest's CPU counters, so only it
counts usage from the first reading; the lower fidelities count from zero.

Which fidelities a runtime serves is now the runtime's declaration: the
shared request validator takes the supported set, the micro-VM runtime
passes all three, and gVisor passes VOLUMES and MEMORY because its
checkpoint image holds memory and rootfs changes together. gVisor rejects a
ROOTFS checkpoint or restore before taking the actor lock, naming the
levels it does serve.
The micro-VM runtime now captures and restores ROOTFS snapshots, so the
public API stops rejecting the level outright. What decides whether a
template may ask for it is its sandbox class: gVisor keeps rootfs changes
inside its memory checkpoint and cannot serve ROOTFS, while micro-VM can.
The check moves from a field-level rule on preferred_fidelity to the
template-level hook that already holds the other cross-field rules, and
rejects ROOTFS unless sandbox_config.sandbox_class is SANDBOX_CLASS_MICROVM.
An actor can only be repointed to a template with an identical sandbox
config, so a ROOTFS template never ends up under a gVisor actor.

The control plane needs no change: a ROOTFS snapshot restores at ROOTFS,
and a repointed actor drops to VOLUMES as it does for MEMORY, since a rootfs
upper is only valid over the image it was written on. Tests pin both, along
with a pause recording ROOTFS. Docs and the micro-VM counter demo describe
the level; the demo keeps MEMORY because its in-RAM counter is the point.
The lifecycle tests could not tell a ROOTFS snapshot from a VOLUMES one:
the counter app kept one counter in memory and one in a durable volume,
and both fidelities lose the first and keep the second. Give the app a
third counter in a file on its root filesystem and print it in the
response. MEMORY and ROOTFS snapshots carry it across pause and suspend,
VOLUMES resets it, so every row now states what it expects of rootfs
writes and the existing MEMORY and VOLUMES rows gain an assertion that
those runtimes handle them as documented.

Add micro-VM-only ROOTFS rows to the durable-dir lifecycle tests, plain
and suspended from PAUSED: the memory counter restarts, the file and
rootfs counters continue, and the suspend records a ROOTFS snapshot. The
template-repoint test gains a ROOTFS row too, pinning that the repoint
drops the restore to VOLUMES and resets the rootfs counter, that a pause
under the new template restores at ROOTFS again, and that the revert
cold-boots. gVisor cannot serve ROOTFS yet, so the new rows skip off
micro-VM clusters.
Comment thread cmd/ateom-microvm/restore.go Outdated
// container's rootfs upper: put those back before the overlays are
// assembled, then cold-boot on top of them instead of a pristine upper.
tUpper := time.Now()
if err := untarRootfsUpper(rootfsUpperDir(p.actorDirs), restoreDir, containerNames(p.containers)); err != nil {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 should-fix 🟡 – untarRootfsUpper runs os.RemoveAll on the upper dir before cleanupSandboxState has dropped any leftover overlay mounts, which only happens later inside coldBootActor. teardownActor documents that the overlays must be unmounted before the upperdir is removed. The MEMORY and VOLUMES paths both clean up first. Consider calling s.cleanupSandboxState(ctx, p.actorUID) before the untar, as restoreMemoryFidelity does.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested fix: clean up once in the ROOTFS arm before the untar, matching the MEMORY path.

case ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS:
	s.cleanupSandboxState(ctx, p.actorUID)
	if err := untarRootfsUpper(...); err != nil { ... }

@dberkov Dmitry Berkovich (dberkov) Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in a84e095, by moving the upper staging into coldBootActor rather than calling cleanup from the ROOTFS arm. The cold boot now runs cleanupSandboxState first (it moved ahead of buildActorContainers, which only touches the atelet-owned bundle dirs, not the sandbox dirs cleanup clears), and only then stages the upper, so the overlay mounts are gone before untarRootfsUpper removes the dir, matching restoreMemoryFidelity and teardownActor. A retry re-cleans and re-stages from the same snapshot dir, which also removes the "keep what is there" reasoning.

Comment thread cmd/ateom-microvm/run.go Outdated
// A ROOTFS restore has already re-materialized the upper and boots on it;
// that holds across a retry too, since a guest that never reached its agent
// wrote nothing and cleanupSandboxState only drops the overlay mounts.
if !p.keepRootfsUpper {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 should-fix 🟡 – The new ROOTFS behavior in ateom-microvm has no unit test: the keepRootfsUpper skip here, the ROOTFS restore arm, and shipsRootfs in checkpoint. Only the micro-VM e2e rows cover it. Could the upper-dir decision and the per-fidelity file set be pulled into small helpers with table-driven tests, so a regression shows up without a KVM lane?

@dberkov Dmitry Berkovich (dberkov) Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done in a84e095. The ladder is now two predicates in fidelity.go, includesRootfs and includesMemory, used by the checkpoint capture set and the restore resume decision, with TestFidelityLadder pinning all four values. The upper-dir decision is stageRootfsUpper(actorDirs, snapshotDir, containers) in rootfsupper.go: empty dir means pristine, otherwise re-materialize from the tars. TestStageRootfsUpper covers both branches on real directories, including that a stale previous activation is wiped either way. The keepRootfsUpper flag is gone; actorBootParams carries rootfsUpperSnapshotDir instead.

Comment thread cmd/ateom-microvm/restore.go Outdated
// A ROOTFS snapshot holds no guest state either, but it carries each
// container's rootfs upper: put those back before the overlays are
// assembled, then cold-boot on top of them instead of a pristine upper.
tUpper := time.Now()

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 question 🟢 – This untar runs serially before the cold boot. restoreMemoryFidelity runs the same untar in a goroutine overlapped with bundle preparation. For an actor with a large upper, the ROOTFS resume pays the full untar time. Is overlapping it worth doing here too?

@dberkov Dmitry Berkovich (dberkov) Oct 11, 2026 •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, and it is now overlapped. In a84e095 the staging runs as a background goroutine inside coldBootActor, started right after cleanupSandboxState and joined just before stageMergedRootfs, so it hides behind buildActorContainers and the resource-envelope check exactly as the MEMORY path hides its untar behind bundle preparation. Same drain-on-error discipline as there, so an early return never leaves a writer racing a retry. The separate rootfs_upper phase observation went away with the serial untar; the restore record now carries total like VOLUMES.

Comment thread cmd/ateom-microvm/checkpoint.go Outdated
// - Rootfs upper tars (ROOTFS and MEMORY): host-backed like the durable
// volumes — the memory snapshot does not carry rootfs writes. Under
// VOLUMES the workload cold-starts on restore, discarding rootfs state.
shipsRootfs := scope == ateompb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS ||

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 nit 🟢 – "This level includes rootfs" is spelled out separately here, in the switch above, and in resumesGuest in restore. A small helper such as includesRootfs(f) would keep the ladder logic in one place.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done: includesRootfs and includesMemory in fidelity.go are the single place the ladder is spelled out; checkpoint uses the first for its capture set and restore uses the second for resumesGuest. The precondition switch in checkpoint keeps its explicit cases since it is about what each level needs present, not what it includes.

Comment on lines +71 to +72
if value.GetSnapshotConfig().GetPreferredFidelity() == ateapipb.SnapshotFidelity_SNAPSHOT_FIDELITY_ROOTFS &&
value.GetSandboxConfig().GetSandboxClass() != ateapipb.SandboxClass_SANDBOX_CLASS_MICROVM {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Self-note: we won't be able to check this once we move sandbox class to the workerpool.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed. Once the sandbox class lives on the WorkerPool the template no longer knows it at admission, so this check has to move to where the class is first known: scheduling or the resume workflow, rejecting a ROOTFS actor that lands on a gVisor pool. The runtime-side guard in ateom-gvisor stays as the backstop either way, so nothing can produce a ROOTFS checkpoint there in the meantime.

The ROOTFS restore untarred the rootfs uppers before handing off to the
cold boot, which only later cleared stale sandbox state. A previous
activation's overlay mounts could therefore still sit on the upper dir
when the untar removed it, the ordering teardown documents as unsafe. The
untar also ran serially, so a large upper added its full extraction time
to the resume, where the MEMORY path overlaps the same work with bundle
preparation.

Make the cold boot own the upper dir. It clears sandbox state first, moved
ahead of bundle preparation, which touches only the atelet-owned bundle
dirs, then stages the upper in the background and joins it right before
the overlay mounts need it: pristine for a plain boot, re-materialized from
the snapshot's tars when the boot parameters name a snapshot dir. A retry
stages again from that dir, so nothing depends on what an earlier attempt
left behind, and the ROOTFS restore arm shrinks to naming the dir.

Spell the ladder out once: includesRootfs and includesMemory decide the
checkpoint's capture set and whether a restore resumes the guest, and
stageRootfsUpper is the one place a cold boot decides between a pristine
and a restored upper. Both have table tests, so a regression in what each
fidelity carries shows up without a KVM lane.
@zoez7
Zoe Zhao (zoez7) added this pull request to the merge queue Oct 11, 2026
Merged via the queue into agent-substrate:main with commit b73ee7f Oct 11, 2026
14 checks passed
Eitan Yarmush (EItanya) added a commit to kagent-dev/substrate that referenced this pull request Oct 13, 2026
* cmd/atenet: Remove the router's "all" mode (agent-substrate#2441)

All of our in-tree deployments explicitly deploy the router in either
ingress or egress mode.

Attempting to one run router instance that handles both ingress and
egress seems pretty confusing, so we should get rid of the flag that
lets us do so. In the future, we might want to split the ingress router
and the egress router into separate atenet subcommands (or even separate
binaries).

* test: Fix macos unit test failures (agent-substrate#2429)

# Description

`make test` failed on macOS for two host-specific reasons. The DNS relay
referenced Linux-only network namespace APIs from an otherwise portable
file,
and the object storage tests inherited exported `AWS_PROFILE` and
`AWS_DEFAULT_PROFILE` values from the developer's shell.

Move only the namespace-serving method behind a Linux build tag so
portable
DNS tests still run on macOS. Clear both AWS profile selectors in the
object
storage test so local AWS configuration cannot affect it.

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

---------

Signed-off-by: timflannagan <[email protected]>

* Update x/net to v0.60.0 to fix govulncheck failures (agent-substrate#2401)

Both modules in the post-merge govulncheck matrix resolve
golang.org/x/net v0.58.0 and fail on five newly published HTTP/2
vulnerabilities. Update x/net to v0.60.0 in the root module and the GCP
Secret Manager plugin, include its required transitive dependency
updates, and regenerate vendor.

Go also updates the required x/crypto, x/sync, x/sys, x/term, and x/text
versions. All first-party changes are in the two go.mod/go.sum pairs;
the other changes are generated vendor updates.

Affected advisories: GO-2026-6603, GO-2026-6610, GO-2026-6611,
GO-2026-6612, and [GO-2026-6617](https://pkg.go.dev/vuln/GO-2026-6617).

The [latest failing main
run](https://github.com/agent-substrate/substrate/actions/runs/37911293778)
already selects Go 1.27.2; its remaining failures are all in x/net. The
earlier standard-library findings were resolved by the workflow
selecting the patched stable toolchain. This change preserves the
current Go version requirement.

Validation after merging main `59072f7b`:

- Linux/amd64 with Go 1.27.1: `go test -race -count=1 ./...` passed with
PostgreSQL/envtest integration enabled (`CI=true`).
- All checks in `bash hack/verify-all.sh` passed, including
generated-code and vendor consistency, and the checkout remained clean.
- GCP Secret Manager plugin race tests passed.
- govulncheck v1.8.0 with Go 1.27.2: both root and plugin modules passed
with exit 0, zero vulnerable imported packages, and zero reachable
findings. Both report GO-2026-5932 only at module level in unimported
x/crypto code.

The merge retains main's direct x/net dependency classification at
v0.60.0. Canonical v0.60.0 DNS parser content was restored in vendor
after the generation check found that the automatic merge had retained
three older statements.

Verification ran as an unprivileged user. Root-gated mount tests and
real cluster E2E were not run for this head.


AI assistance was used to resolve the merge conflicts and run
verification.

* ROOTFS snapshot fidelity: micro-VM capture and restore, API opened for micro-VM templates (agent-substrate#2448)

Step III of agent-substrate#2102, tracked in agent-substrate#2447. Step I (agent-substrate#2309) removed `on_pause`;
step II (agent-substrate#2364) introduced the `SnapshotFidelity` ladder with ROOTFS
defined but rejected everywhere.

**ROOTFS becomes real on micro-VM.** A ROOTFS snapshot holds the durable
volumes plus each container's root-filesystem changes, without guest
memory: the actor cold-boots on resume but keeps what it wrote to its
rootfs. gVisor keeps rootfs changes inside its memory checkpoint and
cannot serve it yet; it now declares that explicitly and a gVisor
template that asks for ROOTFS is rejected at admission.

Four commits, meant to be read in order. Each builds and passes tests on
its own, and ROOTFS stays unreachable from the public API until the
third.

## 1. atelet: forward ROOTFS and stop narrowing paused uploads

Atelet accepts every level of the ladder and forwards it; which levels a
runtime serves is the runtime's knowledge. The paused-upload path still
converted a MEMORY pause into a VOLUMES upload, a leftover from the days
of separate pause and commit scopes. Pause and suspend now capture one
fidelity, a paused actor's template cannot change, and golden actors are
never paused, so captured and requested fidelity are always equal; the
conversion is replaced by an equality check that refuses any mismatch as
drift.

## 2. ateom-microvm: capture and restore ROOTFS

Every piece already existed: MEMORY checkpoints tar the host-side rootfs
upper per container alongside the Cloud Hypervisor snapshot, and MEMORY
restores untar those uppers before assembling the overlays. ROOTFS is
the same checkpoint without the VM snapshot, and the upper restore
followed by the VOLUMES cold-boot path with a new `keepRootfsUpper` boot
parameter so the cold boot does not wipe the restored upper (safe across
its retry). Only a MEMORY restore resumes the guest's CPU counters;
ROOTFS counts from zero like VOLUMES.

`resources.ValidateSnapshotFidelity` now takes the runtime's supported
set: micro-VM passes all three, gVisor passes VOLUMES and MEMORY and
rejects ROOTFS before taking the actor lock, naming the levels it
serves.

## 3. ateapi: allow ROOTFS on micro-VM templates

The field-level rejection moves to the template-level cross-field hook:
ROOTFS requires `sandbox_config.sandbox_class: SANDBOX_CLASS_MICROVM`.
An actor can only be repointed to a template with an identical sandbox
config, so a ROOTFS template never ends up under a gVisor actor. The
control plane needs no logic change: a ROOTFS snapshot restores at
ROOTFS, and a repointed actor drops to VOLUMES as it does for MEMORY,
since a rootfs upper is only valid over the image it was written on.

## 4. e2e: prove it on the counter demo

The counter app gains a third counter in a file on its root filesystem,
printed in the response, so a ROOTFS snapshot is distinguishable from
VOLUMES. All lifecycle rows state their rootfs expectation (MEMORY and
ROOTFS keep it, VOLUMES resets it). New micro-VM-only ROOTFS rows cover
pause, suspend, suspend from PAUSED, and the template repoint and
revert. On the gVisor CI lane the ROOTFS rows skip; the new rootfs
assertions on MEMORY and VOLUMES rows run there as a side check.

## Behavior changes

- **API**: `SNAPSHOT_FIDELITY_ROOTFS` is accepted on micro-VM templates
and rejected on gVisor ones with a message naming the requirement. No
field or wire changes; the `preferred_fidelity` field-level custom
validation tag is removed.
- **Atelet**: a paused-upload whose desired fidelity differs from the
captured one is now `FailedPrecondition`; previously a MEMORY capture
could be downgraded to VOLUMES. Unreachable in practice, as explained in
commit 1.
- Unchanged: MEMORY and VOLUMES capture and restore, the golden-actor
MEMORY override, metric labels (`rootfs` was already a member).
- No backward compatibility is kept, per the pre-v1.0 policy.

## Not in this PR

`required_fidelity`, `ResumeActorResponse.resumed_fidelity`, the resume
fallback rule, gVisor ROOTFS (pending the gVisor team's separable rootfs
capture), and restoring a MEMORY snapshot as a ROOTFS cold boot (the
artifacts already allow it; atelet would need the rootfs file subset in
the manifest to skip the memory image).

## Verification

- Unit tests for atelet, ateom-microvm, ateom-gvisor, resources,
ateattr, ateomphaselog, ateapi (validation, controlapi, functional),
kubectl-ate, and apitool pass; the ateom tests are linux-tagged and were
type-checked for linux locally.
- golangci-lint, kube-api-linter, python-protos, boilerplate pass;
codegen shows no drift.
- The micro-VM counter demo was run manually on a GKE cluster with a
nested-virtualization node pool and behaved as expected. The ROOTFS e2e
rows need a micro-VM lane; the gVisor lane runs the rest.

- [x] Breaking change
- [x] Not backward compatible

🤖 Generated with [Claude Code](https://claude.com/claude-code)

* fix(kubectl-ate): validate atespace flags before connecting (agent-substrate#2464)

`get actors` and `get actor-template` now reject missing or conflicting
atespace flags before loading kubeconfig or connecting to the API.
Previously, a kubeconfig or connection error could hide the argument
error.

The tests use a missing kubeconfig to check that invalid flags are
rejected first and valid queries still reach client initialization.

Refs agent-substrate#2463

- [x] Tests pass: `go test -race ./cmd/kubectl-ate/...`
- [x] Full Go suite passes: `env -u NO_COLOR go test -race -p 1 ./...`
- [x] Lint passes: `hack/verify/golangci-lint.sh`

The first full-suite run failed in the CPU timing test and in
color-output tests with `NO_COLOR=1`. The serial run above passed.

The Python license and proto checks could not finish because `pypi.org`
DNS lookup failed. The remaining repository verifiers passed; metrics
and shellcheck used local binaries at the versions pinned by the
scripts.

* Allow ActorTemplates to disable golden snapshot creation (agent-substrate#2328)

ActorTemplates can opt out of automated golden snapshot creation with
`snapshotConfig.goldenSnapshotConfig.mode:
GOLDEN_SNAPSHOT_MODE_DISABLED`, avoiding unused golden actors and
snapshots. Omitting the configuration or leaving the mode `UNSPECIFIED`
defaults to `ENABLED`. The mode is an enum with declarative validation,
following the existing snapshot fidelity pattern.

Disabled templates are skipped during queue resync and reconciliation.
Their regular actors can cold boot and suspend/resume using their own
snapshots. `kubectl ate get actor-templates` displays `DISABLED` in the
`GOLDEN TAG` column. The API guide, field table, glossary, and CLI
documentation describe the opt-out.

The functional regression starts a real `ActorTemplateReconciler`. An
enabled control template proves it is making progress while the disabled
template's golden actor and tag both remain absent. The test also
verifies the disabled template's actor cold-boots, creates its own
durable snapshot on suspend, and restores that snapshot on resume. The
disabled reconciliation case is part of the existing table-driven unit
test.

Implements the golden snapshot opt-out feature discussed in agent-substrate#2068.

AI assistance: Codex helped implement and test these changes.

### Validation

- macOS arm64: race and coverage tests passed for defaulting, API
validation, control-plane unit tests, and CLI printing.
- The CLI table regression failed before adding the disabled state and
passed after the change.
- Linux amd64, Go 1.27.1: `go test -race ./...` passed for all 147
packages with tests.
- With `CI=true`, fresh race tests for control-plane functional tests
and PostgreSQL storage passed: 612 tests/subtests, zero failures and
zero skips.
- Removing both disabled-mode guards made the lifecycle regression fail
because it found the disabled template's golden actor. The complete
implementation passed.
- `hack/verify-all.sh`: all 14 verifiers passed with unmodified
settings. The first attempt hit a linter package-loading timeout; a
longer standalone lint run reported `0 issues`, and the standard
verifier run then passed after the cache warmed.
- Linux builds of `ateapi` and `kubectl-ate` passed.

Functional tests use real PostgreSQL and an envtest Kubernetes API with
a fake atelet. Real gVisor E2E was not run.

- [x] Appropriate changes to documentation are included in the PR.
- [x] Race tests, repository verifiers, and affected binaries were
checked.

---------

Co-authored-by: cloudonly <[email protected]>

* atunnel: strip X-Ate-Assignment-Stale from upstream actor responses (agent-substrate#2410)

### Motivation

`atunnel` defines `StaleAssignmentHeader = "X-Ate-Assignment-Stale"` to
distinguish an `atunnel` infrastructure-level routing rejection (HTTP
421 `Misdirected Request`, returned when an actor is no longer active on
that worker) from an HTTP 421 returned by the guest actor application
itself (`internal/atunnel/ingress.go#L45-L47`).

While `newActorProxy` already strips internal Substrate headers
(`ate-target-port`, `ate-target-actor`) on the request leg into the
sandbox, it lacked response sanitization on the return leg. As a result,
an untrusted actor workload running inside the container could return
`X-Ate-Assignment-Stale: true` and spoof an infrastructure routing
rejection to downstream routers or clients.
### Changes
1. **`internal/atunnel/ingress.go`**: Added a `ModifyResponse` hook to
`newActorProxy()` that removes `X-Ate-Assignment-Stale` from any
response received from the actor upstream.
2. **`internal/atunnel/ingress_test.go`**: Added unit test
`TestServeHTTPStripsStaleAssignmentHeaderFromActor` asserting that even
if the actor application returns `X-Ate-Assignment-Stale: true`, the
header is stripped before the response reaches the client.
### Testing
- `go test -race -count=1 ./internal/atunnel/...`
- `./hack/verify/boilerplate.sh`
- `./hack/verify/gofmt.sh`
- `./hack/verify/golangci-lint.sh`
> The PR title becomes the release note. Write it for users: what
changed for them, not how the code changed.

AI was used to assist in the development of these changes.

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

* WorkerPools resolve their SandboxConfig and protect it from deletion (agent-substrate#2393)

Part of agent-substrate#587. Step 3 of the SandboxConfig management design, after agent-substrate#2332
and agent-substrate#2333.

Each WorkerPool now resolves one SandboxConfig per sandbox class,
reports it in status, and protects it from being deleted while in use.
Workers do not read the resolved config yet; that is the next step.

## How to review

The PR is three commits. Each one builds and passes its tests on its
own, so they can be reviewed in order.

1. **Resolve each WorkerPool's SandboxConfig into status.** The API
change, the resolution rules, and the `Resolved` condition. No events,
no finalizer.
2. **Protect SandboxConfigs in use with a finalizer.** The finalizer,
the reconciler that releases it, the two-step status write for a switch,
and the `ate-setup` and `install-microvm-deps.sh` changes that deal with
the finalizer.
3. **Emit events for a WorkerPool's SandboxConfig and add e2e tests.**
The Warning and `SandboxConfigSwitched` events, and an e2e suite that
walks a pool through every move.

## Resolution

Without `configRef`: the pool uses the class default.

```yaml
apiVersion: ate.dev/v1alpha1
kind: WorkerPool
metadata:
  name: counter-pool
  namespace: ate-demo
  generation: 1
spec:
  replicas: 2
  workerImage: ateom-gvisor:v1
  sandboxClasses:
  - name: gvisor                  # no configRef
status:
  replicas: 2
  readyReplicas: 2
  selector: ate.dev/worker-pool=counter-pool
  sandboxClasses:
  - name: gvisor
    configRef:
      name: gvisor-default        # newest gvisor config annotated is-class-default
    conditions:
    - type: Resolved
      status: "True"
      reason: Resolved
      message: resolved to SandboxConfig "gvisor-default"
      observedGeneration: 1
```

With `configRef` pointing at a config that does not exist: the pool
keeps the config it resolved last time.

```yaml
apiVersion: ate.dev/v1alpha1
kind: WorkerPool
metadata:
  name: pinned-pool
  namespace: ate-demo
  generation: 2
spec:
  replicas: 2
  workerImage: ateom-gvisor:v1
  sandboxClasses:
  - name: gvisor
    configRef:
      name: gvisor-release2       # does not exist
status:
  replicas: 2
  readyReplicas: 2
  selector: ate.dev/worker-pool=pinned-pool
  sandboxClasses:
  - name: gvisor
    configRef:
      name: gvisor-release1       # still in use, still protected
    conditions:
    - type: Resolved
      status: "False"
      reason: SandboxConfigNotFound
      message: SandboxConfig "gvisor-release2" not found; keeping "gvisor-release1"
      observedGeneration: 2
```

- With `configRef`: that config is used. It must exist, belong to the
same class, and not be in the process of being deleted.
- Without `configRef`: the newest config of the class that is annotated
`sandboxconfig.ate.dev/is-class-default: "true"` and is not being
deleted is used.
- If no config can be resolved, the status `configRef` keeps the config
resolved last time, and the `Resolved` condition is `False` with reason
`SandboxConfigNotFound`, `SandboxClassMismatch`,
`SandboxConfigDeleting`, or `DefaultConfigNotFound`. A Warning event is
emitted when the reason changes.
- `kubectl get workerpool` gains `SANDBOXCONFIG` and `RESOLVED` columns.
- The counter demo no longer pins its pool to `gvisor-default`. It
follows the class default.

## Switching configs

When a class switches to a different config, the controller writes
status twice. First it records the new config in
`status.sandboxClasses[].pendingConfigRef`, which protects the new
config while the current one stays in `configRef`. Then it reads the new
config from the API server. If the config exists, belongs to the class,
and is not being deleted, `configRef` moves to it. Otherwise the class
keeps the current config and `Resolved` becomes `False`. This prevents a
pool from recording a config that was deleted between the cache read and
the status write.

## Events

Each time a class's `configRef` changes, the controller emits a Normal
event with reason `SandboxConfigSwitched`. The event is emitted after
the status write that records the change, so each switch shows up once.
A reconcile that leaves the config unchanged emits nothing, so the
pool's events show the configs it has used, in order. A class that fails
to resolve emits a Warning event with the condition reason.

To see the history for one pool:

```
kubectl -n ate-demo events --for workerpool/counter-pool
```

Example for the counter pool, which has no `configRef` and so follows
the class default. It first resolved to `gvisor-default`. Then a newer
config `gvisor-default-v2` annotated `is-class-default: "true"` was
created, and the pool switched to it on its own. Then
`gvisor-default-v2` lost the annotation, got it back, and was finally
deleted. A config that is being deleted is no longer a class default, so
the pool moved back to `gvisor-default`, and the finalizer on
`gvisor-default-v2` was released once no pool used it.

```
$ kubectl -n ate-demo-counter events --for workerpool/counter
LAST SEEN   TYPE     REASON                  OBJECT               MESSAGE
61s         Normal   SandboxConfigSwitched   WorkerPool/counter   sandbox class "gvisor" uses SandboxConfig "gvisor-default"
53s         Normal   SandboxConfigSwitched   WorkerPool/counter   sandbox class "gvisor" switched from SandboxConfig "gvisor-default" to "gvisor-default-v2"
46s         Normal   SandboxConfigSwitched   WorkerPool/counter   sandbox class "gvisor" switched from SandboxConfig "gvisor-default-v2" to "gvisor-default"
39s         Normal   SandboxConfigSwitched   WorkerPool/counter   sandbox class "gvisor" switched from SandboxConfig "gvisor-default" to "gvisor-default-v2"
33s         Normal   SandboxConfigSwitched   WorkerPool/counter   sandbox class "gvisor" switched from SandboxConfig "gvisor-default-v2" to "gvisor-default"
```

## Deletion protection

- Before recording a config in status, the WorkerPool controller adds
the finalizer `sandboxconfig.ate.dev/workerpool-protection` to it.
- A new `SandboxConfigProtectionReconciler` removes the finalizer once
no WorkerPool status references the config. It checks the cache first,
then confirms with a live list from the API server before removing it.
- A SandboxConfig that is being deleted is never resolved for a new
WorkerPool and is no longer treated as the class default. WorkerPools
that already use it keep using it, and because of the finalizer the
config is not actually deleted until every such WorkerPool has moved to
another config or has been deleted.

This is what the bundled `gvisor-default` looks like once a WorkerPool
has resolved it. Nothing else in the config changes.

```
kubectl get sandboxconfig gvisor-default -o yaml
```

```yaml
apiVersion: ate.dev/v1alpha1
kind: SandboxConfig
metadata:
  name: gvisor-default
  annotations:
    sandboxconfig.ate.dev/is-class-default: "true"
  finalizers:
  - sandboxconfig.ate.dev/workerpool-protection   # added by the WorkerPool controller
spec:
  sandboxClass: gvisor
  defaultVersion: v1
  versions:
  - name: v1
    state: Enabled
    ...
```

After `kubectl delete sandboxconfig gvisor-default` while a WorkerPool
still uses it, the delete does not finish. The config gets a
`deletionTimestamp` and stays in the cluster with the finalizer until
the last WorkerPool moves off it:

```yaml
apiVersion: ate.dev/v1alpha1
kind: SandboxConfig
metadata:
  name: gvisor-default
  annotations:
    sandboxconfig.ate.dev/is-class-default: "true"
  deletionTimestamp: "2026-10-11T18:42:07Z"
  finalizers:
  - sandboxconfig.ate.dev/workerpool-protection   # kept while a WorkerPool status references the config
spec:
  ...
```

To find out which WorkerPools are holding it:

```
kubectl get workerpool -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,SANDBOXCONFIG:.status.sandboxClasses[*].configRef.name
```

## Testing

- Unit tests cover resolution, the status write, the finalizer, the
two-step switch, and the events, including that a switch is reported
once when the first status write is rejected.
- The new e2e suite `internal/e2e/suites/sandboxconfig` creates a pool
with no `configRef` and no workers and walks it through the sequence in
the events example above. Run it with `hack/run-e2e.sh
./internal/e2e/suites/sandboxconfig/...`.

## Breaking changes

- A new WorkerPool gets no Deployment until every sandbox class has a
resolved config. Before upgrading, make sure each class has a config
annotated `is-class-default: "true"`, or set `configRef` on the pool.
The bundled `gvisor-default` and the micro-VM config are already
annotated. Existing pools keep their Deployment.
- `spec.sandboxClasses[].name` is now immutable. To change the class,
create a new pool.
- Rolling atecontroller back to a version before this PR leaves the
finalizer on SandboxConfigs. Remove it with `kubectl patch sandboxconfig
<name> --type=merge -p '{"metadata":{"finalizers":null}}'`.

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

* Simplify Snapshot API for Actor Lifecycle v2 (agent-substrate#2449)

The initial API developed in agent-substrate#2289 had a repeated SnasphotStorage
structure keyed by Durability (RESIDENT, LOCAL and DURABLE). However
this was overly descriptive. RESIDENT and LOCAL storage details are
local to atelet, and the promotion of snapshots from RESIDENT -> LOCAL
-> DURABLE is also driven by atelet. Putting the information in Snapshot
API meant that every action triggered by atelet has to invoke ateapi
first to record the transition.

Instead, this PR simplifies the sturcture by having a single
SnapshotStorage to contain the durable storage details. Locality details
will not be recorded in the Snapshot API. We only record that a snapshot
also has locality on a certain node.

#### Breaking change
LocalSnapshot is removed and replaced by a Snapshot.locality field.

Part of agent-substrate#2069

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

---------

Signed-off-by: timflannagan <[email protected]>
Signed-off-by: Eitan Yarmush <[email protected]>
Co-authored-by: Taahir Ahmed <[email protected]>
Co-authored-by: Tim Flannagan <[email protected]>
Co-authored-by: cloudonly <[email protected]>
Co-authored-by: Dmitry Berkovich <[email protected]>
Co-authored-by: ktsoator <[email protected]>
Co-authored-by: cloudonly <[email protected]>
Co-authored-by: Pancho Trujillo <[email protected]>
Co-authored-by: Zoe Zhao <[email protected]>
Co-authored-by: Michelle Au <[email protected]>
Dmitry Berkovich (dberkov) pushed a commit to dberkov/substrate that referenced this pull request Oct 17, 2026
…gent-substrate#2581)

Fixes agent-substrate#1709

Follow-up to agent-substrate#1997, which did the same for atelet's AteworkerHerder
RPCs. This brings declarative validation (DV) to the other end of that
call: the `Ateworker` service both sandbox runtimes (`ateworker-gvisor`,
`ateworker-microvm`) implement.

Generated validation lives in a new `internal/ateworkervalidation`
package, since both binaries are `package main` and must share one set
of rules. Every Ateworker handler calls its validator first thing,
before taking any lock, and turns failures into `InvalidArgument`
through `resources.ToAPIError`.

Rules are the ones atelet already enforces on the same values one hop
earlier, shared through `internal/resources` so the two ends cannot
drift: short names and UUIDs for identity, `ValidateMountPath` for every
mount path, `ValidateHostPort` for the egress gateway,
`ValidateHTTPGetPath` for the wakeup probe, `ValidateActorDirs` for the
seven actor directories. Bounds mirror the producer's: `containers ≤
10`, each mount list ≤ 32, `cpu_milli ≤ 999999`, `runtime_asset_paths ≤
16` entries like atelet's `ArchAssets.files`.

Two places ateworker's shape differs from atelet's, and the rules follow
the shape:

- A container's mounts arrive in four typed lists rather than atelet's
one. Each list's key keeps its own paths unique, so a hook on
`Container` gathers all four and holds their paths unique and
non-nesting across them, reporting each finding on the mount it belongs
to.
- `WorkloadSpec.containers` is optional on the message, because the
fallback spec atelet's Terminate sends for a template-less actor carries
no containers. Run and Restore require at least one through a hook on
their `spec` field, so neither runtime boots a sandbox around nothing.

#### Hand-written checks this replaces

  - `validateActorDirs` in both binaries, now the `ActorDirs` hook.
  - The `actor_uid` presence checks in both stats handlers.
- The empty-container guards in ateworker-microvm's boot and restore
paths, now the `spec` hooks above.
- ateworker-microvm's cap of 25 containers, which no accepted spec could
reach once `containers` is bounded at 10.
`ateworkervalidation.MaxContainers` names the bound.
- The `actor UID is required` guard in each binary's `hostActor`,
unreachable once every request that hosts an actor requires the UID.

 #### Removed fields

`CheckpointWorkloadRequest.snapshot_uri` and
`RestoreWorkloadRequest.snapshot_uri` date from when ateworker moved
snapshots to object storage itself. atelet does that now, through the
object-store plugin; nothing sets the fields and neither runtime reads
them. They are removed rather than validated.

 #### What is deliberately *not* validated here

- **Snapshot fidelity support.** DV only bounds `fidelity` to the enum.
Which levels a runtime can capture (gVisor cannot do ROOTFS without
MEMORY) stays the per-runtime check in each binary, as designed in
agent-substrate#2448: atelet forwards every level and ateworker refuses what it cannot
serve.
- **`runsc_path` / `runtime_asset_paths` contents.** They must be files
under the node's static-files root; that depends on host state and stays
a hand check in the binary. DV bounds their length only.
- **`container_spec`.** atelet derives it from template fields it
already validated, merging the image's ENTRYPOINT, CMD and ENV in, so
the template's bounds do not apply. It is marked opaque with that
reason.

 #### Testing

- `internal/ateworkervalidation`: one table per request type, plus
tables for `WorkloadSpec`, `Container` (which covers the mount rules,
including the cross-list cases) and
  `ActorDirs`. The probe is covered from the Run table.
- Each binary's edge test asserts every RPC with a request body rejects
an empty one. Existing handler tests that sent minimal requests now
build valid ones, so they
  test the check they are aimed at.
- `go vet` for `GOOS=linux` on both binaries at every commit (their
tests are Linux-only and run in CI).
  - `hack/update/codegen.sh` is clean at every commit.

  - [x] Tests pass
  - [x] Appropriate changes to documentation are included in the PR
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants