Update Snapshots API to support Actor Lifecycle v2 - #2289

…ferred_fidelity (agent-substrate#2364)

**This is a rename, not a behavior change.** `SnapshotConfig.onCommit`
becomes `preferredFidelity`, and the two-valued `SnapshotContentScope`
enum becomes the ordered `SnapshotFidelity` enum, with the same
vocabulary carried through atelet, ateom, and the metric label. `MEMORY`
is exactly what `FULL` was and `VOLUMES` is exactly what `DATA` was:
what gets captured on pause and suspend, what gets restored, the
golden-actor override, and the paused-upload narrowing are all
unchanged. The point is to put the lifecycle-v2 names in place before
step III adds `required_fidelity` and `resumed_fidelity` on top of them.
The one addition is `ROOTFS`, defined in the enum so it is complete but
rejected everywhere, since no runtime can produce it yet.

Part of agent-substrate#2102, step II as tracked in agent-substrate#2351. Step I (agent-substrate#2309) removed
`on_pause`.

Two commits, meant to be read in order.

## 1. ateapi: `SnapshotFidelity` and `preferred_fidelity`

- New ordered enum `SnapshotFidelity { VOLUMES = 1, ROOTFS = 2, MEMORY =
3 }`, each level including the ones below it. `SnapshotContentScope` is
deleted. MEMORY is the old FULL and VOLUMES the old DATA, so behavior is
unchanged.
- `SnapshotConfig.on_commit` becomes `preferred_fidelity`, documented as
a best-effort ceiling: the system never packages more than this level
and may package less when a layer cannot be captured or is too expensive
at the time. A process resumed in place keeps its memory regardless.
Defaults to MEMORY. Field numbers rearranged to `storage_location = 1`,
`preferred_fidelity = 2`.
- `ExternalSnapshot` and `LocalSnapshot` record `fidelity` instead of
`content_scope`.
- ROOTFS is in the enum so it is complete, but no sandbox runtime
captures rootfs without memory yet, so a new field-level validator
rejects it at template admission. Actors only take their fidelity from a
template, so none can carry it.
- kubectl-ate tag output, demos, benchmarking manifests, glossary, API
guide, and architecture doc use the new vocabulary. The obsolete apitool
exemption is removed.

## 2. atelet, ateom: the same enum on the wire

- Both internal protos replace `SnapshotScope` with a `SnapshotFidelity`
mirroring the public one; request fields become `fidelity` and
`desired_fidelity`. Every hop maps level by level, ROOTFS included, and
nothing is promoted: an unexpected value goes out as UNSPECIFIED for the
next hop to reject.
- Defense in depth for ROOTFS: atelet rejects it in checkpoint, restore,
and paused-upload validation, and both ateom runtimes reject it on
`CheckpointWorkload` and `RestoreWorkload` through a shared
`resources.ValidateSnapshotFidelity`.
- A golden snapshot must record MEMORY (UNSPECIFIED is no longer
tolerated).
- The metric attribute `ate.snapshot.scope` (`full`/`data`/`unknown`)
becomes `ate.snapshot.fidelity` (`volumes`/`rootfs`/`memory`/`unknown`)
in the OTel registry, `ateattr`, the ateom phase logs, and the snapshot
manifest atelet writes and reads back when narrowing a paused upload.
Dashboards keyed on the old label need updating.

## Behavior changes

- **Breaking API change.** Templates that set `onCommit` or use
`SNAPSHOT_CONTENT_SCOPE_*` values are rejected. Per the pre-v1.0 policy
there is no compatibility shim, and previously stored templates and
snapshots are not expected to exist.
- Snapshot manifests written with the old `scope` key read back with no
fidelity, which the upload path already refuses with a clear message.
- Unchanged: what MEMORY and VOLUMES capture and restore, the
golden-actor MEMORY override, and the paused-upload narrowing from
MEMORY to VOLUMES.

## Relationship to agent-substrate#2289

agent-substrate#2289 restructures the snapshot status into `Snapshot` and
`SnapshotStorage`. If it lands first, the `fidelity` field in this PR
moves onto `SnapshotStorage` (per copy), which is where it belongs
long-term; the enum, the template field, and the data-plane change are
the same either way.

## Follow-ups (step III)

`required_fidelity` with `required <= preferred` validation and the
CRASHED rule, `ResumeActorResponse.resumed_fidelity`, and ROOTFS support
in the micro-VM runtime.

## Verification

- `go test` for ateapi (unit and functional), kubectl-ate, atelet,
ateom-microvm, ateom-gvisor, ateomphaselog, ateattr, resources, and
apitool pass; new validator at 100% coverage.
- golangci-lint, kube-api-linter, python-protos, boilerplate verifiers
pass; codegen, proto-fmt, licenses, go.mod show no drift.
- Not run locally: the Weaver metrics-registry check and shellcheck,
which need Docker; relying on CI for those. e2e suites compile but were
not run against a cluster.

- [x] Breaking change
- [x] Not backward compatible

🤖 Generated with [Claude Code](https://claude.com/claude-code)

The initial API developed in agent-substrate#2289 had a repeated SnasphotStorage
structure keyed by Durability (RESIDENT, LOCAL and DURABLE). However
this was overly descriptive. RESIDENT and LOCAL storage details are
local to atelet, and the promotion of snapshots from RESIDENT -> LOCAL
-> DURABLE is also driven by atelet. Putting the information in Snapshot
API meant that every action triggered by atelet has to invoke ateapi
first to record the transition.

Instead, this PR simplifies the sturcture by having a single
SnapshotStorage to contain the durable storage details. Locality details
will not be recorded in the Snapshot API. We only record that a snapshot
also has locality on a certain node.

#### Breaking change
LocalSnapshot is removed and replaced by a Snapshot.locality field.

Part of agent-substrate#2069

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

* ateapi: record one scheduler assignment for each resume (#2244)

Part of https://github.com/agent-substrate/substrate/issues/2243

A worker can hold more than one actor, so two resumes can race for its
last free slot. The loser is retried transparently, but each losing
attempt was recorded on `ate.scheduler.assignment.duration` as
`ate.scheduler.outcome="error"`. One resume gave two records instead of
one.

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

* Components honor OTEL_TRACES_EXPORTER=none and treat OTEL_METRICS_EXPORTER=prometheus as none where they serve /metrics (#2190)

Follow-up to #1878 (#1877). Uses the shared exporter parser from #2193,
as agreed in #2067.

- `OTEL_METRICS_EXPORTER=prometheus` stops the OTLP push on the
components that serve `/metrics`, since the specification defines it as
the pull exporter they already provide. Before, it warned and kept the
push, so each series arrived twice. `otlp,prometheus` keeps the push.
ateom serves no `/metrics`, so there `prometheus` is skipped as unknown
with a warning and the push stays on.
- `OTEL_TRACES_EXPORTER` is honored with the same rules, knowing `otlp`
(the default) and `none`. With `none` the provider keeps sampling and
propagating trace context but exports nothing.

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

* fake-dataplane: let the fake data plane host actors (#2300)

This PR builds on the fake data plane skeleton (#2272) so actors can
actually be placed and resumed on fake Workers. With the skeleton, every
resume failed with `no_capacity`. All changes are benchmark-only, under
`cmd/benchmarking/isolate/`. **No production code changes.** atelet,
ateom, ate-api-server and atecontroller are untouched, and nothing here
runs in a normal install.

The fake data plane should cost ate-api-server what a real one does on
every call a benchmark exercises. Otherwise control-plane throughput
measured on it would look better than it really is. Each commit closes
one gap where a fake Worker behaved differently from a real one.

## What's in this PR

Six commits, best reviewed one at a time:

1. **fake-atelet: write a placeholder object for each external
snapshot.** ate-api-server copies an actor's snapshot when it creates a
tag, golden tags included, and refuses to copy an empty one. On an
external `Checkpoint` and on `UploadPausedCheckpoint`, fake-atelet now
writes one empty object under the snapshot URI. It writes to the backend
atelet would use, chosen by `ATE_STORAGE_BACKEND`. The write runs inside
the configured delay. If the write fails, the call fails, because a
later tag of that snapshot would fail too.

2. **Report fake Worker capacity through fake-atelet.** This fixes the
known limitation from #2272. ate-api-server accepts a capacity report
only from the atelet on the Worker's node. So fake-workersync plays each
fake Worker's ateom and sends the report to the fake-atelet on that
Worker's node, which relays `RegisterWorker` under its atelet identity.
- **Same values a real Worker reports:** cpu and memory come from the
pool's `template.resources.limits`, or from the node's allocatable when
no limit is set, which is what the downward API gives ateom. Max actors
is ateom's default of 1000 unless the `ate.dev/fake-max-actors`
annotation overrides it. The hardware identity, which ate-api-server
requires, comes from `hardware.ProbeHost()`, as ateom's does. Each
Worker reports once, when it is made, and keeps that capacity, as ateom
reads its limits once at startup.
- **`readyReplicas` means usable:** a pool counts a Worker ready only
once ate-api-server has accepted its capacity.
- **A bad annotation doesn't remove the pool:** if a pool's annotation
doesn't parse, or is out of range (1 to 2³¹−1), the controller reports
an error and leaves the pool's Workers as they are.
- **Relay security:** the relay listens on its own port (`--relay-port`,
default 8086) and implements only `RegisterWorker`. Both ends use mutual
TLS with the pod-identity certificates and check the peer's SPIFFE ID.
fake-atelet accepts only fake-workersync, and fake-workersync accepts
only the atelet identity. Without that check, any caller could set any
Worker's capacity. Both ends reread the trust bundle at each handshake,
as the herder does, so a CA rotation during a long run doesn't need a
restart.
- **Relay connections:** fake-workersync finds each node's relay by
listing the fake-atelet pods and keeps one connection per relay. A
connection is closed once its relay leaves the pod list.
- **Tests:** `fakeworkertest` issues certificates shaped like
pod-identity ones, so both ends of the relay are tested over real TLS
handshakes, including a trust bundle rotation.

3. **fake-atelet: mint each actor's certificate on activation.** In
production, every `Run` and `Restore` with tunneled egress costs
ate-api-server a certificate mint. The default install turns on tunneled
egress. fake-atelet now mints under its atelet identity, and the call
fails if the mint fails, just as tunneled egress fails closed in ateom.
The mint time is added on top of `--delay` rather than counted inside
it, so if mints slow down under load, resumes slow down by the same
amount. `--mint-actor-certificate=false` leaves the mint out, to measure
what it costs.

4. **fake-workersync: drain a Worker before deleting it.** Before this
change, a Worker that no pool wanted was deleted at once, even with
Actors still on it. Now it is retired the way a real worker pod is:
drained, then deleted once `ListWorkerActorAssignments` finds no Actors
on it. In production, ateom stops a terminating pod's Actors, but
nothing does that here. So after `--drain-grace` (default 1m) the Worker
is deleted anyway, and `DeleteWorker` releases the Actors still on it,
as the real syncer's delete does. A draining Worker doesn't count as a
replica. After a restart, `adopt` reads the draining state back from
ate-api-server.

5. **fake-workersync: follow WorkerPool label and template edits.**
These match what the real worker syncer and a Deployment rollout do:
- A label edit on a pool is written onto its Workers in place with
`UpdateWorker`. The Worker's version is the precondition, and a
conflicting write is retried on the next pass.
- An edit to the sandbox class, `template.resources.limits` or the
`ate.dev/fake-max-actors` annotation replaces the pool's Workers.
`sandbox_class` can't be changed on a Worker, and a real Worker's
capacity never changes, since ateom reads its limits and actor count
once at startup. A change in a node's allocatable leaves its Workers as
they are, as in production.
- Edits made while fake-workersync was down are still applied after it
restarts. An adopted Worker keeps the capacity ate-api-server recorded
and isn't reported again; one whose recorded capacity no longer matches
its pool is replaced.

6. **fake-workersync: register each replacement as a new Worker.** A
real Worker is named after its pod's UID, so a replacement pod is always
a new Worker. Before this change, a replaced fake Worker came back under
the same name, pod UID and IP. Then anything ate-api-server keyed by the
old Worker could carry over to its successor, which production never
allows.
- **New identity for each Worker:** each fake Worker is now named with a
fresh UUID, which is also its pod UID. `fakeworker.Name` becomes the
stand-in pod name, shared by every Worker that fills the same index of a
pool, and `adopt` finds this run's Workers by that pod name.
- **Replacement while the old Worker drains:** a replaced pod gets its
new Worker in the same pass, as a Deployment creates a replacement pod
without waiting for the old one to terminate. This covers sandbox class,
limits and max-actors edits, scaling down and back up, and Workers whose
node has left the benchmark set; those are replaced on a benchmark node,
as a pod is when its node goes away. If no benchmark node is listed at
all, Workers aren't moved off their nodes, since nothing could replace
them.
- **Status counts only usable Workers:** a pool's status counts only the
Worker currently serving each wanted pod. A Worker made from an older
template (sandbox class, limits or max actors), past `spec.replicas`, or
off the benchmark nodes doesn't count as a replica even if its drain
failed. It gets no label write or capacity report while the drain is
retried.

## Known limitations

- **Other template edits:** only sandbox class, limits and max-actors
edits replace fake Workers. Edits to the image, env or template labels
roll a real pool's pods but leave fake Workers in place.
- **One mint per activation:** fake-atelet mints an actor certificate
once per `Run`/`Restore`. Real ateom also renews it while the actor
runs, so multi-hour runs with long-lived actors under-count mint load.
- **List calls while draining:** each draining Worker costs one
`ListWorkerActorAssignments` per pass until its Actors leave or
`--drain-grace` runs out. Large scale-downs therefore add some load
during a measurement.
- **Storage client at startup:** fake-atelet creates its object-storage
client when it starts and exits if that fails. It runs as the atelet
service account, which already needs storage access.
- **Relay node check:** the relay client checks the relay's SPIFFE ID
and chain, but not that the relay is on the Worker's node, which
`internal/ateletdial` does check. A misrouted report is rejected by
ate-api-server and retried on the next pass.

## Not in this PR

The deploy manifest and the `benchmarking/workloads/deploy.sh
--fake-data-plane` mode come in a follow-up PR. Deploying these binaries
needs the following, which that PR adds:
- the fake-atelet relay port
- the servicedns trust bundle mount, for fake-atelet to verify
ate-api-server
- a Role that lets fake-workersync list the fake-atelet pods in
`ate-system`

Addresses #2261

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

* ateom-gvisor: emit actor usage records (#2194)

Part of #1748, split out of #1984: the gVisor ateom emits the actor
usage records, using #2005's building blocks. #1984 keeps the atelet
side.

- **Sampling.** The ateom samples its actors every
`--usage-sample-interval` (default 1m) and writes one
`ate.actor.usage_sampled` record per actor per sample.
- **Order.** Per activation: `initial` when it starts, then `periodic`,
then `final` after a checkpoint or terminate succeeds. The final reading
is taken before the snapshot, because a Full checkpoint stops the
sandbox's cgroup.
- **Discovery read.** `GetActiveWorkloadStats` serves the latest sample,
so atelet's poll reads no cgroup.
- **Re-host.** A read goes through the activation's own state, so a
re-host during it cannot mix two activations' numbers.

**Until #1984 lands**, atelet still writes its own stdout usage record,
so each gVisor actor has two usage records, with different messages and
keys.

**Tested:** unit tests. The combined change was tested on ate-dev
earlier, with `initial`, `periodic`, and `final` per epoch for both
checkpoint and terminate.

* ateom-microvm: emit actor usage records (#2195)

Part of #1748, split out of #1984: the micro-VM ateom emits the actor
usage records, using #2005's building blocks. #1984 keeps the atelet
side.

- **Sampling.** The ateom samples its actors every
`--usage-sample-interval` (default 1m) and writes one
`ate.actor.usage_sampled` record per actor per sample.
- **Order.** Per activation: `initial` once the guest answers, then
`periodic`, then `final` after a checkpoint or terminate succeeds. The
final record reuses the newest sample, so the checkpoint waits on no
guest read.
- **Discovery read.** `GetActiveWorkloadStats` serves the latest sample,
so atelet's poll reads no guest.
- **Re-host.** Every guest read goes through the activation's own state,
so a re-host during it cannot mix two activations' numbers.
- **Behaviour change.** A guest-stats target that names another actor
now logs and reports the actor pending instead of failing the discovery
RPC.
- **Initial read.** It runs off the resume path and recovers from a
panic, as a sweep does.

**Until #1984 lands**, atelet still writes its own stdout usage record,
so each micro-VM actor has two usage records, with different messages
and keys.

**Tested:** unit tests. The combined change was tested on ate-dev
earlier: CPU rises within an epoch and restarts at zero for resumed
guests.

* Delete unused `actor_template_ref` from `ActorAssigment` sub-resource (#2322)

The field name doesn't follow the convention and ends with "ref", but
more importantly, it's not really used.

Part of #1378

* Return the caller's context error when an atelet call fails (#2236)

When the caller's context ends during an atelet call, for example
because its deadline passes during a slow restore, the control API keeps
the actor assigned but returns `Internal` to the caller. So
`ResumeActor` can leave an actor `RESUMING` on a worker whose restore
already succeeded while the caller receives a server-failure status.

When the workflow context has ended, return its error, so the caller
gets `Canceled` or `DeadlineExceeded` (the `apierror` convention for
context errors). The actor keeps its worker assignment, so a retry can
finish the operation. A `Canceled` or `DeadlineExceeded` from atelet
with a live context still returns `Internal`, as on main, so the caller
is not told that it cancelled. `Unavailable` keeps the mapping from
#2196. Permanent errors keep their crash handling. This follows the
shape that @laoj2 suggested in review.

The unit tests cover a cancelled and an expired workflow context, atelet
`Canceled` and `DeadlineExceeded` with a live context, and the existing
permanent failures. The gRPC functional test checks `Unavailable`,
atelet `Canceled` and `DeadlineExceeded`, and a caller deadline that
passes during a slow restore. In each case the actor stays `RESUMING`
with its worker assignment, and a retry reaches `RUNNING` on that
worker. A permanent local restore failure still returns `Internal`,
records a crash, clears the assignment, and releases worker capacity.

### Validation

On main `0b91488d79637052458a29a6ba6c757bcf7a9292`, on 2026-10-08, the
`controlapi` (51.4 s) and `functionaltest` (64.9 s,
`REQUIRE_DOCKER=true`) race suites pass. Linux-targeted `go vet` and
`gofmt` are clean. Without the context-error branch, three
`TestHandleAteletError` subtests fail.

### Earlier lab evidence

These trials used the first version of this PR, which also mapped
atelet's own `Canceled` and `DeadlineExceeded` with a live context. With
the current version, the second and third rows return `Internal` again.
The actor still stays assigned and the retry still works.

A separate two-worker ARM64 lab ran three trials per error code. A
diagnostic interceptor replaced the reply only after the real atelet
`Restore` handler succeeded. It tested selected error replies, rather
than physical packet loss.

| Injected reply code | Before: client code | With this mapping: client
code | Fresh trials with the fix |
| --- | --- | --- | ---: |
| `Unavailable` | `Unavailable` | `Unavailable` | 3 passed |
| `Canceled` | `Internal` | `Canceled` | 3 passed |
| `DeadlineExceeded` | `Internal` | `DeadlineExceeded` | 3 passed |

All nine trials passed 72 assertions. The actor retained its worker
assignment and one observed live guest; explicit retry reached `RUNNING`
with the same actor UID, guest execution ID, worker Pod UID, and
checkpoint from reply loss through retry. A local AX outcome-reporting
candidate consequently retained an uncertain result instead of reporting
a definitive failure.

Those trials used the unchanged AX candidate
`55602feda4accf71d17705d5e6c48bc2c1a5ff1e` on `be7b1a8`, and Substrate
mapping commit `438291462203f253366b1e5f8fc4f5b3fd63af9f` on an earlier
lab base containing the availability candidate from #2055. This PR
contains only the three-file mapping/test patch, freshly checked on
current upstream main; the earlier trials are supporting evidence, not a
live run of this rebased branch.

The richer retry contract discussed in #1562 and automatic worker
recovery remain separate work.

* benchmark(ateom-gvisor): log per-actor checkpoint and restore phase b… (#2302)

### ateom-gvisor: log per-actor checkpoint and restore phase breakdowns

Follow-up to #1941, which gave ateom-microvm the per-phase `Checkpoint
timing breakdown` / `Restore timing breakdown` records. This PR gives
ateom-gvisor the same, so the report in #2009 covers both sandbox
classes.

#### Why it need

On a gVisor pool the breakdown stops at atelet's `ateom_restore` /
`ateom_checkpoint` phase. That phase is most of a gVisor wake — roughly
165 ms of a 190 ms resume in local measurements — and nothing inside it
is timed, so there is no way to say whether a slow resume was the
network setup, the pause container, the app container's `runsc restore`,
or the wakeup probe. gVisor is the default sandbox class, so this is the
larger blind spot of the two.

#### What

- **`cmd/ateom-gvisor`**: `CheckpointWorkload` and `RestoreWorkload`
emit one joinable record per operation from a `defer`, with the actor
identity, snapshot scope, `error.type` on failure, and one float-seconds
field per phase under
`ateom.actor.{checkpoint,restore}.duration.<phase>`. Both records are
written on failure too, carrying the phases that completed and the
failing step's elapsed time.
- restore phases: `prep`, `egress_prepare`, `egress_join`, `net_setup`,
`durable_dir`, `pause_rootfs`, `pause_create`, `pause_restore`,
`app_rootfs`, `app_create`, `app_restore`, `wakeup_probe`, `activate`,
`total` — sequential, they partition the total. The restore record also
carries the container count.
- checkpoint phases: `pause`, `checkpoint`, `resume`, `teardown`,
`total`.
A small `restoreTiming` / `checkpointTiming` pair and a `lap()` helper
stamp the phases as the handler advances; `phaselog_test.go` covers
them.
- **`internal/ateomphaselog`** (new): the record rendering both ateoms
now share — `Phase`, `SnapshotPhaseAttrs`, `LogSnapshotPhases`,
`ScopeLogValue` — moved out of `cmd/ateom-microvm/phaselog.go` with its
tests. Phase *names* stay private to each binary, as in #1941: they are
implementation details of one runtime, not values of the
`ate.snapshot.phase` metric enum, and this PR adds nothing to the
registry.
- **`cmd/ateom-microvm`**: a mechanical refactor onto the shared
package; the micro-VM records are unchanged.
- **`docs/observability.md`**: the ateom paragraph now describes both
runtimes' records and their failure semantics.

No new instruments, no registry change, no behavior change.

#### Example

```json
{"msg":"Restore timing breakdown","ate.actor.uid":"8f2a…","ate.snapshot.scope":"full",
 "ateom.actor.restore.duration.prep":0.004,
 "ateom.actor.restore.duration.net_setup":0.011,
 "ateom.actor.restore.duration.pause_restore":0.038,
 "ateom.actor.restore.duration.app_restore":0.097,
 "ateom.actor.restore.duration.wakeup_probe":0.021,
 "ateom.actor.restore.duration.total":0.183,
 "container_count":2}
```

Joined with atelet's record for the same actor (which carries
`ate.sandbox.class=gvisor`), `phase_report.py` from #2009 lists the
gVisor rows alongside the micro-VM ones. The analyzer needs a small
follow-up to stop assuming every ateom record is micro-VM (class from
the paired atelet record, residual rule by class); that goes with #2148.

### Testing

- `go test ./internal/ateomphaselog/...` — 4 tests (record shape,
seconds not nanoseconds, scope mapping, `error.type` for gRPC / context
/ plain errors).

* Don't default name in store layer. (#2359)

Be consistent with other sub-resources (e.g. EgressPolicy) and validate
in the RPC layer and let store honor what the client specifies.

* kubectl ate admin can rotate the actor JWT signing key (#2140)

Part of #1756.

- Adds `kubectl ate admin add-jwt-key`, `activate-jwt-key`, and
`remove-jwt-key`, so rotating the actor JWT signing key no longer means
editing the pool Secret by hand. A new key is published but signs
nothing until activated, which gives relying parties time to fetch it.
- Adds `list-jwt-keys`, which prints each key's ID and algorithm and
marks the active one, so finding the key to remove doesn't mean decoding
a Secret full of private keys.
- `ConcretePool` gains `AddAuthority`, `Activate`, `RemoveAuthority`,
and `ActiveID`. They refuse a duplicate or empty key ID, activating a
key the pool doesn't hold, and removing the key that signs.
- Each command writes the Secret back conditional on the resourceVersion
it read and retries on a conflict, so a concurrent edit isn't lost.
- Generated RS256 keys are now 4096-bit RSA instead of 2048-bit. RS256
stays only for relying parties that require it, such as Azure workload
identity federation; ES256 is still the default.
- The pool commands share one Kubernetes client helper, and
make-jwt-pool's flag variables are renamed so the new commands can reuse
them.

The commands don't wait for relying parties; the operator decides when
to activate and remove. The `--key-id` help describes the PKIX key IDs
from #2132, which main doesn't use until that PR merges.

Testing: unit tests for the pool methods, and fake-clientset tests for a
full rotation, a conflict retry, refused changes, and the listing, plus
a test that generated RS256 keys are 4096-bit. Not run against a
cluster.

- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR

* cdiinject: confine device paths and host mounts to allowed roots (#2315)

Validate CDI device node paths under /dev via os.OpenRoot so symlinks
cannot escape the device root, and require CDI mount HostPaths (and ELF
DT_SONAME reads during SONAME symlink staging) to resolve within an
explicit AllowedHostDirs allowlist via os.OpenRoot).

Fixes https://github.com/agent-substrate/substrate/issues/2091

This code isn't used at the moment, but we know we want to add back
device injection in the future, and we know AI security scans will find
it. For now I think that's cheaper than deleting and re-introducing (and
possibly bringing back this bug)

* Add sections about sub-resources and singletons to the API style guide. (#2354)

Also add a resource-model.md markdown file that goes over the current
resources and reference it from AGENTS.md in API proto directory.

* ate-setup: choose the StorageClass of the bundled PostgreSQL (#2314)

Adds `--postgres-storage-class` / `ATE_INSTALL_POSTGRES_STORAGE_CLASS`.
Unset keeps the cluster default, since the claim template is immutable
on existing installs.

Fixes #2298

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

* Move snapshot transfers behind the object-store snapshot plugin (#2224)

Part of #1827

Move external snapshot transfers in atelet and ate-api-server behind the
`objectstoresnapshot.v1` plugin API, and ship a reference plugin for GCS
and S3 that runs as a sidecar next to each.

## What changes

| Commit | Change |
|---|---|
| objectstoreplugin: add object-store plugin and snapshot-plugin binary
| `internal/objectstoreplugin` implements `NodeProvider` and
`ControlProvider` on GCS/S3. `cmd/snapshot-plugin` serves one of them on
a Unix socket (`node` or `control` mode). A `healthcheck` mode checks a
running plugin's gRPC health service, for the sidecar's startup probe. |
| atelet: transfer snapshots through NodeProvider | atelet fetches and
uploads external snapshots through the node plugin instead of calling
object storage. |
| ateapi: clean up and copy snapshots through ControlProvider |
ate-api-server deletes and copies external snapshots through the control
plugin instead of its own storage client. |
| manifests: run snapshot-plugin as atelet and ate-api-server sidecars |
Native sidecars next to atelet (node mode) and ate-api-server (control
mode), each sharing a pod-local emptyDir for the socket. Each sidecar
has an exec `startupProbe` running `snapshot-plugin healthcheck` against
its socket, so the main container starts only once the plugin serves. |

## Reviewing

- **Moved code.** The node plugin's per-file upload and download are
atelet's code moved as is, so it can be found in the old location on
`main`. The control plugin wraps the existing `objectstore.DeletePrefix`
and `CopyPrefix`. New code is limited to the gRPC glue, request
validation, the startup health check, the binary, and the manifests.
- **Unchanged object layout.** Files are stored as `<file>.zstd` with an
uncompressed `manifest.json`, so snapshots written before and after this
change are interchangeable. The manifest still goes last, in its own
call, and remains the commit marker.
- **Validation added at the plugin boundary.** The plugin now serves a
socket API, so it checks every request: the snapshot URI, that the local
directory is inside `--root` (`/var/lib/ate`), and that file names are
base names with no duplicates. Every directory atelet passes today is
under `/var/lib/ate`, and the file-name checks are the ones atelet
already applied before transferring.
- **A plugin that never comes up fails fast.** Clients dial with
`WaitForReady`, so without a check the first transfer would hang. atelet
and ate-api- server run `objectstoreplugin.WaitReady` (a gRPC health
check) at startup with a one-minute bound and exit if the plugin is not
serving. The sidecar's `startupProbe` runs that same health check by
executing `snapshot-plugin healthcheck` inside the container, so the
kubelet does not start atelet or ate-api-server until the plugin serves.
Transfer RPCs get no fixed deadlines, since one would be wrong for large
snapshots
- **Shared backend selection.** atelet's `ATE_STORAGE_BACKEND` selection
moves to `objectstorage.NewFromEnv` in `pkg/objectstorage`, along with
`UsesS3` and `NewS3ClientFromEnv`, so atelet and both plugin modes
select the backend the same way. These are new exported functions in
`pkg/`.
- **Package location.** The plugin lives in
`internal/objectstoreplugin`, not `internal/plugins/`, because atelet
and ate-api-server tests import its in-process clients
(`objectstoreplugintest`) and it imports `internal/objectstore`.

## Not in this PR

- `actor_jwt` is not set yet. Per-actor authorization is #777.

## Breaking change
The object storage settings (`ATE_STORAGE_BACKEND`, `AWS_*`) move from
the `ate-api-server` container to its snapshot-plugin` sidecar, which is
now the only process in that pod that touches object storage for
snapshots. ate-api-server exits at startup if `ATE_STORAGE_BACKEND` is
still set on it. An install that patches these settings onto the
`ate-api-server` container must patch the sidecar instead; see the kind
overlay for an S3 example.

## Testing

- `make test` and `make verify`.
- New unit tests for the plugins (`internal/objectstoreplugin`), backend
selection (`pkg/objectstorage`), and the shared validation helpers
(`internal/resources`).
- E2E on kind runs in CI (gVisor and micro-VM, S3 via rustfs).
- E2E on GKE (GCS)

* atespace isolation tests (#1966)

Add an e2e suite for the actor isolation boundary



Tests added (internal/e2e/suites/security):

- TestCrossTenantIsolation: the victim tenant writes a marker to its
durable-dir volume, reads it back, and is suspended; the attacker tenant
then resumes on the same worker. Controls confirm the probe reports
missing files as errors and can read its own durable data.
- VictimDurableMarkerNotVisibleThroughAttackerMount: the victim's marker
is not readable through the attacker's own /data mount.
- VictimHostStateNotReachableFromAttackerSandbox: the victim's durable
directory, projected actor identity, and sandbox asset record are not
readable at their host paths (taken from internal/ateompath), and the
attacker cannot write into the victim's durable directory.
- TagScopedToAtespaceCannotSeedAnotherTenant: an ATESPACE-scoped Tag
created from the attacker's snapshot cannot seed an Actor in the other
atespace. The test asserts the tag-scope rejection message, not just
FailedPrecondition, since other preconditions on that path return the
same code. A same-atespace clone from the tag restores the source's
durable data, proving the tag was usable.

Fixtures (internal/e2e/fixtures/security): a pool-bearing tenant, a
namespace-only second tenant that must schedule onto the first's pool,
and a shared probe-based ActorTemplate with a system-info volume and a
durable-dir volume. The pool-bearing half is registered in
sandbox_test.go's fixture rendering checks.

Fixes #<issue_number_goes_here>

> It's a good idea to open an issue first for discussion.

- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR

* SandboxConfig now holds multiple named versions with a default (#2333)

A SandboxConfig can now carry several sandbox versions, with
`spec.defaultVersion` selecting the one actors boot with:

```yaml
kind: SandboxConfig
metadata:
  name: gvisor-default
  annotations:
    sandboxconfig.ate.dev/is-class-default: "true"
spec:
  sandboxClass: gvisor
  defaultVersion: v1
  versions:
  - name: v1
    state: Enabled
    pauseImage: "registry.k8s.io/pause:3.10.2@sha256:..."
    assets: {...}
  - name: v0
    state: Disabled
    pauseImage: "registry.k8s.io/pause:3.10.2@sha256:..."
    assets: {...}
```

- `spec.pauseImage` and `spec.assets` move into `spec.versions[]`. Each
entry has `name` (a DNS label, unique in the list), `state` (`Enabled`
by default, or `Disabled`), `pauseImage` and `assets`. A config holds
1–16 versions.
- CEL requires `defaultVersion` to name an existing version that is not
`Disabled`, and applies the per-class pauseImage rule to every version.
- The ValidatingAdmissionPolicy checks the per-class required assets on
every version.
- The shipped `gvisor-default` and `microvm` configs now use
`defaultVersion: v1` and carry the
`sandboxconfig.ate.dev/is-class-default` annotation. More than one
config of a class may carry it. Nothing reads it yet; a later change
resolves it newest-wins.
- ate-api and atelet prewarm read only the default version for now.

#### Breaking change

Every SandboxConfig must use the new shape. The apiVersion stays
`v1alpha1` and there is no conversion, so once the CRD is updated,
`spec.pauseImage` and `spec.assets` are pruned from existing objects,
and ate-api rejects actors that name them (FailedPrecondition) until
they are re-applied with `defaultVersion` and `versions`. The install
re-applies `gvisor-default` and `microvm`; custom SandboxConfigs must be
rewritten by hand. Roll out the CRD together with ate-api: an older
ate-api reading a new-shape config gets an empty pause image.

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

* Egress gateway can mint and cache actor JWTs (#2135)

Part of #1660.

- Adds the `egress/actorjwt` package, whose `Minter` mints actor JWTs
through `Control.MintActorJWT`. Tokens are cached by actor, sorted
audiences, and lifetime, keyed by the deterministic protobuf encoding of
the mint request, and each is reused until a third of its lifetime
remains.
- Concurrent misses share one mint. It keeps running if the request that
started it goes away and has no timeout of its own, so a slow mint still
fills the cache.
- The gateway knows the actor only by name, so it sends no UID (#2207).
An actor recreated under the same name can get its predecessor's token
until the refresh point.

Nothing calls the minter yet, so `actor_jwt` entries still get a 501.
The next PR in the series wires it into `applyEffects`.

Testing: unit tests call a fake Control server over bufconn and cover
reuse, keying, errors, and concurrent and canceled callers. They pass
under `-race`. E2E hasn't run.

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR (no
user-facing change yet; docs come later in the #1660 series)

* Build the OCI spec in ateom instead of atelet (#2293)

Today atelet wrote each container's `config.json`, and each ateom loaded
it and reshaped it for its runtime.

To better decouple atelet and ateom, the ateom Container message now
carries each container's `argv`, `environment`, `capabilities` and
resource limits next to its volume mounts. atelet still resolves argv
and env, because it pulled the image and has the image config. It also
still resolves the capability set from the template.

`ocispec.Build` now takes the ateom Container and the actor's
directories.
- ateom-gvisor builds and saves the spec before runsc create or restore,
including the pause container's.
- ateom-microvm builds it in memory. For the bundle, atelet only pulls
images and writes the overlay spec.

The pause container also starts with an empty environment instead of the
pause image's ENV and a default PATH.


- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

* security: prevent path traversal in snapshot storage locations (#2372)

see: #2077 

prevents path traversal. we append to the base path here with a random
actor UID so it should be hard to exploit this but let's prevent path
traversals anyhow.

still discussing if we should make other changes, but this is cheap to
do now.

* Replace SnapshotContentScope with the SnapshotFidelity ladder and preferred_fidelity (#2364)

**This is a rename, not a behavior change.** `SnapshotConfig.onCommit`
becomes `preferredFidelity`, and the two-valued `SnapshotContentScope`
enum becomes the ordered `SnapshotFidelity` enum, with the same
vocabulary carried through atelet, ateom, and the metric label. `MEMORY`
is exactly what `FULL` was and `VOLUMES` is exactly what `DATA` was:
what gets captured on pause and suspend, what gets restored, the
golden-actor override, and the paused-upload narrowing are all
unchanged. The point is to put the lifecycle-v2 names in place before
step III adds `required_fidelity` and `resumed_fidelity` on top of them.
The one addition is `ROOTFS`, defined in the enum so it is complete but
rejected everywhere, since no runtime can produce it yet.

Part of #2102, step II as tracked in #2351. Step I (#2309) removed
`on_pause`.

Two commits, meant to be read in order.

## 1. ateapi: `SnapshotFidelity` and `preferred_fidelity`

- New ordered enum `SnapshotFidelity { VOLUMES = 1, ROOTFS = 2, MEMORY =
3 }`, each level including the ones below it. `SnapshotContentScope` is
deleted. MEMORY is the old FULL and VOLUMES the old DATA, so behavior is
unchanged.
- `SnapshotConfig.on_commit` becomes `preferred_fidelity`, documented as
a best-effort ceiling: the system never packages more than this level
and may package less when a layer cannot be captured or is too expensive
at the time. A process resumed in place keeps its memory regardless.
Defaults to MEMORY. Field numbers rearranged to `storage_location = 1`,
`preferred_fidelity = 2`.
- `ExternalSnapshot` and `LocalSnapshot` record `fidelity` instead of
`content_scope`.
- ROOTFS is in the enum so it is complete, but no sandbox runtime
captures rootfs without memory yet, so a new field-level validator
rejects it at template admission. Actors only take their fidelity from a
template, so none can carry it.
- kubectl-ate tag output, demos, benchmarking manifests, glossary, API
guide, and architecture doc use the new vocabulary. The obsolete apitool
exemption is removed.

## 2. atelet, ateom: the same enum on the wire

- Both internal protos replace `SnapshotScope` with a `SnapshotFidelity`
mirroring the public one; request fields become `fidelity` and
`desired_fidelity`. Every hop maps level by level, ROOTFS included, and
nothing is promoted: an unexpected value goes out as UNSPECIFIED for the
next hop to reject.
- Defense in depth for ROOTFS: atelet rejects it in checkpoint, restore,
and paused-upload validation, and both ateom runtimes reject it on
`CheckpointWorkload` and `RestoreWorkload` through a shared
`resources.ValidateSnapshotFidelity`.
- A golden snapshot must record MEMORY (UNSPECIFIED is no longer
tolerated).
- The metric attribute `ate.snapshot.scope` (`full`/`data`/`unknown`)
becomes `ate.snapshot.fidelity` (`volumes`/`rootfs`/`memory`/`unknown`)
in the OTel registry, `ateattr`, the ateom phase logs, and the snapshot
manifest atelet writes and reads back when narrowing a paused upload.
Dashboards keyed on the old label need updating.

## Behavior changes

- **Breaking API change.** Templates that set `onCommit` or use
`SNAPSHOT_CONTENT_SCOPE_*` values are rejected. Per the pre-v1.0 policy
there is no compatibility shim, and previously stored templates and
snapshots are not expected to exist.
- Snapshot manifests written with the old `scope` key read back with no
fidelity, which the upload path already refuses with a clear message.
- Unchanged: what MEMORY and VOLUMES capture and restore, the
golden-actor MEMORY override, and the paused-upload narrowing from
MEMORY to VOLUMES.

## Relationship to #2289

#2289 restructures the snapshot status into `Snapshot` and
`SnapshotStorage`. If it lands first, the `fidelity` field in this PR
moves onto `SnapshotStorage` (per copy), which is where it belongs
long-term; the enum, the template field, and the data-plane change are
the same either way.

## Follow-ups (step III)

`required_fidelity` with `required <= preferred` validation and the
CRASHED rule, `ResumeActorResponse.resumed_fidelity`, and ROOTFS support
in the micro-VM runtime.

## Verification

- `go test` for ateapi (unit and functional), kubectl-ate, atelet,
ateom-microvm, ateom-gvisor, ateomphaselog, ateattr, resources, and
apitool pass; new validator at 100% coverage.
- golangci-lint, kube-api-linter, python-protos, boilerplate verifiers
pass; codegen, proto-fmt, licenses, go.mod show no drift.
- Not run locally: the Weaver metrics-registry check and shellcheck,
which need Docker; relying on CI for those. e2e suites compile but were
not run against a cluster.

- [x] Breaking change
- [x] Not backward compatible

🤖 Generated with [Claude Code](https://claude.com/claude-code)

* Add sandboxClasses with an optional SandboxConfig ref to WorkerPool (#2332)

WorkerPool now declares its sandbox class as `spec.sandboxClasses`, with
an optional reference to a SandboxConfig:

```yaml
spec:
  sandboxClasses:
  - name: gvisor
    configRef:
      name: gvisor-default
```

- `spec.sandboxClass` is replaced by `spec.sandboxClasses`, a list of
`{name, configRef}`. The list is required and must have exactly one
entry for now. `name` is `gvisor` or `microvm` and has no default.
- `configRef` is optional and names a cluster-scoped SandboxConfig.
Nothing reads it yet: actors still get their sandbox binaries from the
ActorTemplate's SandboxConfig.
- Code that needs the pool's class calls
`WorkerPoolSpec.DefaultSandboxClass()`.

#### Breaking change

Every WorkerPool must now set `spec.sandboxClasses`. Once the CRD is
updated, `spec.sandboxClass` is pruned, so re-apply each existing pool
with `sandboxClasses` set (`- name: gvisor` for pools that used the old
default).
- [ ] Tests pass
- [x] Appropriate changes to documentation are included in the PR

* Fail snapshot plugin calls with Unavailable after 30s when the plugin is down (#2380)

Calls to the snapshot plugin used `WaitForReady(true)`, so once the
sidecar went down after startup a call waited out the caller's deadline
(10m in ate-api-server, unbounded without one).

- Calls wait at most 30s for the connection, then fail with
`Unavailable`. Calls already running aren't cut short, and a sidecar
restart within the window is ridden through.
- atelet and ate-api-server now return a plugin `Unavailable` as
`Unavailable` instead of `Internal`, so ate-api-server retries instead
of crashing the actor.
- The upload right after `CheckpointWorkload` still waits, since the
workload is already gone and failing there would lose state.

Tested against a real Unix-socket plugin: a 40s outage fails at 30s, a
12s outage succeeds.

- [x] Tests pass

---------

Signed-off-by: Anish Ramasekar <[email protected]>

* docs(api): fix systeminfo volume examples (#2346)

Remove the extra `spec` wrapper from the SystemInfo volume examples and
use the `ACTOR_METADATA_FIELD_*` enum names, so the examples can be
parsed by `kubectl ate create actor-template`.

To reproduce, copy the `actorMetadata` YAML example from the SystemInfo
Volumes section of
[docs/api-guide.md](https://github.com/agent-substrate/substrate/blob/760e577a38e66c0de51d05b3dad9a4db7dd0ff51/docs/api-guide.md#L188)
into `actor-template.yaml`, then run from the repository root:

```bash
go run ./cmd/kubectl-ate create actor-template -f actor-template.yaml
```

This fails with `unknown field "spec"`. Remove the top-level `spec` key
and unindent its contents, then run the same command again. It now fails
with `invalid value for enum field field: "name"`.

* atelet: drop its actor usage records and count each activation's CPU once (#1984)

The ateoms now write the actor usage records and serve their latest
sample (#2194, #2195), so atelet stops writing its own and counts CPU
per activation.

- **atelet CPU.** Baselines are keyed by actor and epoch. A new
activation counts in full if it began after atelet started, and a worker
that does not answer keeps its baselines. This fixes the undercount on
micro-VM restores and the double charge when a silent worker's baselines
were dropped.
- **atelet records removed.** The ateoms' `ate.actor.usage_sampled`
records replace them, ending the period with two usage records per
actor.
- **Timeout.** `GetActiveWorkloadStats` answers from memory, so atelet's
`statsRPCTimeout` drops from 55s to 5s. The 50s poll floor stays.
- **Cleanup.** `UsageEmitter.EmitFinal` and its flush, atelet's stdout
`SyncedWriter`, and the exported `actorlog.LabelsKey` go; nothing uses
them now.
- **Docs.** The proto comments describe the cached discovery read. The
registry and the guide say when the final record is written: whenever
the ateom tears an activation down (checkpoint, terminate, graceful
shutdown, or a micro-VM re-host), read just before the teardown.

#### Known limitation and future work

atelet's CPU counter only sees samples of actors still hosted, so it
loses each activation's tail: the CPU from the last sample atelet read
to the teardown, up to one poll interval plus one sample interval. An
activation shorter than that, the common checkpoint-when-idle case, can
lose most or all of its CPU in the metric. The usage records keep it,
through the final record.

To fix it later, the ateoms can own the counter: each ateom adds an
activation's CPU increase at every reading, initial, periodic, and
final, and exports it through the existing OTLP relay. No gRPC change is
needed. atelet then drops the baseline map, its start time, and the
silent-worker rule; this also removes the baselines a crashed worker's
leftover directory keeps. The cost is cardinality: the series become one
per worker pod per label set instead of one per node, still bounded by
pods rather than actors, plus a short-lived series per replaced pod. A
collector can drop the pod attributes if that matters.

#### Breaking change

The stdout usage record moves from atelet to the ateoms, with flat
`ate.*` keys and the message `Actor usage sampled`. Log filters on
atelet's `Actor usage sample` record stop matching: filter on
`jsonPayload.msg="Actor usage sampled"` and the `ate.*` keys instead.
Identity is no longer in the label group, so on GKE it does not land in
`LogEntry.labels`.

Part of #1748 (steps 4 to 6).

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

* ateom-gvisor logs a warning when a running actor's sandbox dies (#2258)

Part of #2212 (o11y side of #2211).

If the gVisor sandbox dies after start (sentry crash, kill, OOM kill),
nothing notices today. The actor stays RUNNING and its usage samples
keep reading like an idle actor's, because the pause cgroup leaf and its
counters outlive the processes in it.

This makes the death visible:

- `cgroupstats` also reads `populated` from `cgroup.events` and
`oom_kill` from `memory.events`. If either file is missing, it never
reads as a dead sandbox.
- The ateom's usage sweep (`sweepUsage`, every
`--usage-sample-interval`, 1m by default) logs one WARN per activation
when the leaf is empty. Since #2194, `GetActiveWorkloadStats` serves the
sweep's cached sample and reads no cgroup, so the check lives in the
sweep, the only periodic cgroup read; `measure` hands back the raw
`cgroupstats.Sample` for it. The record uses `ateattr.ActorLogAttrs`
plus `ate.sandbox.oom_kills`, so filtering on `ate.actor.uid` finds it
with the rest of the actor's logs.
- Actors with a lifecycle RPC holding or waiting on their lock are
skipped, since checkpoint and terminate empty the cgroup on purpose.
`actorlock` gets a non-blocking `Busy` for this. An activation whose
final record is written (checkpoint, terminate, drain) is no longer
swept at all.
- `readKeyedUint` walks lines in place instead of using a
`bufio.Scanner`, which allocated 4 KiB per file. With the two extra
reads, reading 1000 actors' cgroups takes about 9% longer than before
and allocates half the bytes.

It does not change actor state, RPC responses or the stats samples.
Moving the actor to CRASHED is #2211. It also does not catch an app that
exits while the sentry stays up.

Tested on kind and on GKE (substrate-relay) with the branch image on the
egress pool, before the rebase onto #2194 (the check then ran in
`GetActiveWorkloadStats`). Unit tests pass at the rebased head; live
re-verification on kind is pending:

| Case | kind | GKE |
|---|---|---|
| OOM kill via `memory.max` | 1 WARN, `oom_kills=3` | 1 WARN,
`oom_kills=1` |
| `kill -9` the sentry | 1 WARN, `oom_kills=0` | 1 WARN, `oom_kills=0` |
| Normal suspend | none | none |
| Suspend then resume | none | none |
| Idle actors | none (5) | none (3) |

Each warning showed up on the first sweep after the fault and was not
repeated. On GKE it lands in Cloud Logging as severity WARNING with the
`ate.*` fields. The stats pipeline was unchanged.

One caveat: `oom_kill` counts over the cgroup's lifetime, so a leaf left
behind by an earlier activation of the same actor (#2107) carries its
count forward.

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

* Update Snapshots API to support Actor Lifecycle v2 (#2289)

Partially implements the proposed API changes in:
https://docs.google.com/document/d/1lQ6fhrRb2sLCeFrvhsfytaoKVbA8X4w0TEwj84pr8uU/edit?tab=t.0

Depends on (and temporily includes)
https://github.com/agent-substrate/substrate/pull/2222, review that one
first.

Part of #2069

This PR tries not to make major changes to existing semantics. They will
be done as a followup.

Other things in the proposal that are not implemented in this PR
(tracked separately)
- hardware information
(https://github.com/agent-substrate/substrate/issues/2246)
- sandbox information
- combining pause/resume and introducing detach
(https://github.com/agent-substrate/substrate/issues/798)


#### Breaking change

The Snapshots API has been completely rewritten and replaces
- ActorStatus in_progress_snapshot_uri, external_snapshot,
local_snapshot, in_progress_local_snapshot_name fields.
- TagStatus snapshot, storage_location fields


----
- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR

* benchmarking/analysis: aggregate suspend/resume phase breakdown log r… (#2009)

## benchmarking/analysis: aggregate suspend/resume phase breakdown log
records

### Why

#1941 makes atelet and ateom-microvm write one structured `Checkpoint
timing breakdown` / `Restore timing breakdown` record per operation per
layer, so that `SuspendActor` / `ResumeActor` latency can be attributed
to the phases inside it. A log line is not a benchmark result, though:
something has to get those records off the cluster and turn them into
percentiles. Today nothing in `benchmarking/` reads node-side logs — the
runner keeps its own `logs.txt`/`traces.txt` and the orchestrator tails
the locust job. This PR adds that consumer, so a run can answer "where
did the 6 s go?" from data rather than from one anecdote.

### What

Everything lives under `benchmarking/analysis/`; no production code
changes, and **no new log output** — this only reads what the binaries
already write.

- **`collect_logs.sh`** — runs `kubectl logs` for every atelet pod
(`ate-system`) and every worker pod (`benchmark-workloads`) for the
run's window and writes them to `--dest`, next to the locust artifacts.
Two records per lifecycle operation is the entire volume it collects.
- **`phase_report.py`** — parses the records from those dumps (tolerant
of `kubectl` line prefixes and unrelated lines) and prints:
- **Phase percentiles** per layer, operation, scope and phase (count,
p50, p90, p95, max), with the atelet `ateom_checkpoint` /
`ateom_restore` bucket split further by the ateom rows.
- An **`unattributed`** row for the two checkpoint layers: the total
minus what the logged phases account for, counting the three concurrent
captures once. It is the time the instrumentation does not yet name.
- **Waterfalls** of the slowest N operations, with the ateom record
nested under the atelet phase it decomposes and the gap between the two
(RPC and queueing between the layers).
- Failed operations are excluded from the percentiles via the
`error.type` marker on the record.
  - `--csv` writes `phase_percentiles.csv` for run-over-run comparison.
- **`test_phase_report.py`** — 10 unittest tests covering parsing of
both layers, nanosecond timestamps, prefixed lines, failure exclusion,
the residual arithmetic and the waterfall join.
- **`README.md`** — the record shapes, how to collect and aggregate, how
to read each section and its concurrency caveats.
`benchmarking/README.md` gains a short section pointing at it.
- docs/dev/suspend-resume-phase-breakdown.md — an end-to-end runbook for
a dev cluster: deploy the benchmark workloads on a micro-VM pool, drive
glutton suspend/resume load with a sized working set, collect (kubectl
or Cloud Logging), report, read the tables, and the failure modes that
come up in practice. Linked from CONTRIBUTING.md beside the local
micro-VM guide.

### Scope and follow-ups

- *Standalone tool*. Nothing invokes it yet.
benchmarking/automation/orchestrator.py deletes the workload and
ate-system pods right after each test, so the collection has to run
before that teardown. The follow-up is to call collect_logs.sh and
phase_report.py from run_test() before the teardown and write the phase
percentiles into the run's --dest as stats.jsonl rows (the suite's
existing result format: timestamp, tag, test_name, metric,
measurements), so they travel with the rest of the run's artifacts.
 
 
**Follow-up (#2148):** the automated runs will consume this through the
locust runner, which reads the pod logs itself before the orchestrator's
teardown and appends the percentiles to stats.jsonl — that lands in a
separate PR stacked on this one (branch phase-logs-runner; PR link to
follow), so this PR stays the tool and its docs.

* Reject requests with duplicate ate-target-actor headers across HTTP and CONNECT (#2308)

Reject ingress requests that carry multiple `ate-target-actor` header
entries across HTTP (`404 Not Found`), `CONNECT`
(`%CEL(request.headers['ate-target-actor'])%` filter state + outer
`CONNECT` precedence), and `atunnel` (`421 Misdirected Request`),
preventing header smuggling mismatches between upstream Envoy RBAC/CEL
authorization policies and `ext_proc` actor routing.

Fixes #2307

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

* docs: add substrate security guide and principles (#2326)

Add Substrate Security documentation covering security philosophy,
scope, extensibility, and defensible multi-tenancy principles across
identity, storage, networking, authorization, and auditability.

Update README.md to reference the new security guide.

* Rename some `ExternalVolume` fields (#2404)

Naming the field `volume_name` is redundant. Rename to to `name`.

Also rename `actor_volumes` to `external_volumes`.

Part of #1378

* kubectl-ate: add --source to select container or lifecycle log lines (#2337)

Part of https://github.com/agent-substrate/substrate/issues/294. Takes
over the half that #1067 left open and supersedes the stale #311.

## Summary

`kubectl ate logs actors` gains `--source` to pick a class of lines from
an actor's stream:

| Flag | Shows |
|---|---|
| `--source=all` (default) | everything, unchanged |
| `--source=containers` | every container's output with the lifecycle
events removed |
| `--source=lifecycle` | only the lifecycle events (`Actor
starting/started`, `checkpointing/checkpointed`, `restoring/restored`,
`terminated`) |
| `-c NAME` | unchanged; implies container output, so it combines with
`--source=containers` and is rejected with `--source=lifecycle` |

The selection is client-side, as the issue intended: lifecycle events
are the lines ateom emits with no `ate.actor.container.name` label
(`ActorLogLabels` omits the key when the container name is empty). Named
"lifecycle" rather than "supervisor" or "ateom" because the code and
docs already use that word for these events and it leaks no
implementation detail.

## Testing

**Unit** (`go test ./cmd/kubectl-ate/...`): new filter table cases for
every source, `TestNewLogLineFilter` for the flag validation, and
`TestLogsActorRunner_Run_OneShot_SourceFilter` covering each source
against an interleaved stream plus the two rejected combinations, which
must fail before any logs are fetched.

**End to end** on a kind cluster (`kind-ate-dev`) against the counter
demo actor, using the plugin built from this branch.

One-shot. Lines starting with `#` are annotations added for review; `$`
lines are the commands as run. Two commands are negative cases and are
**expected** to exit 1.

```
# The main commit this branch was built on; the plugin binary under test was built from the branch with make build-atectl.
$ git rev-parse --short HEAD
df788825

# Precondition: the actor is running, so its current worker's log stream is available.
$ kubectl ate get actors -a ate-demo-counter
ATESPACE           NAME          TEMPLATE                   STATE                 WORKER POD                                  WORKER IPS    VERSION   AGE
ate-demo-counter   learner-one   ate-demo-counter/counter   ACTOR_STATE_RUNNING   ate-demo-counter/counter-79b587778f-8lh4h   10.244.0.22   19        14h

# Expected: only lifecycle events, no container output. Each suspend/resume cycle on this worker shows as a checkpointing/checkpointed or restoring/restored pair.
$ kubectl ate logs actors learner-one -a ate-demo-counter --source=lifecycle
{"time":"2026-10-07T21:31:23.367965976Z","message":"Actor restoring","span_id":"31bf9ef08d13d4d6","trace_flags":"01","trace_id":"61eb2ac56531db2b0308c31c564e6379"}
{"time":"2026-10-07T21:31:23.550431306Z","message":"Actor restored","span_id":"31bf9ef08d13d4d6","trace_flags":"01","trace_id":"61eb2ac56531db2b0308c31c564e6379"}
{"time":"2026-10-07T21:31:48.648599369Z","message":"Actor checkpointing","span_id":"c475c334c8df2dc4","trace_flags":"01","trace_id":"e617b6130fcc28674cc9899ec702e596"}
{"time":"2026-10-07T21:31:48.791435032Z","message":"Actor checkpointed","span_id":"c475c334c8df2dc4","trace_flags":"01","trace_id":"e617b6130fcc28674cc9899ec702e596"}
{"time":"2026-10-07T21:31:59.772566601Z","message":"Actor restoring","span_id":"a68c5afa10e86252","trace_flags":"01","trace_id":"b387404020eaca71d604b8f0c81d43c1"}
{"time":"2026-10-07T21:31:59.96129419Z","message":"Actor restored","span_id":"a68c5afa10e86252","trace_flags":"01","trace_id":"b387404020eaca71d604b8f0c81d43c1"}
{"time":"2026-10-07T21:38:25.962753328Z","message":"Actor checkpointing","span_id":"e181224bf8792b12","trace_flags":"01","trace_id":"52963b1666af63486d765bc1527fc8bc"}
{"time":"2026-10-07T21:38:26.096296577Z","message":"Actor checkpointed","span_id":"e181224bf8792b12","trace_flags":"01","trace_id":"52963b1666af63486d765bc1527fc8bc"}
{"time":"2026-10-07T21:43:14.718178039Z","message":"Actor restoring","span_id":"718a8f92f8735ed3","trace_flags":"00","trace_id":"42123dfac30d07fa86164199ecdbf0ac"}
{"time":"2026-10-07T21:43:14.920089778Z","message":"Actor restored","span_id":"718a8f92f8735ed3","trace_flags":"00","trace_id":"42123dfac30d07fa86164199ecdbf0ac"}
{"time":"2026-10-07T21:44:04.990909876Z","message":"Actor checkpointing","span_id":"107b6f12dd7f0e8b","trace_flags":"01","trace_id":"0bb09c29559dd8cfedb20f44a24bda4a"}
{"time":"2026-10-07T21:44:05.134534519Z","message":"Actor checkpointed","span_id":"107b6f12dd7f0e8b","trace_flags":"01","trace_id":"0bb09c29559dd8cfedb20f44a24bda4a"}
{"time":"2026-10-07T21:44:07.73397801Z","message":"Actor restoring","span_id":"7d062e8b5207afe9","trace_flags":"00","trace_id":"42f6d2926dd56687c222b1de832a072e"}
{"time":"2026-10-07T21:44:07.978274791Z","message":"Actor restored","span_id":"7d062e8b5207afe9","trace_flags":"00","trace_id":"42f6d2926dd56687c222b1de832a072e"}
{"time":"2026-10-07T21:44:11.552735135Z","message":"Actor checkpointing","span_id":"86954a4e96fbf250","trace_flags":"01","trace_id":"fb7199b8ee3491dcf561a25a8f60b1df"}
{"time":"2026-10-07T21:44:11.69238189Z","message":"Actor checkpointed","span_id":"86954a4e96fbf250","trace_flags":"01","trace_id":"fb7199b8ee3491dcf561a25a8f60b1df"}
{"time":"2026-10-07T21:44:13.26693018Z","message":"Actor restoring","span_id":"a896d4c30c7b664a","trace_flags":"00","trace_id":"607e6dbd891d7faa115e061dc4f473dc"}
{"time":"2026-10-07T21:44:13.516304563Z","message":"Actor restored","span_id":"a896d4c30c7b664a","trace_flags":"00","trace_id":"607e6dbd891d7faa115e061dc4f473dc"}
[exit 0]

# Expected: default output is unchanged (every line). Baseline for the two counts below.
$ kubectl ate logs actors learner-one -a ate-demo-counter | wc -l
5484

# Expected: baseline minus the lifecycle events.
$ kubectl ate logs actors learner-one -a ate-demo-counter --source=containers | wc -l
5466

# Expected: 0, i.e. no lifecycle event survives --source=containers.
$ kubectl ate logs actors learner-one -a ate-demo-counter --source=containers | grep -c '"message":"Actor '
0

# Expected: containers + lifecycle = baseline (5466 + 18 = 5484).
$ kubectl ate logs actors learner-one -a ate-demo-counter --source=lifecycle | wc -l
18

# Expected: same as --source=containers, since this actor has a single container (off by one only because the actor logs a line every 10s).
$ kubectl ate logs actors learner-one -a ate-demo-counter -c counter | wc -l
5467

# Expected: ordinary application output, unchanged in shape.
$ kubectl ate logs actors learner-one -a ate-demo-counter --source=containers | tail -3
{"time":"2026-10-08T12:22:35.319622976Z","count":5322,"fshash":"dorEZDUO6NCvAZmlU346thAgz9cSxuVYFZ2yOLq8rqg","level":"INFO","msg":"Count"}
{"time":"2026-10-08T12:22:45.322833947Z","count":5323,"fshash":"dorEZDUO6NCvAZmlU346thAgz9cSxuVYFZ2yOLq8rqg","level":"INFO","msg":"Count"}
{"time":"2026-10-08T12:22:55.316121067Z","count":5324,"fshash":"dorEZDUO6NCvAZmlU346thAgz9cSxuVYFZ2yOLq8rqg","level":"INFO","msg":"Count"}

# NEGATIVE CASE, error is expected: -c and --source=lifecycle can never match the same line, so the CLI rejects the combination with exit 1 before fetching any logs.
$ kubectl ate logs actors learner-one -a ate-demo-counter -c counter --source=lifecycle
Error: --container cannot be combined with --source=lifecycle: lifecycle events are not emitted by a container
[exit 1]

# NEGATIVE CASE, error is expected: an unknown --source value is rejected with exit 1 and the list of valid values.
$ kubectl ate logs actors learner-one -a ate-demo-counter --source=supervisor
Error: invalid --source "supervisor": must be one of all, containers, lifecycle
[exit 1]

# Expected: the new flag is documented in --help next to --container.
$ kubectl ate logs actors --help | grep -A1 -E -- "--source|--container"
  -c, --container string   Show only logs from this container.
  -f, --follow             Specify if the logs should be streamed.
--
      --source string      Which lines to show: all, containers (every container's output, no lifecycle events), or lifecycle (only the actor's lifecycle events). (default "all")
```

Follow mode, with a suspend and a resume issued from another shell while
`-f --source=lifecycle` was streaming. The stream shows only the
lifecycle pair for each transition and reconnects once the actor is
running again (on the same worker pod here, since the kind cluster has
one).

```
$ kubectl ate logs actors learner-one -a ate-demo-counter -f --source=lifecycle   # started, then suspend + resume in another shell
Actor is currently running on pod ate-demo-counter/counter-79b587778f-8lh4h
{"time":"2026-10-08T12:25:08.096184297Z","message":"Actor checkpointing","span_id":"b8dd6fc306a3ccf9","trace_flags":"01","trace_id":"45cae2a35d684ba8f1d6305c09333eb3"}
{"time":"2026-10-08T12:25:08.291926175Z","message":"Actor checkpointed","span_id":"b8dd6fc306a3ccf9","trace_flags":"01","trace_id":"45cae2a35d684ba8f1d6305c09333eb3"}
{"time":"2026-10-08T12:25:08.096184297Z","message":"Actor checkpointing","span_id":"b8dd6fc306a3ccf9","trace_flags":"01","trace_id":"45cae2a35d684ba8f1d6305c09333eb3"}
{"time":"2026-10-08T12:25:08.291926175Z","message":"Actor checkpointed","span_id":"b8dd6fc306a3ccf9","trace_flags":"01","trace_id":"45cae2a35d684ba8f1d6305c09333eb3"}
{"time":"2026-10-08T12:25:33.660662951Z","message":"Actor restoring","span_id":"d01fcf2c7f734ff2","trace_flags":"01","trace_id":"dfae910c43670918c5ab1cb3354863e3"}
{"time":"2026-10-08T12:25:33.968364046Z","message":"Actor restored","span_id":"d01fcf2c7f734ff2","trace_flags":"01","trace_id":"dfae910c43670918c5ab1cb3354863e3"}
[follow exited 124]
```

```
[12:25:07Z] $ kubectl ate suspend actor learner-one -a ate-demo-counter
ATESPACE           NAME          TEMPLATE                   STATE                   WORKER POD   WORKER IPS   VERSION   AGE
ate-demo-counter   learner-one   ate-demo-counter/counter   ACTOR_STATE_SUSPENDED   <none>                    21        14h
[12:25:33Z] $ kubectl ate resume actor learner-one -a ate-demo-counter
ATESPACE           NAME          TEMPLATE                   STATE                 WORKER POD                                  WORKER IPS    VERSION   AGE
ate-demo-counter   learner-one   ate-demo-counter/counter   ACTOR_STATE_RUNNING   ate-demo-counter/counter-79b587778f-8lh4h   10.244.0.22   23        14h
[12:26:04Z] $ kubectl ate get actors -a ate-demo-counter
ATESPACE           NAME          TEMPLATE                   STATE                 WORKER POD                                  WORKER IPS    VERSION   AGE
ate-demo-counter   learner-one   ate-demo-counter/counter   ACTOR_STATE_RUNNING   ate-demo-counter/counter-79b587778f-8lh4h   10.244.0.22   23        14h
```

The repeated `checkpointing`/`checkpointed` pair after the suspend is
the existing follow-mode reconnect replay (the resume cursor is a
second-granularity `sinceTime`); it is unchanged by this PR.

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

* Egress policies can inject actor JWTs (#2141)

Part of #1660.

- `actor_jwt` entries in `replace_headers` are now injected on `http`
and `https` rules alike, like `credential_uri` entries since #2158, when
the request carries the header, instead of being denied with 501. The
gateway mints a token for the actor behind the tunnel, with the policy's
audiences and lifetime, and writes the prefix plus the token into the
header.
- Actor JWTs need no credential provider, so a gateway without one still
injects them. A `credential_uri` entry still denies the request when no
provider is configured.
- A failed mint denies the request: 403 if the actor was deleted, 503 if
ateapi is unavailable or the mint timed out, and 500 otherwise.
- The `credential_uri` path moved into its own function. Each injection
failure is wrapped with its header, and the credential URI for provider
entries, so the router's log line says which injection denied the
request.

With `--experimental-enable-authz` set on ateapi, the gateway can only
mint once #2026 lands.

Testing: unit tests cover injection with no provider, the skip when the
header is absent, and each denial on both legs. The e2e check is in
#2256, stacked on this PR.

- [x] Tests pass
- [ ] Appropriate changes to documentation are included in the PR (the
user guide lands later in the #1660 series)

* More robust Actor DNS Relay (#2370)

* Reduces go-routine and socket usage per Actor
* Moves limits to per-Actor instead of across all Actors (noisy
neighbor)
* Clean-up of logic that can support later extension if needed.

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

* ate-setup sets the actor JWT issuer (ACTOR_JWT_ISSUER), and ate-api-server requires it (#2257)

Part of #1756.

- ate-api-server's `--actor-jwt-issuer` is now required instead of
defaulting to `https://idp.<pod namespace>.svc`, so the issuer comes
from one setting that ate-idp-server can share rather than from two
binaries' defaults agreeing.
- ate-setup takes the issuer from a new `actorJWT.issuer` setting
(`ACTOR_JWT_ISSUER`, `--actor-jwt-issuer`, or the config file),
defaulting to `https://idp.<namespace>.svc`, and validates it before
touching the …

* cmd/atenet: Remove the router's "all" mode (agent-substrate#2441)

All of our in-tree deployments explicitly deploy the router in either
ingress or egress mode.

Attempting to one run router instance that handles both ingress and
egress seems pretty confusing, so we should get rid of the flag that
lets us do so. In the future, we might want to split the ingress router
and the egress router into separate atenet subcommands (or even separate
binaries).

* test: Fix macos unit test failures (agent-substrate#2429)

# Description

`make test` failed on macOS for two host-specific reasons. The DNS relay
referenced Linux-only network namespace APIs from an otherwise portable
file,
and the object storage tests inherited exported `AWS_PROFILE` and
`AWS_DEFAULT_PROFILE` values from the developer's shell.

Move only the namespace-serving method behind a Linux build tag so
portable
DNS tests still run on macOS. Clear both AWS profile selectors in the
object
storage test so local AWS configuration cannot affect it.

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

---------

Signed-off-by: timflannagan <[email protected]>

* Update x/net to v0.60.0 to fix govulncheck failures (agent-substrate#2401)

Both modules in the post-merge govulncheck matrix resolve
golang.org/x/net v0.58.0 and fail on five newly published HTTP/2
vulnerabilities. Update x/net to v0.60.0 in the root module and the GCP
Secret Manager plugin, include its required transitive dependency
updates, and regenerate vendor.

Go also updates the required x/crypto, x/sync, x/sys, x/term, and x/text
versions. All first-party changes are in the two go.mod/go.sum pairs;
the other changes are generated vendor updates.

Affected advisories: GO-2026-6603, GO-2026-6610, GO-2026-6611,
GO-2026-6612, and [GO-2026-6617](https://pkg.go.dev/vuln/GO-2026-6617).

The [latest failing main
run](https://github.com/agent-substrate/substrate/actions/runs/37911293778)
already selects Go 1.27.2; its remaining failures are all in x/net. The
earlier standard-library findings were resolved by the workflow
selecting the patched stable toolchain. This change preserves the
current Go version requirement.

Validation after merging main `59072f7b`:

- Linux/amd64 with Go 1.27.1: `go test -race -count=1 ./...` passed with
PostgreSQL/envtest integration enabled (`CI=true`).
- All checks in `bash hack/verify-all.sh` passed, including
generated-code and vendor consistency, and the checkout remained clean.
- GCP Secret Manager plugin race tests passed.
- govulncheck v1.8.0 with Go 1.27.2: both root and plugin modules passed
with exit 0, zero vulnerable imported packages, and zero reachable
findings. Both report GO-2026-5932 only at module level in unimported
x/crypto code.

The merge retains main's direct x/net dependency classification at
v0.60.0. Canonical v0.60.0 DNS parser content was restored in vendor
after the generation check found that the automatic merge had retained
three older statements.

Verification ran as an unprivileged user. Root-gated mount tests and
real cluster E2E were not run for this head.


AI assistance was used to resolve the merge conflicts and run
verification.

* ROOTFS snapshot fidelity: micro-VM capture and restore, API opened for micro-VM templates (agent-substrate#2448)

Step III of agent-substrate#2102, tracked in agent-substrate#2447. Step I (agent-substrate#2309) removed `on_pause`;
step II (agent-substrate#2364) introduced the `SnapshotFidelity` ladder with ROOTFS
defined but rejected everywhere.

**ROOTFS becomes real on micro-VM.** A ROOTFS snapshot holds the durable
volumes plus each container's root-filesystem changes, without guest
memory: the actor cold-boots on resume but keeps what it wrote to its
rootfs. gVisor keeps rootfs changes inside its memory checkpoint and
cannot serve it yet; it now declares that explicitly and a gVisor
template that asks for ROOTFS is rejected at admission.

Four commits, meant to be read in order. Each builds and passes tests on
its own, and ROOTFS stays unreachable from the public API until the
third.

## 1. atelet: forward ROOTFS and stop narrowing paused uploads

Atelet accepts every level of the ladder and forwards it; which levels a
runtime serves is the runtime's knowledge. The paused-upload path still
converted a MEMORY pause into a VOLUMES upload, a leftover from the days
of separate pause and commit scopes. Pause and suspend now capture one
fidelity, a paused actor's template cannot change, and golden actors are
never paused, so captured and requested fidelity are always equal; the
conversion is replaced by an equality check that refuses any mismatch as
drift.

## 2. ateom-microvm: capture and restore ROOTFS

Every piece already existed: MEMORY checkpoints tar the host-side rootfs
upper per container alongside the Cloud Hypervisor snapshot, and MEMORY
restores untar those uppers before assembling the overlays. ROOTFS is
the same checkpoint without the VM snapshot, and the upper restore
followed by the VOLUMES cold-boot path with a new `keepRootfsUpper` boot
parameter so the cold boot does not wipe the restored upper (safe across
its retry). Only a MEMORY restore resumes the guest's CPU counters;
ROOTFS counts from zero like VOLUMES.

`resources.ValidateSnapshotFidelity` now takes the runtime's supported
set: micro-VM passes all three, gVisor passes VOLUMES and MEMORY and
rejects ROOTFS before taking the actor lock, naming the levels it
serves.

## 3. ateapi: allow ROOTFS on micro-VM templates

The field-level rejection moves to the template-level cross-field hook:
ROOTFS requires `sandbox_config.sandbox_class: SANDBOX_CLASS_MICROVM`.
An actor can only be repointed to a template with an identical sandbox
config, so a ROOTFS template never ends up under a gVisor actor. The
control plane needs no logic change: a ROOTFS snapshot restores at
ROOTFS, and a repointed actor drops to VOLUMES as it does for MEMORY,
since a rootfs upper is only valid over the image it was written on.

## 4. e2e: prove it on the counter demo

The counter app gains a third counter in a file on its root filesystem,
printed in the response, so a ROOTFS snapshot is distinguishable from
VOLUMES. All lifecycle rows state their rootfs expectation (MEMORY and
ROOTFS keep it, VOLUMES resets it). New micro-VM-only ROOTFS rows cover
pause, suspend, suspend from PAUSED, and the template repoint and
revert. On the gVisor CI lane the ROOTFS rows skip; the new rootfs
assertions on MEMORY and VOLUMES rows run there as a side check.

## Behavior changes

- **API**: `SNAPSHOT_FIDELITY_ROOTFS` is accepted on micro-VM templates
and rejected on gVisor ones with a message naming the requirement. No
field or wire changes; the `preferred_fidelity` field-level custom
validation tag is removed.
- **Atelet**: a paused-upload whose desired fidelity differs from the
captured one is now `FailedPrecondition`; previously a MEMORY capture
could be downgraded to VOLUMES. Unreachable in practice, as explained in
commit 1.
- Unchanged: MEMORY and VOLUMES capture and restore, the golden-actor
MEMORY override, metric labels (`rootfs` was already a member).
- No backward compatibility is kept, per the pre-v1.0 policy.

## Not in this PR

`required_fidelity`, `ResumeActorResponse.resumed_fidelity`, the resume
fallback rule, gVisor ROOTFS (pending the gVisor team's separable rootfs
capture), and restoring a MEMORY snapshot as a ROOTFS cold boot (the
artifacts already allow it; atelet would need the rootfs file subset in
the manifest to skip the memory image).

## Verification

- Unit tests for atelet, ateom-microvm, ateom-gvisor, resources,
ateattr, ateomphaselog, ateapi (validation, controlapi, functional),
kubectl-ate, and apitool pass; the ateom tests are linux-tagged and were
type-checked for linux locally.
- golangci-lint, kube-api-linter, python-protos, boilerplate pass;
codegen shows no drift.
- The micro-VM counter demo was run manually on a GKE cluster with a
nested-virtualization node pool and behaved as expected. The ROOTFS e2e
rows need a micro-VM lane; the gVisor lane runs the rest.

- [x] Breaking change
- [x] Not backward compatible

🤖 Generated with [Claude Code](https://claude.com/claude-code)

* fix(kubectl-ate): validate atespace flags before connecting (agent-substrate#2464)

`get actors` and `get actor-template` now reject missing or conflicting
atespace flags before loading kubeconfig or connecting to the API.
Previously, a kubeconfig or connection error could hide the argument
error.

The tests use a missing kubeconfig to check that invalid flags are
rejected first and valid queries still reach client initialization.

Refs agent-substrate#2463

- [x] Tests pass: `go test -race ./cmd/kubectl-ate/...`
- [x] Full Go suite passes: `env -u NO_COLOR go test -race -p 1 ./...`
- [x] Lint passes: `hack/verify/golangci-lint.sh`

The first full-suite run failed in the CPU timing test and in
color-output tests with `NO_COLOR=1`. The serial run above passed.

The Python license and proto checks could not finish because `pypi.org`
DNS lookup failed. The remaining repository verifiers passed; metrics
and shellcheck used local binaries at the versions pinned by the
scripts.

* Allow ActorTemplates to disable golden snapshot creation (agent-substrate#2328)

ActorTemplates can opt out of automated golden snapshot creation with
`snapshotConfig.goldenSnapshotConfig.mode:
GOLDEN_SNAPSHOT_MODE_DISABLED`, avoiding unused golden actors and
snapshots. Omitting the configuration or leaving the mode `UNSPECIFIED`
defaults to `ENABLED`. The mode is an enum with declarative validation,
following the existing snapshot fidelity pattern.

Disabled templates are skipped during queue resync and reconciliation.
Their regular actors can cold boot and suspend/resume using their own
snapshots. `kubectl ate get actor-templates` displays `DISABLED` in the
`GOLDEN TAG` column. The API guide, field table, glossary, and CLI
documentation describe the opt-out.

The functional regression starts a real `ActorTemplateReconciler`. An
enabled control template proves it is making progress while the disabled
template's golden actor and tag both remain absent. The test also
verifies the disabled template's actor cold-boots, creates its own
durable snapshot on suspend, and restores that snapshot on resume. The
disabled reconciliation case is part of the existing table-driven unit
test.

Implements the golden snapshot opt-out feature discussed in agent-substrate#2068.

AI assistance: Codex helped implement and test these changes.

### Validation

- macOS arm64: race and coverage tests passed for defaulting, API
validation, control-plane unit tests, and CLI printing.
- The CLI table regression failed before adding the disabled state and
passed after the change.
- Linux amd64, Go 1.27.1: `go test -race ./...` passed for all 147
packages with tests.
- With `CI=true`, fresh race tests for control-plane functional tests
and PostgreSQL storage passed: 612 tests/subtests, zero failures and
zero skips.
- Removing both disabled-mode guards made the lifecycle regression fail
because it found the disabled template's golden actor. The complete
implementation passed.
- `hack/verify-all.sh`: all 14 verifiers passed with unmodified
settings. The first attempt hit a linter package-loading timeout; a
longer standalone lint run reported `0 issues`, and the standard
verifier run then passed after the cache warmed.
- Linux builds of `ateapi` and `kubectl-ate` passed.

Functional tests use real PostgreSQL and an envtest Kubernetes API with
a fake atelet. Real gVisor E2E was not run.

- [x] Appropriate changes to documentation are included in the PR.
- [x] Race tests, repository verifiers, and affected binaries were
checked.

---------

Co-authored-by: cloudonly <[email protected]>

* atunnel: strip X-Ate-Assignment-Stale from upstream actor responses (agent-substrate#2410)

### Motivation

`atunnel` defines `StaleAssignmentHeader = "X-Ate-Assignment-Stale"` to
distinguish an `atunnel` infrastructure-level routing rejection (HTTP
421 `Misdirected Request`, returned when an actor is no longer active on
that worker) from an HTTP 421 returned by the guest actor application
itself (`internal/atunnel/ingress.go#L45-L47`).

While `newActorProxy` already strips internal Substrate headers
(`ate-target-port`, `ate-target-actor`) on the request leg into the
sandbox, it lacked response sanitization on the return leg. As a result,
an untrusted actor workload running inside the container could return
`X-Ate-Assignment-Stale: true` and spoof an infrastructure routing
rejection to downstream routers or clients.
### Changes
1. **`internal/atunnel/ingress.go`**: Added a `ModifyResponse` hook to
`newActorProxy()` that removes `X-Ate-Assignment-Stale` from any
response received from the actor upstream.
2. **`internal/atunnel/ingress_test.go`**: Added unit test
`TestServeHTTPStripsStaleAssignmentHeaderFromActor` asserting that even
if the actor application returns `X-Ate-Assignment-Stale: true`, the
header is stripped before the response reaches the client.
### Testing
- `go test -race -count=1 ./internal/atunnel/...`
- `./hack/verify/boilerplate.sh`
- `./hack/verify/gofmt.sh`
- `./hack/verify/golangci-lint.sh`
> The PR title becomes the release note. Write it for users: what
changed for them, not how the code changed.

AI was used to assist in the development of these changes.

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

* WorkerPools resolve their SandboxConfig and protect it from deletion (agent-substrate#2393)

Part of agent-substrate#587. Step 3 of the SandboxConfig management design, after agent-substrate#2332
and agent-substrate#2333.

Each WorkerPool now resolves one SandboxConfig per sandbox class,
reports it in status, and protects it from being deleted while in use.
Workers do not read the resolved config yet; that is the next step.

## How to review

The PR is three commits. Each one builds and passes its tests on its
own, so they can be reviewed in order.

1. **Resolve each WorkerPool's SandboxConfig into status.** The API
change, the resolution rules, and the `Resolved` condition. No events,
no finalizer.
2. **Protect SandboxConfigs in use with a finalizer.** The finalizer,
the reconciler that releases it, the two-step status write for a switch,
and the `ate-setup` and `install-microvm-deps.sh` changes that deal with
the finalizer.
3. **Emit events for a WorkerPool's SandboxConfig and add e2e tests.**
The Warning and `SandboxConfigSwitched` events, and an e2e suite that
walks a pool through every move.

## Resolution

Without `configRef`: the pool uses the class default.

```yaml
apiVersion: ate.dev/v1alpha1
kind: WorkerPool
metadata:
  name: counter-pool
  namespace: ate-demo
  generation: 1
spec:
  replicas: 2
  workerImage: ateom-gvisor:v1
  sandboxClasses:
  - name: gvisor                  # no configRef
status:
  replicas: 2
  readyReplicas: 2
  selector: ate.dev/worker-pool=counter-pool
  sandboxClasses:
  - name: gvisor
    configRef:
      name: gvisor-default        # newest gvisor config annotated is-class-default
    conditions:
    - type: Resolved
      status: "True"
      reason: Resolved
      message: resolved to SandboxConfig "gvisor-default"
      observedGeneration: 1
```

With `configRef` pointing at a config that does not exist: the pool
keeps the config it resolved last time.

```yaml
apiVersion: ate.dev/v1alpha1
kind: WorkerPool
metadata:
  name: pinned-pool
  namespace: ate-demo
  generation: 2
spec:
  replicas: 2
  workerImage: ateom-gvisor:v1
  sandboxClasses:
  - name: gvisor
    configRef:
      name: gvisor-release2       # does not exist
status:
  replicas: 2
  readyReplicas: 2
  selector: ate.dev/worker-pool=pinned-pool
  sandboxClasses:
  - name: gvisor
    configRef:
      name: gvisor-release1       # still in use, still protected
    conditions:
    - type: Resolved
      status: "False"
      reason: SandboxConfigNotFound
      message: SandboxConfig "gvisor-release2" not found; keeping "gvisor-release1"
      observedGeneration: 2
```

- With `configRef`: that config is used. It must exist, belong to the
same class, and not be in the process of being deleted.
- Without `configRef`: the newest config of the class that is annotated
`sandboxconfig.ate.dev/is-class-default: "true"` and is not being
deleted is used.
- If no config can be resolved, the status `configRef` keeps the config
resolved last time, and the `Resolved` condition is `False` with reason
`SandboxConfigNotFound`, `SandboxClassMismatch`,
`SandboxConfigDeleting`, or `DefaultConfigNotFound`. A Warning event is
emitted when the reason changes.
- `kubectl get workerpool` gains `SANDBOXCONFIG` and `RESOLVED` columns.
- The counter demo no longer pins its pool to `gvisor-default`. It
follows the class default.

## Switching configs

When a class switches to a different config, the controller writes
status twice. First it records the new config in
`status.sandboxClasses[].pendingConfigRef`, which protects the new
config while the current one stays in `configRef`. Then it reads the new
config from the API server. If the config exists, belongs to the class,
and is not being deleted, `configRef` moves to it. Otherwise the class
keeps the current config and `Resolved` becomes `False`. This prevents a
pool from recording a config that was deleted between the cache read and
the status write.

## Events

Each time a class's `configRef` changes, the controller emits a Normal
event with reason `SandboxConfigSwitched`. The event is emitted after
the status write that records the change, so each switch shows up once.
A reconcile that leaves the config unchanged emits nothing, so the
pool's events show the configs it has used, in order. A class that fails
to resolve emits a Warning event with the condition reason.

To see the history for one pool:

```
kubectl -n ate-demo events --for workerpool/counter-pool
```

Example for the counter pool, which has no `configRef` and so follows
the class default. It first resolved to `gvisor-default`. Then a newer
config `gvisor-default-v2` annotated `is-class-default: "true"` was
created, and the pool switched to it on its own. Then
`gvisor-default-v2` lost the annotation, got it back, and was finally
deleted. A config that is being deleted is no longer a class default, so
the pool moved back to `gvisor-default`, and the finalizer on
`gvisor-default-v2` was released once no pool used it.

```
$ kubectl -n ate-demo-counter events --for workerpool/counter
LAST SEEN   TYPE     REASON                  OBJECT               MESSAGE
61s         Normal   SandboxConfigSwitched   WorkerPool/counter   sandbox class "gvisor" uses SandboxConfig "gvisor-default"
53s         Normal   SandboxConfigSwitched   WorkerPool/counter   sandbox class "gvisor" switched from SandboxConfig "gvisor-default" to "gvisor-default-v2"
46s         Normal   SandboxConfigSwitched   WorkerPool/counter   sandbox class "gvisor" switched from SandboxConfig "gvisor-default-v2" to "gvisor-default"
39s         Normal   SandboxConfigSwitched   WorkerPool/counter   sandbox class "gvisor" switched from SandboxConfig "gvisor-default" to "gvisor-default-v2"
33s         Normal   SandboxConfigSwitched   WorkerPool/counter   sandbox class "gvisor" switched from SandboxConfig "gvisor-default-v2" to "gvisor-default"
```

## Deletion protection

- Before recording a config in status, the WorkerPool controller adds
the finalizer `sandboxconfig.ate.dev/workerpool-protection` to it.
- A new `SandboxConfigProtectionReconciler` removes the finalizer once
no WorkerPool status references the config. It checks the cache first,
then confirms with a live list from the API server before removing it.
- A SandboxConfig that is being deleted is never resolved for a new
WorkerPool and is no longer treated as the class default. WorkerPools
that already use it keep using it, and because of the finalizer the
config is not actually deleted until every such WorkerPool has moved to
another config or has been deleted.

This is what the bundled `gvisor-default` looks like once a WorkerPool
has resolved it. Nothing else in the config changes.

```
kubectl get sandboxconfig gvisor-default -o yaml
```

```yaml
apiVersion: ate.dev/v1alpha1
kind: SandboxConfig
metadata:
  name: gvisor-default
  annotations:
    sandboxconfig.ate.dev/is-class-default: "true"
  finalizers:
  - sandboxconfig.ate.dev/workerpool-protection   # added by the WorkerPool controller
spec:
  sandboxClass: gvisor
  defaultVersion: v1
  versions:
  - name: v1
    state: Enabled
    ...
```

After `kubectl delete sandboxconfig gvisor-default` while a WorkerPool
still uses it, the delete does not finish. The config gets a
`deletionTimestamp` and stays in the cluster with the finalizer until
the last WorkerPool moves off it:

```yaml
apiVersion: ate.dev/v1alpha1
kind: SandboxConfig
metadata:
  name: gvisor-default
  annotations:
    sandboxconfig.ate.dev/is-class-default: "true"
  deletionTimestamp: "2026-10-11T18:42:07Z"
  finalizers:
  - sandboxconfig.ate.dev/workerpool-protection   # kept while a WorkerPool status references the config
spec:
  ...
```

To find out which WorkerPools are holding it:

```
kubectl get workerpool -A -o custom-columns=NAMESPACE:.metadata.namespace,NAME:.metadata.name,SANDBOXCONFIG:.status.sandboxClasses[*].configRef.name
```

## Testing

- Unit tests cover resolution, the status write, the finalizer, the
two-step switch, and the events, including that a switch is reported
once when the first status write is rejected.
- The new e2e suite `internal/e2e/suites/sandboxconfig` creates a pool
with no `configRef` and no workers and walks it through the sequence in
the events example above. Run it with `hack/run-e2e.sh
./internal/e2e/suites/sandboxconfig/...`.

## Breaking changes

- A new WorkerPool gets no Deployment until every sandbox class has a
resolved config. Before upgrading, make sure each class has a config
annotated `is-class-default: "true"`, or set `configRef` on the pool.
The bundled `gvisor-default` and the micro-VM config are already
annotated. Existing pools keep their Deployment.
- `spec.sandboxClasses[].name` is now immutable. To change the class,
create a new pool.
- Rolling atecontroller back to a version before this PR leaves the
finalizer on SandboxConfigs. Remove it with `kubectl patch sandboxconfig
<name> --type=merge -p '{"metadata":{"finalizers":null}}'`.

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

* Simplify Snapshot API for Actor Lifecycle v2 (agent-substrate#2449)

The initial API developed in agent-substrate#2289 had a repeated SnasphotStorage
structure keyed by Durability (RESIDENT, LOCAL and DURABLE). However
this was overly descriptive. RESIDENT and LOCAL storage details are
local to atelet, and the promotion of snapshots from RESIDENT -> LOCAL
-> DURABLE is also driven by atelet. Putting the information in Snapshot
API meant that every action triggered by atelet has to invoke ateapi
first to record the transition.

Instead, this PR simplifies the sturcture by having a single
SnapshotStorage to contain the durable storage details. Locality details
will not be recorded in the Snapshot API. We only record that a snapshot
also has locality on a certain node.

#### Breaking change
LocalSnapshot is removed and replaced by a Snapshot.locality field.

Part of agent-substrate#2069

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

---------

Signed-off-by: timflannagan <[email protected]>
Signed-off-by: Eitan Yarmush <[email protected]>
Co-authored-by: Taahir Ahmed <[email protected]>
Co-authored-by: Tim Flannagan <[email protected]>
Co-authored-by: cloudonly <[email protected]>
Co-authored-by: Dmitry Berkovich <[email protected]>
Co-authored-by: ktsoator <[email protected]>
Co-authored-by: cloudonly <[email protected]>
Co-authored-by: Pancho Trujillo <[email protected]>
Co-authored-by: Zoe Zhao <[email protected]>
Co-authored-by: Michelle Au <[email protected]>