atelet: pass SandboxRuntime on Run and Restore - #2755

…rs advertise (agent-substrate#2731)

> **Rebased onto main through agent-substrate#2755**, which puts `sandbox_runtime` on
Run and Restore. This PR still resolves from `sandbox_assets`; switching
to `sandbox_runtime` and removing `sandbox_assets` is the next PR. Four
commits: the index, then three rounds of review fixes, kept separate so
each thread's change is visible.

Step 3 of agent-substrate#2675: atelet keeps an index of the sandbox runtimes its
workers advertise, persisted on the node, and resolves an actor's assets
through it.

**The index.** The worker sandbox syncer already resolves each local
worker pod to its pool's SandboxConfig and hashes every enabled version
into the worker's `sandbox-runtimes.json`. It now also registers each
hash with the asset record it was computed from. Two maps: runtimes by
hash, never released (a request may carry a hash a worker advertised
earlier; entries are a few URLs; TODO to release stale ones), and what
each worker currently advertises by pod UID and sandbox class. A pool
runs one class today; the shape admits several.

**Order of registration.** The hashes are registered before the worker's
file is written, so a hash is resolvable before any worker can register
it with the control plane; a SandboxConfig or WorkerPool change
registers the affected pools' runtimes from the event handler at once,
ahead of the jittered rewrite of the files, which reads the listers
only. What a worker advertises is recorded only after its file is
written, and only for the runtimes the file names, so the index never
credits a worker with a default its file does not carry. Two versions
with one hash have the same assets and pause image, so the first
registration's record is kept.

**Persistence, no startup gate.** The runtimes map is also one file
under the node state dir, `sandbox-runtime-index.json`, the map
serialized, rewritten durably (`atomicfile.WriteSync`, data then
directory) before a new entry enters memory. A worker registers a hash
only after atelet wrote the worker's own file, which happens after the
index was written, so every hash a worker registered before an atelet
restart is on disk and loads before atelet serves. There is therefore no
wait on the syncer at startup and no window in which a registered
worker's request can miss. On load, an entry whose record does not hash
to its key is dropped and a file that does not decode starts the index
empty; the syncer writes the current runtimes again either way. A sync
that cannot persist a runtime fails and is retried. The file stays
small: a node sees a handful of configs with a few versions in its
lifetime. Memory stays the read path.

**Run and Restore** hash the request's assets with the same function the
syncer uses and stage the index's record. A miss fails the request with
FailedPrecondition naming the hash: it means the control plane placed
the actor on a worker whose pool does not run that config or version.
There is deliberately no fallback to the request's own assets. The
hashing goes away in the next PR, which resolves from `sandbox_runtime`.

**Readiness.** /readyz holds 503 from the health server's first answer
until the gRPC port is open, through a check registered with the
readiness object, instead of answering 200 during the trust bundle sync
and the plugin wait. A syncer that cannot start is fatal: without it no
runtime is ever indexed.

**Tests.** Index registration with a shared hash across two configs, a
re-plan replacing a worker's set while the old runtime stays resolvable,
forgetting a gone pod, and early registration without a worker entry.
Persistence: register, reopen, resolve with provenance; a mis-keyed
entry dropped alone; a corrupt file loads empty; an unwritable path
fails the registration. Resolution hit, miss, and unpinned pause image.
A Run for an unadvertised runtime is refused with nothing fetched and no
actor dir created, then resolves once advertised. The syncer indexes the
pods already on the node and retries an unresolvable one in the
background; a changed config's runtimes resolve before the jittered sync
with both pods queued after it; a worker's entry follows its written
file through a failed sync and a partial file; the process test asserts
the index entry's fate in every case. Every lifecycle test that sends
assets advertises them first.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

…_runtime (agent-substrate#2788)

> Follows agent-substrate#2755, which put `sandbox_runtime` on Run and Restore.

Run and Restore stop hashing the request's `sandbox_assets` to find the
runtime to stage, and read the `sandbox_bundle_hash` attribute of the
request's `sandbox_runtime` instead. That field is what ate-api-server
fills from the worker assignment: the runtime the worker registered from
the file this node's atelet wrote, so the hash is in the index by
construction. The stopgap hashing and its TODOs go.

**Required.** A request without `sandbox_runtime`, or without the
attribute, is InvalidArgument. Every worker ate-api-server can place an
actor on has registered a default runtime, because
`internal/ateworker/register` registers only once the runtimes file
names one, so an assignment always has a runtime to send. An unknown
hash stays FailedPrecondition. A hash advertised under another sandbox
class than the request names is FailedPrecondition too, since the hash
does not cover the class.

**Not yet.** `sandbox_assets` stays in the request and is no longer
consulted. Removing it from the protos, ate-api-server and atelet is the
next PR, so this one stays a change to how atelet reads the request.

**Tests.** The lifecycle tests' Run and Restore requests carry the
runtime ate-api-server would send for an actor placed on the worker that
advertises their assets. A request with assets but no runtime is
InvalidArgument, and still before any on-node work: the invalid-Run test
now proves the detach stays scheduled that way instead of through assets
for another architecture, which the resolver no longer reads. The
resolver test covers the miss, the hit, four malformed runtimes and the
class mismatch.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Follow-up to agent-substrate#2660 and agent-substrate#2755.

Sandbox runtime and version selection live on the `WorkerPool`: its
`sandboxClasses` entry resolves a `SandboxConfig`, atelet reports the
enabled versions on `WorkerStatus` as
`defaultRuntime`/`restorableRuntimes`, and the chosen runtime is stamped
onto `WorkerAssignment` and `Snapshot` and sent to atelet on Run and
Restore. The `ActorTemplate` no longer has a say in which sandbox an
actor runs in, so its `sandbox_config` field goes.

**Breaking:** `ActorTemplate.sandbox_config` and the `SandboxConfig`
message are removed from the Control API. ate-api-server rejects unknown
fields, so manifests or clients still sending `sandboxConfig` fail with
`InvalidArgument`. Stored templates are unaffected (proto bytes; the old
field is carried as unknown). `kubectl ate get actortemplates` loses its
`SANDBOX CLASS` column.

**API**
- Remove `sandbox_config` and the `SandboxConfig` message from
`ActorTemplate` in `ateapi.proto`, and its validation from
ate-api-server.
- Remove `sandboxConfig` from `kubectl-ate` output, e2e fixtures, demo
and benchmark `ActorTemplate` manifests (sweperf included), and the
docs.

**ateapi**
- `sandbox_assets` on Run/Restore is unchanged on the wire; ateapi now
resolves it from the assigned worker's pool instead of the template:
`WorkerAssignment` → `WorkerPool` → the controller-confirmed
`SandboxConfig` (`WorkerPool.ResolvedSandboxConfig()`, new) → default
version. `ActorWorkflow` takes a `WorkerPoolLister` for this; the
`sandboxconfigs` RBAC stays. Marked TODO: this lookup, both listers and
the RBAC go once atelet resolves the version from `sandbox_runtime` and
`sandbox_assets` is dropped (the follow-up noted in agent-substrate#2755).
- Scheduler: `Constraints.SandboxClass` is no longer taken from the
template. It is the class of the snapshot the actor restores from
(local, else latest durable, golden tags included), empty on a cold boot
— so a worker of another class is never a candidate, not even as the
volumes-only fallback when `RequiredRuntime` is relaxed. A cold-booting
actor gets the class of the pools its `workerSelector` matches.
- Repointing `actor_template` on a SUSPENDED actor no longer requires
the same SandboxConfig; the snapshot's class and runtime checks on
resume cover compatibility.
- Metrics: `ate.sandbox.class` on lifecycle and scheduler operations is
derived from the actor's assignment or snapshot runtime rather than the
template, and omitted when neither is known yet; registry updated.

**Behavior change to flag:** a template with no `workerSelector` whose
actor has no snapshot can be placed on a pool of any class. Previously
`sandboxConfig.sandboxClass` alone pinned the class; now the selector
has to.

**Tests**
- `TestResolveSandboxAssets` covers pool-based resolution (resolved,
unresolved pool, missing pool, missing config, no default version).
- Scheduler `Applies` cases for the snapshot-class pin;
`assignWorkerAttempt` proves a microvm worker is never the fallback for
a gvisor snapshot at MEMORY or VOLUMES fidelity.
- Wire-request tests seed the pool lister; the "missing SandboxConfig"
cases now go through the pool.
- Functional tests write the pool's `status.sandboxClasses[].configRef`
via `UpdateStatus` (status subresource) and wait for it in the informer.
- `ReleaseWindowWorkflows` makes the worker ineligible via
`workerSelector` instead of a sandbox-class mismatch.

No atelet changes beyond two comments.

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR