atelet, ateworker: probe CPU features and watch sandbox-runtimes.json for dynamic worker registration - #2660

Follow-up to agent-substrate#2660 and agent-substrate#2755.

Sandbox runtime and version selection live on the `WorkerPool`: its
`sandboxClasses` entry resolves a `SandboxConfig`, atelet reports the
enabled versions on `WorkerStatus` as
`defaultRuntime`/`restorableRuntimes`, and the chosen runtime is stamped
onto `WorkerAssignment` and `Snapshot` and sent to atelet on Run and
Restore. The `ActorTemplate` no longer has a say in which sandbox an
actor runs in, so its `sandbox_config` field goes.

**Breaking:** `ActorTemplate.sandbox_config` and the `SandboxConfig`
message are removed from the Control API. ate-api-server rejects unknown
fields, so manifests or clients still sending `sandboxConfig` fail with
`InvalidArgument`. Stored templates are unaffected (proto bytes; the old
field is carried as unknown). `kubectl ate get actortemplates` loses its
`SANDBOX CLASS` column.

**API**
- Remove `sandbox_config` and the `SandboxConfig` message from
`ActorTemplate` in `ateapi.proto`, and its validation from
ate-api-server.
- Remove `sandboxConfig` from `kubectl-ate` output, e2e fixtures, demo
and benchmark `ActorTemplate` manifests (sweperf included), and the
docs.

**ateapi**
- `sandbox_assets` on Run/Restore is unchanged on the wire; ateapi now
resolves it from the assigned worker's pool instead of the template:
`WorkerAssignment` → `WorkerPool` → the controller-confirmed
`SandboxConfig` (`WorkerPool.ResolvedSandboxConfig()`, new) → default
version. `ActorWorkflow` takes a `WorkerPoolLister` for this; the
`sandboxconfigs` RBAC stays. Marked TODO: this lookup, both listers and
the RBAC go once atelet resolves the version from `sandbox_runtime` and
`sandbox_assets` is dropped (the follow-up noted in agent-substrate#2755).
- Scheduler: `Constraints.SandboxClass` is no longer taken from the
template. It is the class of the snapshot the actor restores from
(local, else latest durable, golden tags included), empty on a cold boot
— so a worker of another class is never a candidate, not even as the
volumes-only fallback when `RequiredRuntime` is relaxed. A cold-booting
actor gets the class of the pools its `workerSelector` matches.
- Repointing `actor_template` on a SUSPENDED actor no longer requires
the same SandboxConfig; the snapshot's class and runtime checks on
resume cover compatibility.
- Metrics: `ate.sandbox.class` on lifecycle and scheduler operations is
derived from the actor's assignment or snapshot runtime rather than the
template, and omitted when neither is known yet; registry updated.

**Behavior change to flag:** a template with no `workerSelector` whose
actor has no snapshot can be placed on a pool of any class. Previously
`sandboxConfig.sandboxClass` alone pinned the class; now the selector
has to.

**Tests**
- `TestResolveSandboxAssets` covers pool-based resolution (resolved,
unresolved pool, missing pool, missing config, no default version).
- Scheduler `Applies` cases for the snapshot-class pin;
`assignWorkerAttempt` proves a microvm worker is never the fallback for
a gvisor snapshot at MEMORY or VOLUMES fidelity.
- Wire-request tests seed the pool lister; the "missing SandboxConfig"
cases now go through the pool.
- Functional tests write the pool's `status.sandboxClasses[].configRef`
via `UpdateStatus` (status subresource) and wait for it in the informer.
- `ReleaseWindowWorkflows` makes the worker ineligible via
`workerSelector` instead of a sandbox-class mismatch.

No atelet changes beyond two comments.

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

…lans it (agent-substrate#2741)

> **Rebased onto main after agent-substrate#2731 merged.** Three commits: the change,
then two rounds of review fixes kept separate so each thread's change is
visible. Since agent-substrate#2660 the syncer fetches a gVisor version's runsc to
probe its CPU features; that fetch now goes through the same asset cache
and is bounded by the prewarmer's attempt timeout.

Step 4 of agent-substrate#2675: the default runtime of each worker on the node is
prewarmed by the syncer that plans the worker, through the runtime
index, and the cluster-wide prewarmer goes away.

**Before.** `sandbox_prewarm.go` watched every SandboxConfig in the
cluster, queued by config name, and fetched each config's default
version, gated only by whether the node had `/dev/kvm`. Most of what it
fetched no worker on the node would ever run, and nothing related it to
what the node's workers advertise.

**After.** The worker sandbox syncer already plans each local worker
pod, registers its runtimes in the index, and writes its versions file.
It now also hands the worker's default runtime hash to a
`runtimePrewarmer` right after registering the worker's hashes in the
index, before the build probes the versions: the build fetches and
probes every restorable version serially, longest on a fresh node, and
the file it ends with is what lets the worker take its first actor, so
the probes are the prewarm's head start and the file never waits on a
download. The prewarmer's queue is keyed by hash, and the prewarmer
keeps the hashes whose prewarm succeeded and skips them, so the workers
of one pool, every re-plan and every retry of a sync cost one prewarm of
a runtime, not one per pod; a failed prewarm is queued again. Its single
worker takes the record from the index, stages it into a scratch
directory under the base path through the same `ensureSandboxAssets` Run
uses, removes the scratch directory, and pulls the pause image
alongside. Eight retries with exponential backoff, a ten-minute bound
per attempt (a field on the prewarmer, like the hook a test uses to wait
for an attempt), and an unknown hash is dropped with an error log since
the syncer registers before it enqueues. The syncer's runsc probe keeps
its own five-minute backstop rather than borrowing the prewarmer's.
Prewarm stays a latency optimization: every failure is logged and left
to the on-demand fetch.

**Removed.** `sandbox_prewarm.go`, its tests, and `microvmNodeCapable`,
which only it used. Locality comes from the pods on the node: a micro-VM
pod only lands where it can run.

**Scope.** Only default runtimes are prewarmed. The first restore of a
non-default version a worker advertises pays its download.

**Tests.** `TestRuntimePrewarmProcess` (retry until the cap, then
forget; unknown hash forgotten at once),
`TestRuntimePrewarmFillsBothCaches` (asset in the cache, scratch dir
empty, pause image in the image cache),
`TestRuntimePrewarmAssetFailureStillPullsImage`,
`TestRuntimePrewarmAttemptTimeout`, `TestSyncPodPrewarmsDefaultRuntime`
(exactly the default hash of a multi-version config, handed over before
the build fetches for a probe, and still handed over when the build then
fails), `TestRuntimePrewarmOnce` (a prewarmed hash is skipped by later
enqueues, a failed one is queued again), and
`TestDefaultRuntimePrewarmThroughSyncer`, the end-to-end run through
fake clientsets and informers that the old download test did. The old
enqueue-filter, class-gate, and KVM tests go with the behavior they
covered.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

…nt-substrate#2880)

Now that gVisor supports `SNAPSHOT_FIDELITY_ROOTFS` alongside micro-VM
(agent-substrate#2673), a snapshot's rootfs upper layer depends only on the container
image (`ActorTemplate`) and not on the worker's sandbox runtime version
or host CPU features.
- **`requiredSnapshotRuntime`**: Only impose a `RequiredRuntime`
placement constraint when restoring `SNAPSHOT_FIDELITY_MEMORY`
snapshots; `SNAPSHOT_FIDELITY_ROOTFS` and `SNAPSHOT_FIDELITY_VOLUMES`
snapshots can schedule onto any eligible worker of the same sandbox
class without runtime matching.
- **`ensureAteletRestored`**: When a durable `MEMORY` snapshot falls
back onto a runtime-incompatible worker, downgrade the restore fidelity
to `SNAPSHOT_FIDELITY_ROOTFS` (preserving both rootfs writes and durable
volumes) instead of `SNAPSHOT_FIDELITY_VOLUMES`. Snapshots whose
`ActorTemplate` was replaced continue to downgrade to
`SNAPSHOT_FIDELITY_VOLUMES`.
- **Tests & Docs**: Update `workflow_resume_test.go`,
`functionaltest/actor_test.go`, and `docs/api-guide.md` to reflect
`ROOTFS` fallback semantics.
Follow-up to agent-substrate#2673 and agent-substrate#2660.
<img width="1364" height="1688" alt="image"
src="https://github.com/user-attachments/assets/7a105b64-2c30-440f-8679-5afbeb591f7d"
/>



- [ ] Tests pass
- [ ] Appropriate changes to documentation are included in the PR