[release-1.0] Cherry-pick 23 main changes through 4488a6667 - #3536

…ate#3309)

The locust runner uploads the same stats file twice, as `stats.jsonl`
and `{name}.jsonl`. The second copy was kept for a data migration (see
the removed TODO). Nothing in the repo reads it now.

I used Claude Code to find the duplicate upload and make the edit. I
reviewed the diff.

Signed-off-by: Haiyan Meng <[email protected]>
(cherry picked from commit 0d662b8)
Signed-off-by: Eitan Yarmush <[email protected]>

<!-- The PR title becomes the release note. Write it for users. -->

Fixes agent-substrate#3268

### Problem

Envoy asks for config in a fixed order: clusters, then the 2 secrets the
clusters need, then listeners, then the 2 secrets the listeners need. It
won't ask for listeners until every cluster is ready.

Over state-of-the-world xDS, the router's ADS cache [answers a request
that
lists names only if it names every
resource](https://github.com/envoyproxy/go-control-plane/blob/71637ad69bbc5f51fbb2562e612a4365292804a5/pkg/cache/v3/simple.go#L491).
Envoy's first secret request names only the 2 cluster secrets, so it was
ignored. Envoy waited out its 15s `initial_fetch_timeout`, marked the
cluster ready anyway, then asked for listeners and all 4 secrets. With
one
replica, that meant 15s with no ingress on every start.

### Solution

The router Envoy now uses delta xDS (`api_type: DELTA_GRPC`). The delta
path in go-control-plane has no such check, and with `ads=true` it still
orders updates across resource types, which we'll want if the router
ever
serves dynamic config.

### Result

With real Envoy v1.39.5, startup goes from 15.3s to 0.35s, with no
timeout. `TestXdsServer_AnswersClusterSecretsBeforeListeners` checks the
manifest uses delta and replays the startup over a delta stream. It
fails
on main.

AI use: AI-assisted. I reviewed the change and ran it against real
Envoy.

- [x] Tests pass
- [x] Docs updated (none needed)

Signed-off-by: Chuang Wang <[email protected]>
(cherry picked from commit 62efb70)
Signed-off-by: Eitan Yarmush <[email protected]>

…te#3264)

Expand the demo e2e suite to cover asynchronous suspend background
upload, background detach, and CreateTag promotion.

Lower atelet's snapshot detach delay in kind so the suite can wait for
background detach. Checks that wait for a background upload or detach
skip on installs with longer delays, and fail in CI.

AI use: AI-assisted, I reviewed and tested it.

- [ ] Tests pass
- [ ] Docs updated

Signed-off-by: msau42 <[email protected]>
(cherry picked from commit 910671c)
Signed-off-by: Eitan Yarmush <[email protected]>

Follow the docsite to create a animated flow of substrate in README, add
and some metric badges

AI usage: apparently all done by AI.

- [x] Tests pass
- [x] Docs updated

Signed-off-by: Haven Xia <[email protected]>
(cherry picked from commit 7cd9fe7)
Signed-off-by: Eitan Yarmush <[email protected]>

…gent-substrate#3435)

`ate.dev/llms.txt` linked to full Docsy HTML pages instead of standalone
Markdown files, omitted page status markers and schema links, and still
used the old `ateom` component name. Meanwhile, `raw-markdown.html`
wrote into an HTML template buffer and expanded `include-file` with
`replaceRE`, which HTML-escaped code blocks and blockquotes across
`ate.dev/llms-full.txt` and "Copy as Markdown" (`'`, `"`,
`>`), stripped `$VAR` shell references as regexp capture groups, left
relative links unrewritten, and kept decorative `<figure>` HTML blocks.

- Add the `MARKDOWN` output format for doc pages
(`site/layouts/docs/single.md`) so each page publishes a clean
`index.md` alongside `index.html` and exposes `<link rel="alternate"
type="text/markdown">` in `<head>`.
- Return a plain string from `raw-markdown.html`, expand `include-file`
with literal replacement, drop decorative diagram HTML, and rewrite
relative links to absolute URLs across `llms-full.txt`, per-page
`index.md`, and "Copy as Markdown".
- Update `index.llms.txt` to link to each page's `index.md` build,
include `[Status: ...]` labels from front matter, link the
`ateapi.proto`, `ate.dev/v1` CRD, and OpenTelemetry Weaver metric
schemas, and summarize the sandbox model and the split between
`ate.dev/v1` Kubernetes CRDs and the `ate-api-server` gRPC API.
- Replace leftover `ateom`, `ate-api`, and `herder` references in docs
with `ateworker`, `atenet-router`, and `ate-api-server`, and add
template tests in `hack/test-site-templates.sh`.

AI use: AI-assisted, I reviewed and tested it.

- [x] Tests pass
- [x] Docs updated

---------

Signed-off-by: Mofi Rahman <[email protected]>
(cherry picked from commit 61ea6c6)
Signed-off-by: Eitan Yarmush <[email protected]>

…rate#3474)

`README.md` and `docs/concepts/architecture.md` claimed resume
operations under 400ms and more than 1500 activations per second, while
the website and get-started docs say under 500ms and 500+/s. Update both
current-capability claims to less than 500ms and more than 500
activations per second, leaving the aspirational North Star targets in
`docs/concepts/architecture.md` unchanged.

AI use: AI-assisted, I reviewed and verified the changes.

- [x] Tests pass: docs-only change
- [x] Docs updated

---------

Signed-off-by: Mofi Rahman <[email protected]>
(cherry picked from commit 1261766)
Signed-off-by: Eitan Yarmush <[email protected]>

…t-substrate#3468)

Part of agent-substrate#3226

Add docs/reference/sandbox-classes.md comparing the gvisor and microvm
sandbox classes across CPU sizing, memory limits, per-container limits,
NET_RAW capabilities, system mounts, VOLUMES/ROOTFS snapshots and gVisor
fscheckpoint compatibility requirement for cross-version filesystem
restores. Update stale references across architecture, glossary,
quickstart, and guides.

AI use:
AI-assisted, I reviewed and tested it.

- [ ] Tests pass
- [x] Docs updated

---------

Signed-off-by: Nayana Bidari <[email protected]>
(cherry picked from commit 46b6ea9)
Signed-off-by: Eitan Yarmush <[email protected]>

Snapshots are stored under
`<location>/atespaces/<atespace>/{actors,tags}/<uid>`. A
`storageLocation` that already contained that layout (for example
`gs://snaps/root/atespaces/team-a/actors/<uid>`) put one template's
snapshots inside another actor's prefix, grant and cleanup.

Where a location is set (template create, `--snapshot-location`,
`ate-setup deploy`), each path segment may now contain only ASCII
letters, digits, `_` and `-`, and may not be `atespaces` in any letter
case. The bucket is not restricted.

- Creating a tag checks the template's location against the same rules
and returns `FAILED_PRECONDITION` on a bad one.
- Parsing of recorded snapshot URIs stays lenient, so stored snapshots
still resume and get collected.

Breaking change: if the path of your `--snapshot-location`
(`SNAPSHOT_LOCATION`) has a segment outside `^[A-Za-z0-9_-]+$` (a dot,
`%`, an empty segment, non-ASCII) or equal to `atespaces` in any case,
change it before upgrading. Otherwise ate-api-server refuses to start.

- [x] Tests pass (touched packages, `hack/verify-all.sh`)
- [x] Docs updated

AI use: I used Claude Code to implement and review this change, and
reviewed the diff myself.

---------

Signed-off-by: Anish Ramasekar <[email protected]>
(cherry picked from commit c53f4b2)
Signed-off-by: Eitan Yarmush <[email protected]>

### Summary
Adds `--agentsession-awake` (paired with `--target-rps`) to
`agentsession` so benchmarks can maintain a large pool of registered
sessions while keeping a fixed number resident and swapping idle awake
sessions with parked ones at a controlled rate.

### How It Works
- **Registration (`createSession`)**: When `--agentsession-awake > 0` or
`--lifecycle-mode none`, each session runs step 1 as soon as its actor
serves so the initial actor/snapshot holds real memory and disk state.
In `--lifecycle-mode none` at `--agentsession-think-scale 1`,
`personal-assistant-light` waits 55m before step 1, so a 20m resident
density run would otherwise measure empty sandboxes.
- **Fill**: Once session registration has been idle for 1 minute
(`registrationQuiet`), the conductor wakes `--agentsession-awake`
sessions (`fillConcurrency = 8` at a time) and logs `agentsession:
turnover fill done`.
- **Turnover (`--target-rps`)**: Each second, the conductor swaps
`--target-rps` idle awake sessions (`cmdPark`) with parked sessions
(`cmdWake`). Each woken session immediately runs its next scripted step
while marked `busy` and returns to `idle` when the step finishes.
- **Measurement isolation**:
- Registration parks record under `SuspendActor_Register` /
`SuspendActorAsync_Register` so `SuspendActor` and `SuspendActorAsync`
hold only turnover parks.
- `runner.py` parses `agentsession: turnover fill done`
(`locust/turnover_window.py`), starts `server_telemetry`'s steady-state
window there, and appends a `turnover_window` row to `stats.jsonl` with
per-request counts, failures, and rates from `stats_history.csv` over
the turnover phase.
- **Observability**: Exports
`agentsession_turnover_sessions{state="awake|parked|waking|parking"}`.

### Testing
```bash
go test -race -count=3 -shuffle=on ./internal/benchmarking/boomer/agentsession/...
python3 -P -m unittest discover -s benchmarking/locust/unit_tests -p "test_*.py"
```

---------

Signed-off-by: Chenyi Wang <[email protected]>
(cherry picked from commit 5e22427)
Signed-off-by: Eitan Yarmush <[email protected]>

…bstrate#3404)

Several control-plane docs and the `kubectl-ate` reference still
described earlier field names (`durableSnapshot.status`,
`source_snapshot_tag`), empty-atespace rules, and older CLI columns and
flags.

Updates them to match the current implementation:

- `docs/reference/glossary.md`: lists `Tag` under control-plane records,
and notes that deleting an atespace requires its actors, actor
templates, and tags to be gone.
- `docs/reference/api-guide.md`: fixes `durableSnapshot.state`,
documents deletion checks on `CreateActor` and `UpdateActor`, and adds
`GetWorker` and `ListWorkerActorAssignments`.
- `docs/api-style-guide.md`: renames `source_snapshot_tag` to
`source_tag`.
- `docs/guides/authorization.md`: `CreateActor` skips `can_use_tag` when
`sourceTag` is omitted.
- `docs/reference/kubectl-ate.md`: documents `--preview`,
`--sandbox-class`, `top workers`, the current `get worker` and `get
actor` columns, template removal before `delete atespace`, and automatic
snapshot upload on `create tag`.

AI use: AI-assisted to audit control-plane docs against the codebase and
draft the documentation updates. I reviewed and verified the changes.

- [x] Tests pass
- [x] Docs updated

Signed-off-by: Haiyan Meng <[email protected]>
(cherry picked from commit f475d8b)
Signed-off-by: Eitan Yarmush <[email protected]>

…ent-substrate#3472)

A routed request to an actor in ACTOR_STATE_CRASHED parks in
atenet-router for the whole 5 s budget, makes about 19 `ResumeActor`
calls, holds one parking slot throughout, and then answers 503. The
state never clears on its own, so the wait buys nothing.

Fixes agent-substrate#3269
Part of agent-substrate#1562

`retryable` in `cmd/atenet/internal/router/ingress/resumer.go` drops
`FailedPrecondition` from the park arm, so the request fails on the
first attempt with the same 503 and the same body. `ResourceExhausted`
and `Unavailable` still park, `Aborted` is still always retried, and
nothing changes with parking disabled. No ateapi change.

Behavior change: every `FailedPrecondition` from `ResumeActor` now fails
fast, not only CRASHED. That covers DELETING, SUSPENDING and REVERTING,
a missing template, a volume without a storage ID, a worker without a
node name, and a failed `DetachActor`. Two situations behind SUSPENDING
and REVERTING:

- An operation in progress. `SuspendActor`, `RevertActor` and
`DeleteActor` hold the actor lease for their whole run, so a resume
during one gets `Aborted`, which still parks. Nothing changes here.
- A failed operation. A failed suspend/revert that fails at any step,
leaves the actor SUSPENDING/REVERTING with no lease held. The resume
then gets `FailedPrecondition` and now fails fast. A retried suspend or
revert is what moves the actor on, never the resume retry.

The status code and the body are unchanged.

The e2e `TestRoutedRequestToCrashedActorFailsFast` crashes an actor by
deleting its worker pod and routes one request: 503 naming the state
within 2 s, where a parked request would first wait out the 5 s budget.
It skips on the AgentGateway lane, whose resume path is outside this
repo. `docs/concepts/request-parking.md` now lists `FailedPrecondition`
under fail fast. Verified with `go test -race -count=3 -shuffle=on` on
the router package, `hack/verify-all.sh`, and the demo e2e suite on
kind.

AI use: AI-assisted under my direction; I reviewed the diff and ran the
tests.

- [x] Tests pass
- [x] Docs updated

Signed-off-by: Yuan Gao <[email protected]>
(cherry picked from commit 542eca1)
Signed-off-by: Eitan Yarmush <[email protected]>

…strate#2092)

Stacked on agent-substrate#1838.

## Summary
Ports the microVM dependency setup (`hack/install-microvm-deps.sh` and
`hack/microvm-assets/`) to Go, and folds microVM asset assembly,
staging, and `SandboxConfig` management directly into `deploy
ate-system`, `deploy sandboxconfig`, and `delete ate-system`.

## Changes
- **Fold microVM setup into control-plane deploy/delete
(`cmd/ate-setup/internal/steps/deploy.go`, `delete.go`,
`benchmarks.go`)**:
- Calls `prepareMicroVMDeps` (before cluster resources are modified) and
`applyMicroVMSandboxConfig` inside `DeployAteSystem` (after applying the
system bundle so Kind's in-cluster `rustfs` Deployment and
`rustfs-bucket-init` Job exist before asset staging) and inside
`DeploySandboxConfig`.
- Adds `--sandbox-classes` (default `gvisor,microvm`) to `ate-setup
deploy ate-system` and `ate-setup deploy sandboxconfig`
(`--sandbox-classes=gvisor` opts out of staging microVM assets and
applying `SandboxConfig/microvm`).
- Releases `SandboxConfig` protection finalizers inside
`DeleteAteSystem` before CRD teardown when no `WorkerPool` references
the config.
- Updates `DeployBenchmarks` / `DeleteBenchmarks` since `deploy
ate-system` now installs both `gvisor-default` and `microvm`
`SandboxConfig`s up front, and checks that `sandboxconfig/microvm`
exists when `--sandbox-class=microvm`.
- **Remove `hack/install-microvm-deps.sh` and `hack/microvm-assets/`**:
- Deletes `hack/install-microvm-deps.sh` and the entire
`hack/microvm-assets/` directory (`assemble.sh`, `stage-to-gcs.sh`,
`stage-to-rustfs.sh`, and `README.md`).
- Updates `hack/run-microvm-demo.sh`,
`benchmarking/automation/orchestrator.py`, and
`.github/workflows/pr-workflow.yaml` to rely on `deploy ate-system`.
- **Native Go multi-arch microVM asset assembly and staging
(`cmd/ate-setup/internal/steps/microvm.go`)**:
- Streams and extracts the Kata `.tar.zst` archive (`vmlinux.container`,
`kata-containers-ubuntu.img` as `rootfs.img`, `virtiofsd`) and Cloud
Hypervisor binary in Go (`runtime.GOARCH` on Kind; both `amd64` and
`arm64` on non-Kind clusters) into `bin/microvm-assets/<arch>/`.
- Checks the SHA-256 digests of all four assets (`cloud-hypervisor`,
`virtiofsd`, `vmlinux`, `rootfs.img`) against the digests pinned in
`manifests/ate-install/sandboxconfig-microvm.yaml.tmpl` (both after
assembly and when reusing a cached `.asset-versions` directory).
- Stages assets by content digest under `microvm-assets/<arch>/<sha256>`
in Kind's `rustfs` bucket or GKE's `<bucket>-assets` GCS bucket, then
renders and applies
`manifests/ate-install/sandboxconfig-microvm.yaml.tmpl`.
- **Docs, manifests & unit tests**:
- Moves `sandboxconfig-microvm.yaml.tmpl` to
`manifests/ate-install/sandboxconfig-microvm.yaml.tmpl` with per-arch
`microvm-assets/<arch>/<sha256>` URLs and removes stale version-number
comments.
- Folds the remaining asset-set and snapshot-location notes from
`hack/microvm-assets/README.md` into `docs/dev/microvm-local.md` and
updates `docs/reference/api-guide.md`,
`docs/guides/operator-install.md`, `cmd/ate-setup/README.md`,
`cmd/ate-setup/commands.md`, `cmd/ate-setup/differences.md`,
`demos/counter/README.md`, and `hack/release/draft-release.sh`.
- Adds unit tests in `cmd/ate-setup/internal/steps/microvm_test.go`,
`cmd/ate-setup/internal/steps/benchmarks_test.go`,
`cmd/ate-setup/internal/kube/wait_test.go`, and
`cmd/ate-setup/internal/cmd/deploy_test.go`.

## Testing
- Run `go test -race ./cmd/ate-setup/... ./internal/e2e/...` and
`hack/verify-all.sh`.
- Local Kind cluster verification (`go run ./cmd/ate-setup --kind`):
- Verified `deploy ate-system --sandbox-classes=gvisor` deploys the
control plane and installs only `SandboxConfig/gvisor-default`.
- Verified `deploy ate-system` assembles the microVM guest assets,
stages them into the in-cluster `rustfs` bucket, and applies
`SandboxConfig/microvm` alongside `SandboxConfig/gvisor-default`
(reusing cached assets via `.asset-versions` on subsequent runs).
- Verified `delete all` removes `SandboxConfig/microvm` during
control-plane teardown before deleting CRDs.
- Deployed `deploy demo counter` and ran `./hack/run-e2e-kind.sh
./internal/e2e/suites/demo`.

## Folded in from open PRs
Because this PR replaces `hack/microvm-assets/stage-to-rustfs.sh` with
`stageMicroVMAssetsToRustfs`, it folds in the `stage-to-rustfs.sh` fixes
from two open PRs:
- agent-substrate#2905 (@dims): look up the `app=rustfs` pod's `spec.nodeName` so the
S3 upload container runs in the node network namespace admitted by
rustfs's NetworkPolicy on multi-node Kind clusters.
- agent-substrate#1122 (@ygao-g): use `net.JoinHostPort(svc.Spec.ClusterIP, "9000")`
for `AWS_ENDPOINT_URL` so IPv6 ClusterIPs are bracketed.

AI Use: AI assisted implementation & unit testing, human reviewed,
refined, and locally tested.

---------

Signed-off-by: Noah West <[email protected]>
Co-authored-by: jybao <[email protected]>
(cherry picked from commit 8edd9ae)
Signed-off-by: Eitan Yarmush <[email protected]>

…share a CPU architecture (agent-substrate#3458)

Part of agent-substrate#3230

A `SNAPSHOT_FIDELITY_ROOTFS` restore reuses the container's rootfs upper
layer, which can contain architecture-specific binaries written on the
worker where the snapshot was captured. Until the scheduler filters by
CPU architecture on fallback, every `WorkerPool` selected by an
`ActorTemplate` must run on the same CPU architecture.

Previously, this requirement was only mentioned under `Snapshot Storage
Layout` (`sandboxRuntime`) in `docs/reference/api-guide.md`. Surface it
where operators and users configure pools and templates:

- `README.md`: add `Known limitations` under `Status and compatibility`.
- `docs/reference/api-guide.md`: note how to pin and label `WorkerPool`s
in mixed-architecture clusters under `WorkerPoolPodTemplate`, and note
the single-architecture requirement on `ActorTemplate.workerSelector`.
- `api/ateapi/v1/ateapi.proto` (`ateapi.pb.go`): document the
requirement on `ActorTemplate.worker_selector`.

Used an AI assistant to draft the documentation wording and reviewed it
locally.

- [x] Tests pass
- [x] Docs updated

Signed-off-by: Shrutiya <[email protected]>
(cherry picked from commit af42c9d)
Signed-off-by: Eitan Yarmush <[email protected]>

…k/run-microvm-demo-kind.sh (agent-substrate#3509)

Since agent-substrate#2092, `deploy ate-system` stages the microVM guest assets and
applies the `microvm` SandboxConfig, so both scripts can be replaced by
`deploy ate-system` now.

This PR deletes `hack/run-microvm-demo.sh` and
`hack/run-microvm-demo-kind.sh`.

Tested on kind, this fixed the broken counter microVM demo, which failed
before due to missing `ATE_CREDENTIAL_PROVIDER`.

AI usage: test.

- [ ] Tests pass
- [ ] Docs updated

Signed-off-by: Haven Xia <[email protected]>
(cherry picked from commit 29af06e)
Signed-off-by: Eitan Yarmush <[email protected]>

…t-substrate#3510)

<!-- The PR title becomes the release note. Write it for users. -->

Follow-up to
agent-substrate#3498 (comment).
Remove `hello from: %s` and `getCurrentIP()` from
`demos/counter/counter.go`.

Why? `counter.go` included `hello from: <ip>` in its response from early
dyas of Substrate to show that an actor resumed on a different worker
Pod IP. However, now every actor sandbox uses the fixed veth address
`169.254.17.2` (`ActorVethIP` in `internal/ateworker/actornet/net.go`),
so `getCurrentIP()` always returns `169.254.17.2` regardless of which
worker Pod hosts the actor.

Also updated documentations to remove mentioning of "the different
worker Pod IP replied by counter indicates it was resumed on another
pod".

<!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author
trailers. -->
AI use: AI-assisted to update the counter demo, docs, and tests; I
reviewed the diff.

<!-- Breaking change? Describe what users must do on upgrade and ask for
the `breaking-change` label. -->

- [x] Tests pass
- [x] Docs updated

---------

Signed-off-by: Zoe Zhao <[email protected]>
(cherry picked from commit 659832f)
Signed-off-by: Eitan Yarmush <[email protected]>

…ubstrate#3355)

The security-status-report skill passes every column of
`docs/threats.json` to its evaluators, so implementation-status claims
and stale notes in the threat model were steering the scores instead of
the code. This rewrites the suggested mitigations and notes as short
design guidance with no status, and keeps threats and invariants
unchanged except:

- New Trust Boundaries section: the sandbox is the primary boundary, the
worker Pod is trusted with the actors it hosts (its own confinement is
defense in depth for the worker-to-node boundary), and a threat that
needs another control to fail first is High priority at most
- T-12 removed (documentation alone could never close it); other IDs
keep their numbers
- T-01 to T-03 lowered to High, since they require the outer network
layer to fail first
- T-11 invariant: the official path for secrets keeps them out of the
sandbox
- T-21 lowered to High, since it requires a sandbox escape first. Each
worker permission beyond the container defaults needs a documented
reason; the ones the sandbox runtime needs while workers share the host
user namespace (root, runtime capabilities, AppArmor Unconfined) are
accepted. A worker must not be able to modify or execute anything atelet
or other workers trust
- T-25 reworded to a malicious, not corrupt, snapshot, and lowered to
Medium, since it requires control of snapshot storage first (same
attackers as T-39)
- T-26: enforcement points may cache policy as long as changes take
effect within a short time, and must deny rather than serve stale policy
indefinitely. Open connections and already-issued credentials are out of
scope
- T-30 lowered to Medium with the TBD invariant replaced, since nodes
are not a boundary between actors

Skill changes: evaluators score against the invariants rather than
treating suggested mitigations as a checklist, don't penalize items a
threat puts outside Substrate's scope or that its notes accept with a
documented reason, and use a defined 0.0 to 1.0 scale. Each evaluator
also writes a short mitigation summary and remaining gap; the new
`render_table.py` renders them as a markdown table sorted by priority,
and the chart now groups threats by priority.

AI use: AI-assisted. I reviewed each section, and validated the rewrite
with before/after evaluator runs and a full skill run.

- [x] Tests pass
- [x] Docs updated

Signed-off-by: Greg Castle <[email protected]>
(cherry picked from commit 8a2c015)
Signed-off-by: Eitan Yarmush <[email protected]>

<!-- The PR title becomes the release note. Write it for users. -->

Why is this needed? (Or link the issue. Open one first for features,
API, install or CI changes.)

This updates brotli to remove a CVE: CVE-2026-111497

<!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author
trailers. -->
AI use: AI assisted, I have reviewed the code.

<!-- Breaking change? Describe what users must do on upgrade and ask for
the `breaking-change` label. -->

- [X] Tests pass
- [X] Docs updated

Signed-off-by: Max Smythe <[email protected]>
(cherry picked from commit a1ee475)
Signed-off-by: Eitan Yarmush <[email protected]>

…nt-substrate#3464)

The orchestrator ran every entry in the `--tests` file, so running one
test meant copying it into a new file first. `--test-name NAME` runs
only the named entries and can be repeated.

The orchestrator now selects and validates the entries before it waits
for Docker and clones the repo, so a mistyped name fails right away. A
relative `--tests` path now resolves against the working directory
instead of the clone.

<!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author
trailers. -->
AI use: AI-assisted, I reviewed and tested it

<!-- Breaking change? Describe what users must do on upgrade and ask for
the `breaking-change` label. -->

- [x] Tests pass
- [x] Docs updated

Signed-off-by: Haiyan Meng <[email protected]>
(cherry picked from commit 0ddbf78)
Signed-off-by: Eitan Yarmush <[email protected]>

…substrate#3223)

BindActorToWorker and the actor update ran as separate writes, and so
did ReleaseActorFromWorker and the actor update. A crash or lost lease
between them could leave a worker assignment row the actor did not name,
or an actor naming a worker that did not hold it. Recovering from that
needed stale claim lookups, rebinds, and release-only paths.

BindActorToWorker now applies the actor update in its transaction, and
UpdateActorAndRelease replaces ReleaseActorFromWorker. An actor names a
worker if and only if that worker holds its assignment. Remove the
recovery code this makes unreachable. HostingNode reads the node from
the actor's status. BindActorToWorker returns ErrWorkerNotFound for a
missing worker, so the resume can tell it from a deleted actor.

Fixes agent-substrate#1639

AI use: AI-assisted, I reviewed and tested it.

- [x] Tests pass
- [x] Docs updated

---------

Signed-off-by: Scott Funkenhauser <[email protected]>
(cherry picked from commit 74c6ab3)
Signed-off-by: Eitan Yarmush <[email protected]>

…tenet-egress (agent-substrate#3417)

`atenet-egress` runs `cmd/atenet router --mode=egress`, which previously
initialized OpenTelemetry with the hardcoded service name
`"atenet-router"`.

- Set OTel `service.name` in `RouterServer.Run` based on `--mode`
(`"atenet-egress"` for egress, `"atenet-router"` for ingress) and pass
the meter to `extproc.NewRouteDurationHistogram`.
- Remove the unused trace span in `handleStatusz`.
- Include `atenet-egress` in `ate-setup`'s `otelOverrideDeployments` so
`--otlp-endpoint` restarts it too.
- Update the metrics registry/docs and E2E metrics test to verify
`atenet-egress` telemetry is exported under `atenet-egress` and not
`atenet-router`.

Fixes agent-substrate#3308

<!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author
trailers. -->
AI use:

<!-- Breaking change? Describe what users must do on upgrade and ask for
the `breaking-change` label. -->

- [ ] Tests pass
- [ ] Docs updated

Signed-off-by: Bo-Cheng Chu <[email protected]>
(cherry picked from commit 0f7ebfa)
Signed-off-by: Eitan Yarmush <[email protected]>

…trate#2428)

## Summary

Use fixed runtime-specific worker capability sets, following maintainer
feedback that Substrate should own the minimum permissions needed to
implement its sandboxes. Every worker drops ALL container-runtime
default capabilities, adds back its runtime-specific set, and sets
`allowPrivilegeEscalation: false`.

| Component | Previous set | New set | Removed |
| --- | --- | --- | --- |
| gVisor worker | 13 capabilities | 9 capabilities (`0x802030cb`) |
`MKNOD`, `SETPCAP`, `SYS_CHROOT`, `SYS_PTRACE` |
| microVM worker | 15 capabilities | 11 capabilities (`0x882011db`) |
`DAC_READ_SEARCH`, `NET_RAW`, `SYS_CHROOT`, `SYS_PTRACE` |
| virtiofsd serving child | 9 capabilities | Its 8 default capabilities
(`0x880000db`), empty bounding set | `DAC_READ_SEARCH` |

Both workers still run as root and retain `SYS_ADMIN` and `NET_ADMIN`.
This is incremental worker hardening; the existing
deployment-permissions requirements still apply.

Fixes agent-substrate#2422. That issue asked for these settings to be configurable per
pool; following review, the workers get fixed, reviewed defaults
instead. Worker seccomp is split out to agent-substrate#2749, with the profiles in
draft agent-substrate#2750.

## Changes

- Make the tested reduced capability sets fixed defaults rather than
exposing capability drops per pool.
- Remove the proposed WorkerPool security API, its validation and
generated schema changes, and the example worker seccomp profiles and
generator, which move to agent-substrate#2750. Outer worker seccomp and AppArmor remain
Unconfined.
- Keep `SETPCAP` on microVM workers so virtiofsd's serving child can
drop its bounding set with `PR_CAPBSET_DROP`, which libcap-ng only
issues while `CAP_SETPCAP` is effective. Keep `SETFCAP` on gVisor
workers: runsc cannot start its gofer without it.
- Run virtiofsd with `--inode-file-handles=never` and stop granting
`DAC_READ_SEARCH`, which virtiofsd only needs to open file handles. The
shared directory is on the worker's overlayfs rootfs, which cannot
produce decodable file handles, so virtiofsd already used `O_PATH` fds.
virtiofsd also explicitly selects its namespace sandbox and seccomp
filter.
- Verify exact runtime defaults and reconciliation after Deployment
security settings drift.
- `TestWorkerSecurityFilesystem` checks what the kernel enforces, not
only the rendered PodSpec. A testserver `procstatus` subcommand runs in
an ephemeral container in the worker's PID namespace and reports
`/proc/<pid>/status` by process name. The test asserts the worker's
permitted, effective and bounding sets equal the fixed mask with nothing
ambient and `NoNewPrivs: 1`; runsc's gofer and sandbox run under
seccomp; on microVM, virtiofsd's serving child holds its eight
capabilities with an empty bounding set, and every cloud-hypervisor
thread except main runs under seccomp. It still covers non-root-owned
files through pause/resume and suspend/resume, and on microVM non-root
access, cross-UID denial and device-node creation.
- Update the API guide and security documentation, including when the
change rolls worker pools, and the image cache README and comments that
still described privileged workers needing `CAP_MKNOD`. Creating the 0:0
overlay whiteouts has not needed `CAP_MKNOD` since Linux 5.8.
- Rebased on upstream `main` at `0ddbf78d`.

Actor container capabilities remain configured separately through
`ActorTemplate.containers[].securityContext.capabilities`. There is no
new WorkerPool API or opt-in configuration.

## Upgrade

The new capability sets change the worker pod template, so every
WorkerPool rolls when `ate-controller` is upgraded, not only when its
worker image changes. Follow the [upgrade
runbook](docs/guides/upgrade.md) in its order:

1. Do steps 1 and 2 as usual.
2. In step 3, move every WorkerPool to the new worker image before you
upgrade `ate-controller`. A microVM pool left on an older image fails at
the next step: its virtiofsd exits with `can't apply the child
capabilities`, and actors on it cannot start or restore. Check that no
pool is left on the old image:
   ```bash
kubectl get workerpools -A -o
custom-columns='NS:.metadata.namespace,NAME:.metadata.name,CLASS:.spec.sandboxClasses[0].name,IMAGE:.spec.workerImage'
   ```
3. In step 4b, `go run ./cmd/ate-setup deploy ate-controller --config
~/ate-upgrade/new.yaml`. Unlike the runbook says for other releases,
every pool rolls once here, with the same `SIGTERM` and 30 minutes per
actor as step 3. Watch it with `kubectl get deploy -A -l
ate.dev/worker-pool`, then check one worker of each class:
   ```bash
kubectl -n $NS get pod <worker> -o
jsonpath='{.spec.containers[0].securityContext.capabilities.add}'
   ```
gVisor lists 9 capabilities and microVM 11, without `DAC_READ_SEARCH`.
Start one microVM actor to confirm virtiofsd comes up.
4. Finish steps 4c to 4e as usual.

Rollback: roll back step 4 first, as the runbook says. The pools roll
again to the old capability set, which runs both the new and old worker
images. Then roll back step 3. Rolling back step 3 alone, with the new
`ate-controller` still running, puts old microVM images on the new
template and breaks them.

Timing: if this lands in 1.0.0 there is no skew issue, because upgrades
from 0.x are reinstalls. If it lands after 1.0.0, it changes the worker
pod template within a major, which the [version skew
policy](docs/reference/version-skew-policy.md) does not allow, and needs
the `breaking-change` label.

## Validation

On the current revision (`86e684c4`, on upstream `main` at `0ddbf78d`;
`86e684c4` only changes docs, so the checks below ran on `6714b09b`):

- `hack/verify-all.sh`, `go vet` and `go test -race` on the changed
packages: passed.
- Live on EKS (Kubernetes 1.37, AL2023, kernel 6.18), reinstalled from
this revision with the v1 CRDs through `ate-setup deploy`, gVisor on
`m6i.xlarge` and microVM on `i4i.metal`. These passed on both runtimes:
`TestWorkerSecurityFilesystem`, `TestGoldenSnapshotLifecycle`,
`TestActorSnapshotLifecycle`, `TestPauseResumeActor`,
`TestRevertActorToMostRecentDurableSnapshot`,
`TestGracefulWorkerTerminationSuspend`, `TestDurableDirLifecycle`,
`TestMultipleDurableDirLifecycle`, `TestImageLayerOwnership`,
`TestImageUserAndWorkdir`, `TestCombinedVolumes` (external cases
skipped), `TestActorEgress`, `TestActorEgressHTTPS` and
`TestActorIdentity_AfterRestore_IsOwnID_NotGolden`, plus
`TestMicroVMClockAfterDelayedRestore` on microVM.
- Process state read from `/proc` on the nodes:
- gVisor worker `0x802030cb`; runsc sandbox `0x8001f` and gofer
`0x4001f`, both with seccomp.
- microVM worker `0x882011db`; virtiofsd serving child `0x880000db` with
bounding set 0 and `Seccomp: 2`; cloud-hypervisor threads other than
main at `Seccomp: 2`.
  - Both workers have no ambient capabilities and `NoNewPrivs: 1`.
- Upgrade order: a WorkerPool on a microVM worker image from `main`
under this `ate-controller` fails with `can't apply the child
capabilities`, and recovers when moved back to this revision's image.
- A syscall probe in the gVisor worker container gets `EPERM` from
`userfaultfd`, which needs `CAP_SYS_PTRACE` while
`vm.unprivileged_userfaultfd` is 0. It reached the kernel with the
previous capability set.
- The kind e2e lanes in `pr-workflow` cover both sandbox classes; they
need a maintainer to approve the run on this fork PR.

[Test plan and findings for the earlier opt-in revision
(PDF)](https://github.com/tjorourke/substrate/blob/7e3040822b85ac0de2703bc8268e4e7f78f26e11/docs/validation/worker-security-test-report.pdf)

- [x] Tests pass
- [x] Appropriate changes to documentation are included in the PR

AI-assisted.

---------

Signed-off-by: Tom O'Rourke <[email protected]>
(cherry picked from commit bd1f188)
Signed-off-by: Eitan Yarmush <[email protected]>

…hs (agent-substrate#3249)

<!-- The PR title becomes the release note. Write it for users. -->

> Why is this needed? (Or link the issue. Open one first for features,
API, install or CI changes.)

On microVM, an actor’s container runs as uid 0 in the guest kernel.
Before this PR, the worker sent kata-agent empty masked and read-only
path lists. That left the guest’s `/proc/sys` writable: the kernel lets
uid 0 write sysctls without any capability.

The kata-agent skips masked paths that are directories. Of the five
directories, only `/proc/acpi` and `/sys/firmware` exist in the kata
4.3.1 guest kernel. We mount read-only tmpfs similar to
[runc](https://github.com/opencontainers/runc/blob/4ca628d1d4c974f92d24daccb901aa078aad748e/libcontainer/rootfs_linux.go#L1250-L1258).

**Breaking change**: a uid 0 entrypoint can no longer write guest
sysctls, such as `sysctl -w vm.max_map_count=...`. This already fails on
gVisor. Goldens and checkpoints taken before this change keep a writable
`/proc/sys` until they are re-taken.

Part of agent-substrate#3226.

<!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author
trailers. -->
AI use: AI-assisted, I reviewed and tested it.

<!-- Breaking change? Describe what users must do on upgrade and ask for
the `breaking-change` label. -->

- [x] Tests pass
- [ ] Docs updated

---------

Signed-off-by: Huy Pham <[email protected]>
(cherry picked from commit 4488a66)
Signed-off-by: Eitan Yarmush <[email protected]>