[release-1.0] Cherry-pick 19 main changes through 5a5b19219 - #3571

…t-substrate#3498)

`demos/counter/README.md` hid every walkthrough command inside 16
collapsed `<details>` blocks (leaving `<details>` tags in the docsite's
raw Markdown outputs) and had outdated or incomplete instructions across
the default, external volume, and microVM flows:

- Inline the default `gvisor` walkthrough commands in the numbered
steps, note the default `gvisor-default` sandbox upfront, move actor
deletion after `## Reaching a non-default port`, and place the `microVM`
and external CSI volume variants in dedicated sections at the end.
- Document the `atenet-router` `--ingress-allowed-spiffe-ids`
prerequisite for `kubectl ate proxy actor`, keeping
`ATE_INSTALL_KIND=true` exported on `kind`, and point the Quickstart
link to `docs/get-started/quickstart.md`.
- Add the missing `ExternalVolumes` preview gate, `ate-setup setup csi`,
`delete demo counter` and `kubectl wait --for=delete
namespace/ate-demo-counter` before replacing the template, and `kubectl
ate --preview create actor` steps for external volumes, and explain
`--validate-existing-file-path=/external-data/test.txt`.
- Add the `create`, `proxy`, `suspend`, and `delete` steps for `microVM`
in `ate-demo-counter-microvm` (`my-counter-1`).
- Document all three counters incremented by `counter.go` (`preserved
memory count`, `preserved file counter`, and `preserved rootfs counter`)
with expected `curl` output across suspend and resume.

Fixes agent-substrate#3492

AI use: AI-assisted, I reviewed and verified the changes on `kind` and
Kubernetes clusters and built the docsite locally.

- [x] Tests pass: docs-only change (`./hack/site.sh build`,
`./hack/verify/boilerplate.sh`, and live walkthrough verification)
- [x] Docs updated

---------

Signed-off-by: Mofi Rahman <[email protected]>
(cherry picked from commit 30a0dea)
Signed-off-by: Eitan Yarmush <[email protected]>

…rate#3511)

`TestActorSnapshotLifecycle` created a `Tag` and resumed a clone from
it, but did not verify in e2e that the tag's snapshot copy survives
deleting the source actor or that the clone's own snapshot survives
`DeleteTag`.
Delete `sourceName` before creating the clone from `tagRef`, and after
the clone runs, suspend it, wait for background detach, delete `tagRef`,
and resume the clone to verify its counters continue at `(3, 3)`.

AI use:
AI-assisted, I reviewed and tested it

- [x] Tests pass
- [ ] Docs updated

---------

Signed-off-by: Nayana Bidari <[email protected]>
(cherry picked from commit 57fab75)
Signed-off-by: Eitan Yarmush <[email protected]>

…ite (agent-substrate#2775)

A new `apitool generate-docs` now generates reference documentation of
the Substrate API.

This PR has 4 commits:

* Introduce changes to the style guide on how to write doc comments and
a new `doc-format` linter rule to enforce them.
* Prefactor: move methods around to reuse it in `apitool generate-docs`.
* Add `apitool generate-docs` command.
* Make `hack/site.sh content` run the generator.

For now, only `latest` version is included in the site. This is
controlled by the `releases-since` flag in `hack/site content` script.
Once `v1.0.0` is available it will be picked up.

AI use: AI-assisted. Manually reviewed, edited and tested it.

---------

Signed-off-by: Julian Gutierrez Oschmann <[email protected]>
(cherry picked from commit 8b2bb32)
Signed-off-by: Eitan Yarmush <[email protected]>

…gent-substrate#3119)

Part of agent-substrate#1464.

- Rewrites `docs/guides/authentication.md` around who is calling
ate-api: a person running `kubectl ate`, a pod in the Substrate cluster
or another cluster, or a CI job, so people stop leaning on the shared
`ate-client` token.
- Adds pod certificates as an option for client pods (the benchmarks use
them), the `ate.dev/api-access` namespace label the ate-system
NetworkPolicy requires, and the Kubernetes RBAC `kubectl ate` needs to
reach ate-api.
- Warns where an identity is weaker than it looks: another cluster's
issuer can mint `ate-client`, GKE's Kubernetes Engine Developer and
basic Editor roles can act as any ServiceAccount, ate-system
port-forward rights reach loopback-only Envoy admin APIs that can stop
the data plane, and GitHub and Google tokens authenticate anyone.
- Moves the per-user ServiceAccount recipe and the signing-in and
install-time-provider gaps out of `authorization.md`, and drops its
claim that a Secret-based token works there (ate-api rejects its
issuer). The recipe and RBAC use agent-substrate#2851's names (`ate-users`,
`ate-api-client`).

For reviewers: agent-substrate#2851 (`--credential-plugin`) rewrites the same sections,
and its paragraph calling GitHub and Google tokens unsafe needs to match
this one.

Testing: checked each claim against the code on main (GitHub, GKE and
Kubernetes behavior against their docs or source) and that the relative
links resolve. None of the commands or manifests ran on a cluster.

AI use: AI-assisted.

- [x] Tests pass
- [x] Docs updated

---------

Signed-off-by: Max Thompson <[email protected]>
(cherry picked from commit f1973c1)
Signed-off-by: Eitan Yarmush <[email protected]>

Supersedes agent-substrate#3396 by @stshrive, rebased onto current `main`. The commit
keeps his authorship and sign-off.

The GKE provisioning and teardown tooling now lives in
https://github.com/ai-on-gke/substrate-gke under tools/setup-gcp, so
core no longer carries a copy. Docs and comments that pointed at the
removed files now point at substrate-gke. go mod tidy drops the GCP
client modules only setup-gcp used, and vendor/ and _LICENSES/ are
regenerated to match.

Covers agent-substrate#3390, except that hack/teardown.sh is removed here as well;
substrate-gke's tools/setup-gcp/teardown.sh replaces it.

Part of agent-substrate#2304

### Rebase notes
Conflicts were mostly from the Hugo docsite move (agent-substrate#1974):
- `tools/setup-gcp/cloud-sql.md`: deleted, as in the original PR.
- `docs/guides/operator-install.md`, `docs/guides/postgres.md`: the
original edits, re-applied at the new paths.
- `docs/guides/observability.md`: kept main's version, since agent-substrate#1974
already removed the Dashboards section the original PR edited.
- `cmd/kubectl-ate/README.md`: kept main's stub that points to
`docs/reference/kubectl-ate.md`. That page no longer has the setup-gcp
line, so the edit had nowhere to go.

---------

Signed-off-by: Steven Shriver <[email protected]>
Signed-off-by: Grant McCloskey <[email protected]>
Co-authored-by: Steven Shriver <[email protected]>
(cherry picked from commit b66e16a)
Signed-off-by: Eitan Yarmush <[email protected]>

…te#3147)

Split out of the original version of this PR per the review comment: one
area per PR, table-driven where the set-up is shared.

Covers what a suspended actor's v1-stamped MEMORY snapshot restores at
after its pool's SandboxConfig changes, on both lanes:

| Edit | Restores at | Next snapshot stamped |
|---|---|---|
| default moves to v2, v1 still Enabled | MEMORY on v1 via
restorable_runtimes | v1 |
| v1 Disabled | ROOTFS on v2 | v2 |
| v1 Disabled, actor moved to another template with UpdateActor |
VOLUMES on v2 (template replacement takes precedence) | v2 |
| version names swapped (bundles unchanged, with duplicate aliases) |
MEMORY; worker lists the hash once | v1 |
| assets edited under the same version name | ROOTFS | new hash |
| Tag clone after v1 Disabled | ROOTFS on v2 | v2 |

The fixture (`setupSuspendedV1Actor`, `updateVersions`, `waitForWorker`,
`resumeAndCheck`) is what the follow-up PRs build on: paused actor
across version changes, scheduler preferring the compatible pool,
WorkerPool configRef changes and bad asset SHA256.

The controlapi unit test change seeds a worker with a different default
so the snapshot-stamp assertion actually discriminates.

AI-assisted, I reviewed and tested it.

- [x] Tests pass
- [x] Docs updated

---------

Signed-off-by: Shrutiya <[email protected]>
(cherry picked from commit 24482d5)
Signed-off-by: Eitan Yarmush <[email protected]>

… changes (agent-substrate#3370)

`EnsureAPIServerPrerequisites` created
`ConfigMap/ate-api-authentication` and `ConfigMap/ate-api-config` only
when missing (unless `--ateapi-log-level` was passed), ignoring
`--ateapi-expected-jwt-issuer` on re-deploy and leaving legacy or
unknown `1.x` rollback fields in place where `ate-api-server`'s strict
YAML/prototext loaders would `CrashLoopBackOff`. Additionally,
`Deployment/ate-api-server`'s `ate.dev/env-hash` annotation only
digested environment variables rather than the mounted
`ate-api-authentication` and `ate-api-config` ConfigMaps.

Fixes agent-substrate#3271

## Summary

```text
EnsureAPIServerPrerequisites
  ensureAPIAuthenticationConfig -> ateapiauth.ParseAuthenticationConfig
    (rewrites when missing, when --ateapi-expected-jwt-issuer is set, or when stored YAML fails strict schema validation; preserves valid custom issuers and extra jwtProviders)
  ensureAPIConfig -> apiconfig.ParseTextProto
    (rewrites when missing, when --ateapi-log-level is set, or when stored prototext fails strict schema/level validation; preserves valid customized log levels)

DeployAPIServer / UpdateAPIAuthenticationConfig / UpdateAPIConfig
  apiServerEnvHash(env, ate-api-authentication, ate-api-config) -> annotates Deployment/ate-api-server pod template (ate.dev/env-hash)
```

## Evidence

- **Before:** Re-running `ate-setup deploy ate-system
--ateapi-expected-jwt-issuer <issuer>` skipped
`ConfigMap/ate-api-authentication`, and upgrading across schema removals
or rolling back `1.x -> 1.0.0` with unknown fields in
`authentication.yaml` or `apiconfig.textproto` left `ate-api-server` in
`CrashLoopBackOff`.
- **After:** Valid customized ConfigMaps (custom JWT issuers, extra
external `jwtProviders`, hand-edited log levels) survive `ate-setup
deploy` untouched when flags are unset; explicit flags or
invalid/unknown rollback fields rewrite the ConfigMaps cleanly and
update `ate.dev/env-hash`, rolling `Deployment/ate-api-server` to
`Ready`.

## Merge Danger

**Door:** Two-way

**Blast Radius:** Small (`internal/ateapiauth`, `internal/apiconfig`,
and `cmd/ate-setup` prerequisite reconciliation and `ate.dev/env-hash`
computation)

## Rationale

`api-server` crashes on any config field it doesn't recognize, and
`ate-setup` never touched those config maps once they existed. the PR
makes `ate-setup` rewrite the config maps when the server can't parse
it, and keeps valid custom ones.

However this need to be in `1.0,0` otherwise if `1.0.x` adds a config
field, then someone rollback from `1.0.x` to `1.0.0`, the old server
will fail the rollback as it cannot parse the yaml.

AI use: AI-assisted implementation and unit tests; reviewed and verified
by @moficodes on Kind and GKE.

- [x] Tests pass
- [x] Docs updated

---------

Signed-off-by: Mofi Rahman <[email protected]>
(cherry picked from commit 671cb69)
Signed-off-by: Eitan Yarmush <[email protected]>

…substrate#1607)

ateapi stores resources as binary protobuf in `bytea` columns. Binary
protobuf identifies a field only by its number. If a field is
renumbered, existing rows lose its value. If a retired number is reused,
existing rows decode into the wrong field. `atepg.go` unmarshals with
`DiscardUnknown: true`, so the original bytes are dropped on the next
read-modify-write. Nothing reports this at runtime.

- `hack/tools/buf`: adds buf v1.72.0 as a tool module, run through
`hack/run-tool.sh`.
- `buf.yaml`: checks the protos under `api/`, `internal/proto/`, and
`pkg/proto/` with buf's `PACKAGE` breaking rules. Within a proto
package, declarations may move between `.proto` files, but messages,
enums, services, RPCs, fields, enum values, and reserved ranges cannot
be deleted, renamed, renumbered, or retyped.
- `hack/verify/proto-breaking.sh`: runs `buf breaking` on the working
tree against the merge base of `HEAD` and `origin/main` (or `main`). CI
checks out with `fetch-depth: 0`, so the merge base is available there.
It passes `--against-config buf.yaml` so a baseline commit without
`buf.yaml` still builds.
- `AGENTS.md`: removes the Backward Compatibility section. Breaking
changes are no longer acceptable once this check is in.
- `hack/verify-all.sh`: runs every verifier and lists the failures at
the end, instead of stopping at the first one. This is to make sure the
CI runs every verifier on a PR including breaking proto changes. I can
split this into its own PR if you prefer.

I used Claude Code to help write the change and this description.

---------

Signed-off-by: Haiyan Meng <[email protected]>
(cherry picked from commit ae3ac05)
Signed-off-by: Eitan Yarmush <[email protected]>

Why is this needed?

The existing threat model diagram was out of date and the security guide
lacked a high-level overview diagram illustrating Substrate's core
isolation boundaries.

This PR:
1. Adds `docs/assets/security-boundaries.svg` and a new "Security
Boundaries" section to `docs/security.md`, defining the four core
isolation boundaries (sandbox, gateways, control plane, storage) held by
Substrate.
2. Replaces the single threat model diagram with two coordinated SVGs:
- `docs/assets/actor-template-diagram.svg`: Details actor template
creation (T1–T10), from image pull and golden actor warm-up to suspend,
snapshot upload, and template readiness.
- `docs/assets/threat-model-diagram.svg`: Details actor creation (A1–A2)
and the lifecycle of an incoming request through outbound tool
invocation (B1–B11).
3. Adds numbered step tables (`Step | What happens | What's checked`)
below each diagram in `docs/threat-model.md`.
4. Refreshes the Key Components list in `docs/threat-model.md` to
reflect current components (including `ate-idp-server`,
`k8s-credential-provider`, and `atunnel`), sorted alphabetically.

AI use:
AI-assisted, I reviewed and tested it.

- [x] Tests pass
- [x] Docs updated

---------

Signed-off-by: Greg Castle <[email protected]>
(cherry picked from commit dc039f1)
Signed-off-by: Eitan Yarmush <[email protected]>

…upper back from disk (agent-substrate#3485)

Why: In the microVM burn-in (GKE downstream testing), wake p50 went from
~350ms to ~1.1s (!), almost all of it in upper_join. Reported to me by
@annapendleton though I'm not sure who originally found it.

agent-substrate#3353 started writing out snapshot tars as they're written, so their
dirty pages can't run a tight worker pod out of memory. It also dropped
the written pages from the page cache, so a resume on the same node read
the rootfs upper tar back from disk ... which is slow. It also made
every suspend write the tars at disk speed.

Fixes by commit order:
1. Pull summing memory.stat keys out of memoryUsed, no behavior change.
2. Clean pages are reclaimable, so dropping them is irrelevant for OOM
safety.
- The tars now stay cached. Write-out waits on the previous chunk
instead of the one just written, so dirty pages stay at about two chunks
without stalling on each one.
4. Start snapshot writeback after the OnDemand delta merge, which
replaces memory-ranges, so it tracks the merged file instead of the
unlinked delta.
5. Write snapshot files back in the background instead of with a
blocking fdatasync.
- Starting write-out can block for a long time on a busy disk
(https://github.com/torvalds/linux/blob/v6.12/fs/sync.c#L315-L317), and
holding deleted snapshot files open meanwhile leaked disk under load, so
it starts write-out once, 64MiB at a time, and closes a file as soon as
it's unlinked, which drops its dirty pages.
7. What snapshot writes can borrow leaves clean page cache out, so
cached tars don't make a pod look "full".
- A worker pod without a memory limit (e.g. the deploy_locust.sh
default) has nothing to protect, so it lends all of it instead of
capping every suspend as if the pod were full.
9. When the pod can lend memory for the whole archive, the tar skips the
write-behind and goes to the page cache at full speed, and the loan
comes back once it's on disk. Otherwise the write-behind still bounds
its dirty pages.

That said, with respect to workers without limits: We should probably
consider recommending memory limits on worker pods. I will think about
it overnight and propose docs updates tomorrow.

Tested with locust burn-in tests (see comments in the PR).

AI use: AI-assisted, I reviewed and tested it.

- [x] Tests pass
- [x] Docs updated

---------

Signed-off-by: Benjamin Elder <[email protected]>
(cherry picked from commit a6442f9)
Signed-off-by: Eitan Yarmush <[email protected]>

…gent-substrate#3447)

Part of agent-substrate#3095. The objectstore-plugin exchange read actor JWT keys from
a file, a stopgap from before ate-idp-server. agent-substrate#3358 needs an issuer
instead, so its `pkg/` API takes no key file
([thread](agent-substrate#3358 (comment))).

- The exchange fetches keys through
`<issuer>/.well-known/openid-configuration`. `--actor-jwt-jwks-file`,
`oidcjwt.FileKeySource` and the `actor-id-jwt-jwks` volume are removed.
- The sidecar verifies the issuer's certificate against the system roots
plus `--actor-jwt-issuer-ca-file`, which is new and defaults to the
service DNS trust bundle. The default ate-idp-server issuer verifies
with or without an explicit `:443`, and a custom issuer can use a public
certificate. ate-setup writes no CA setting for it. The sidecar re-reads
the bundle when it changes and keeps the last good pool if a re-read
fails. ate-api-server's `JWTProvider` CA files use the same reloading
client and still replace the system roots there.
- Nothing reads the pool Secret's `jwks` key now, so its writers are
removed: the `admin` commands of kubectl-ate, ate-setup, `PublicJWKS`
and `ConcretePool.JWKS`. Existing Secrets keep the stale key.
- The sidecar fetches keys once at start. It exits on a certificate
error and only warns on other errors, so an unreachable issuer fails
exchanges with `UNAVAILABLE` until a fetch succeeds. Cached keys refresh
after 5m, a failed refresh retries after 1m, and a token with an unknown
key ID triggers at most one fetch per 1m. Until a fetch completes,
including one the caller canceled, such a token gets `UNAVAILABLE`, so a
newly rotated key ID is retried.

Upgrade: drop `--actor-jwt-jwks-file` from custom sidecar configs. Set
`--actor-jwt-issuer-ca-file` only if the issuer uses a private CA other
than the service DNS one.

Tests pass for `oidcjwt`, `credbundle`, `objectstoreplugin`,
`cmd/objectstore-plugin`, `cmd/ate-setup` and `apiauthn`. The
snapshotcredentials e2e needed no change. It compiles with `-tags=e2e`
but has not run, as it needs GCS.

- [x] Tests pass
- [x] Docs updated

---------

Signed-off-by: Anish Ramasekar <[email protected]>
(cherry picked from commit 92040fe)
Signed-off-by: Eitan Yarmush <[email protected]>

…bstrate#3533)

Part of agent-substrate#3226.

`gvisor` and `microvm` previously mounted different filesystems at
`/dev/shm`, and `/tmp`:
- `/dev/shm` on gVisor was a `0755` directory on `runsc`'s internal
`dev` filesystem, blocking non-root shared memory writes.
- `/tmp` was part of the host-backed rootfs upper layer on microVM and a
conditional `tmpfs` on gVisor when the image's `/tmp` was empty.
- Dropped the `/run` tmpfs from both runtimes, so `/run` comes from the
image's rootfs, as on Kubernetes.

Move the standard OCI system mounts (`/proc`, `/dev`, `/dev/pts`,
`/dev/shm`, `/dev/mqueue`, `/sys`) into `ocispec.Build` and add an
explicit `mode=1777` `tmpfs` mount at `/tmp`. If a container mounts a
volume at a system mount destination such as `/tmp`, `ocispec.Build`
replaces the default `tmpfs` entry there. `ShapeMicroVM` now rewrites
`bind` mounts in place.

AI use:
AI-assisted, I reviewed and tested it.

Breaking change:

- Old gVisor MEMORY snapshots, including goldens, don't restore. Recut
golden snapshots after upgrading.
- MicroVM MEMORY snapshots and goldens still resume, but keep the old
mounts until they are recut.
- MicroVM `/tmp`: now a tmpfs in guest memory instead of the host-backed
rootfs.
  - Its files count against the actor's memory limit.
  - It is capped at half of the guest's memory.
- ROOTFS snapshots no longer keep it, and files older ROOTFS snapshots
saved there are hidden.
  - Files the image ships under `/tmp` are hidden.
- gVisor `/dev/shm`: now a 64 MiB tmpfs (`mode=1777`) instead of a
root-only directory. Non-root processes can now use it, but it is capped
at 64 MiB.
- gVisor `/tmp`: now always a tmpfs. runsc used to mount one only when
the image's `/tmp` was empty, so files the image ships under `/tmp` are
now hidden.

If an actor needs more space in `/dev/shm` or `/tmp`, or needs files
there kept in ROOTFS snapshots, mount a `durableDir` volume at that
path. It replaces the default tmpfs. Images that ship files under `/tmp`
should create them at startup instead.

- [x] Tests pass
- [x] Docs updated

---------

Signed-off-by: Nayana Bidari <[email protected]>
(cherry picked from commit fd576d3)
Signed-off-by: Eitan Yarmush <[email protected]>

…e#3450)

Concurrency tests in agent-substrate#3255 showed that a request authorized against a
deleted
atespace can apply to a new atespace with the same name. A previous
owner can
regain `owner` on it or delete it.

We decided not to fix this in code. Atespace names must not be reused,
and
nothing enforces it. This documents the limitation in the authorization
guide
and qualifies the claim that a recreated atespace starts with no policy.

I used an AI assistant to analyze the issue and draft the doc text, and
reviewed the result.

---------

Signed-off-by: Shruti Nair <[email protected]>
(cherry picked from commit 9d16b69)
Signed-off-by: Eitan Yarmush <[email protected]>

…ate#3410)

Part of agent-substrate#1661.

- Documents the JSON array form of ate-setup's `--credential-provider`
in `docs/guides/egress-credential-injection.md` and
`cmd/ate-setup/commands.md`, and that the gateway sends each
`ate-secret://` URI to the provider its host names.
- Updates comments that still described one provider: the egress
manifest's marker comment and
`cmd/atenet/internal/router/egress/provider.go`.

Docs and comments only, so it can merge during the freeze.

AI use: AI-assisted draft, reviewed by me.

- [x] Tests pass (no code changes)
- [x] Docs updated

Signed-off-by: Max Thompson <[email protected]>
(cherry picked from commit ac8c621)
Signed-off-by: Eitan Yarmush <[email protected]>

…e#3553)

`delete demo` returned while the demo namespace was still terminating,
so a `deploy demo` right after it failed: the API server refuses new
objects in a terminating namespace. Delete now waits for the namespace
to be gone, as `delete ate-system` does.

Why is this needed? (Or link the issue. Open one first for features,
API, install or CI changes.)

Demo Fix.

Fixes agent-substrate#3525

AI use: I wrote, claude polished.

- [ ] Tests pass
- [x] Docs updated

Signed-off-by: Haven Xia <[email protected]>
(cherry picked from commit e99dffb)
Signed-off-by: Eitan Yarmush <[email protected]>

The claude-code-multiplex demo doesn't work as written:

1. `deploy demo claude-code-multiplex` creates the three agent templates
but no actors, so the dashboard has no agents and "Give a task" fails
with `no agents available in namespace`.
2. Even with actors created by hand, the agents can't reach the
Anthropic API. Every actor starts with **no egress policy**, and the
egress gateway blocks all traffic.
3. The WorkerPool sets no resource limits, so each pod advertises the
whole node and all three agents run at once. The demo is about a third
agent waiting for a free pod, and that never happens.

How this PR fixes it:

- It builds on Adam's change from agent-substrate#2982 is folded in, with him as
co-author. Deploy creates the three actors, the dashboard resumes an
agent when its task starts and suspends it when the task ends, and each
pod is sized for one agent.
- When both pods are busy, a new task stays queued until one finishes,
instead of suspending an agent mid-task.
- Gives each agent an egress policy that lets TLS to
api.anthropic.com:443 pass through the gateway.
- The pod memory is 2560Mi (request = limit), not 2Gi. The worker keeps
part of it for itself, so a 2Gi agent didn't fit.
- Update the README to match the code.

agent-substrate#2488 by @Ziy1-Tan documents the same setup as manual steps. With this
PR those steps aren't needed.

Testing: tested on kind.

AI use: Claude assisted and I reviewed

- [x] Tests pass
- [x] Docs updated

Signed-off-by: Haven Xia <[email protected]>
Co-authored-by: Adam Wathieu <[email protected]>
(cherry picked from commit 43dc887)
Signed-off-by: Eitan Yarmush <[email protected]>