Repository navigation
[release-1.0] Cherry-pick 21 main changes through fc931c4e2 - #3596
Merged
Dmitry Berkovich (dberkov) merged 21 commits intoNov 3, 2026
Conversation
…alkthroughs, and counter volume (agent-substrate#3500) The `sandbox`, `jupyter`, and `multi-template` tutorials linked to retired root `README.md` anchors, omitted required ingress and `ko` settings, and had walkthrough steps out of sync with current CLI and runtime behavior. `demos/multi-template/counter-template.yaml.tmpl` also omitted the `/home/counter` volume mount, so `counter` printed `preserved file counter: -1`. - Mount a `durableDir` volume at `/home/counter` in `demos/multi-template/counter-template.yaml.tmpl` and update the expected `curl` output for all three counters. - List the `atenet-router` `--ingress-allowed-spiffe-ids`, `SNAPSHOT_LOCATION`, and `KO_DOCKER_REPO` (`ko.dockerRepo`) prerequisites in all three tutorials (including the `ATE_INSTALL_KIND=true` defaults on `kind`), and point their install links to `docs/get-started/quickstart.md` and `docs/guides/operator-install.md`. - Clarify per-request subprocess execution in `demos/sandbox/README.md` (`sandbox-client` sends `sh -c` to `/process`), add a non-interactive piped `sandbox-client` command, and remove the duplicate `go install ./cmd/kubectl-ate` step. - Update `demos/jupyter/README.md` and `cmd/ate-setup/internal/demos/jupyter/jupyter.go` to remove unimplemented idle-suspend claims, use `quay.io/jupyter/base-notebook` and `ACTOR_STATE_SUSPENDED`, add a `curl -s http://localhost:8888/api` check, fix step numbering, and document full teardown. - Drop the retired actor DNS routing claim and inaccurate delete-suspend note in `demos/multi-template/README.md`, and stop the background `kubectl ate proxy actor` processes during cleanup. Fixes agent-substrate#3494 AI use: AI-assisted, I reviewed and verified the changes on `kind` and Kubernetes clusters and built the docsite locally. - [x] Tests pass (`./hack/verify/gofmt.sh`, `./hack/verify/boilerplate.sh`, `go test -race ./cmd/ate-setup/...`, `./hack/site.sh build`, and live walkthrough verification) - [x] Docs updated --------- Signed-off-by: Mofi Rahman <[email protected]> (cherry picked from commit a2cc5e4) Signed-off-by: Eitan Yarmush <[email protected]>
…t-substrate#3498) `demos/counter/README.md` hid every walkthrough command inside 16 collapsed `<details>` blocks (leaving `<details>` tags in the docsite's raw Markdown outputs) and had outdated or incomplete instructions across the default, external volume, and microVM flows: - Inline the default `gvisor` walkthrough commands in the numbered steps, note the default `gvisor-default` sandbox upfront, move actor deletion after `## Reaching a non-default port`, and place the `microVM` and external CSI volume variants in dedicated sections at the end. - Document the `atenet-router` `--ingress-allowed-spiffe-ids` prerequisite for `kubectl ate proxy actor`, keeping `ATE_INSTALL_KIND=true` exported on `kind`, and point the Quickstart link to `docs/get-started/quickstart.md`. - Add the missing `ExternalVolumes` preview gate, `ate-setup setup csi`, `delete demo counter` and `kubectl wait --for=delete namespace/ate-demo-counter` before replacing the template, and `kubectl ate --preview create actor` steps for external volumes, and explain `--validate-existing-file-path=/external-data/test.txt`. - Add the `create`, `proxy`, `suspend`, and `delete` steps for `microVM` in `ate-demo-counter-microvm` (`my-counter-1`). - Document all three counters incremented by `counter.go` (`preserved memory count`, `preserved file counter`, and `preserved rootfs counter`) with expected `curl` output across suspend and resume. Fixes agent-substrate#3492 AI use: AI-assisted, I reviewed and verified the changes on `kind` and Kubernetes clusters and built the docsite locally. - [x] Tests pass: docs-only change (`./hack/site.sh build`, `./hack/verify/boilerplate.sh`, and live walkthrough verification) - [x] Docs updated --------- Signed-off-by: Mofi Rahman <[email protected]> (cherry picked from commit 30a0dea) Signed-off-by: Eitan Yarmush <[email protected]>
…rate#3511) `TestActorSnapshotLifecycle` created a `Tag` and resumed a clone from it, but did not verify in e2e that the tag's snapshot copy survives deleting the source actor or that the clone's own snapshot survives `DeleteTag`. Delete `sourceName` before creating the clone from `tagRef`, and after the clone runs, suspend it, wait for background detach, delete `tagRef`, and resume the clone to verify its counters continue at `(3, 3)`. AI use: AI-assisted, I reviewed and tested it - [x] Tests pass - [ ] Docs updated --------- Signed-off-by: Nayana Bidari <[email protected]> (cherry picked from commit 57fab75) Signed-off-by: Eitan Yarmush <[email protected]>
…ite (agent-substrate#2775) A new `apitool generate-docs` now generates reference documentation of the Substrate API. This PR has 4 commits: * Introduce changes to the style guide on how to write doc comments and a new `doc-format` linter rule to enforce them. * Prefactor: move methods around to reuse it in `apitool generate-docs`. * Add `apitool generate-docs` command. * Make `hack/site.sh content` run the generator. For now, only `latest` version is included in the site. This is controlled by the `releases-since` flag in `hack/site content` script. Once `v1.0.0` is available it will be picked up. AI use: AI-assisted. Manually reviewed, edited and tested it. --------- Signed-off-by: Julian Gutierrez Oschmann <[email protected]> (cherry picked from commit 8b2bb32) Signed-off-by: Eitan Yarmush <[email protected]>
…gent-substrate#3119) Part of agent-substrate#1464. - Rewrites `docs/guides/authentication.md` around who is calling ate-api: a person running `kubectl ate`, a pod in the Substrate cluster or another cluster, or a CI job, so people stop leaning on the shared `ate-client` token. - Adds pod certificates as an option for client pods (the benchmarks use them), the `ate.dev/api-access` namespace label the ate-system NetworkPolicy requires, and the Kubernetes RBAC `kubectl ate` needs to reach ate-api. - Warns where an identity is weaker than it looks: another cluster's issuer can mint `ate-client`, GKE's Kubernetes Engine Developer and basic Editor roles can act as any ServiceAccount, ate-system port-forward rights reach loopback-only Envoy admin APIs that can stop the data plane, and GitHub and Google tokens authenticate anyone. - Moves the per-user ServiceAccount recipe and the signing-in and install-time-provider gaps out of `authorization.md`, and drops its claim that a Secret-based token works there (ate-api rejects its issuer). The recipe and RBAC use agent-substrate#2851's names (`ate-users`, `ate-api-client`). For reviewers: agent-substrate#2851 (`--credential-plugin`) rewrites the same sections, and its paragraph calling GitHub and Google tokens unsafe needs to match this one. Testing: checked each claim against the code on main (GitHub, GKE and Kubernetes behavior against their docs or source) and that the relative links resolve. None of the commands or manifests ran on a cluster. AI use: AI-assisted. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Max Thompson <[email protected]> (cherry picked from commit f1973c1) Signed-off-by: Eitan Yarmush <[email protected]>
Supersedes agent-substrate#3396 by @stshrive, rebased onto current `main`. The commit keeps his authorship and sign-off. The GKE provisioning and teardown tooling now lives in https://github.com/ai-on-gke/substrate-gke under tools/setup-gcp, so core no longer carries a copy. Docs and comments that pointed at the removed files now point at substrate-gke. go mod tidy drops the GCP client modules only setup-gcp used, and vendor/ and _LICENSES/ are regenerated to match. Covers agent-substrate#3390, except that hack/teardown.sh is removed here as well; substrate-gke's tools/setup-gcp/teardown.sh replaces it. Part of agent-substrate#2304 ### Rebase notes Conflicts were mostly from the Hugo docsite move (agent-substrate#1974): - `tools/setup-gcp/cloud-sql.md`: deleted, as in the original PR. - `docs/guides/operator-install.md`, `docs/guides/postgres.md`: the original edits, re-applied at the new paths. - `docs/guides/observability.md`: kept main's version, since agent-substrate#1974 already removed the Dashboards section the original PR edited. - `cmd/kubectl-ate/README.md`: kept main's stub that points to `docs/reference/kubectl-ate.md`. That page no longer has the setup-gcp line, so the edit had nowhere to go. --------- Signed-off-by: Steven Shriver <[email protected]> Signed-off-by: Grant McCloskey <[email protected]> Co-authored-by: Steven Shriver <[email protected]> (cherry picked from commit b66e16a) Signed-off-by: Eitan Yarmush <[email protected]>
…te#3147) Split out of the original version of this PR per the review comment: one area per PR, table-driven where the set-up is shared. Covers what a suspended actor's v1-stamped MEMORY snapshot restores at after its pool's SandboxConfig changes, on both lanes: | Edit | Restores at | Next snapshot stamped | |---|---|---| | default moves to v2, v1 still Enabled | MEMORY on v1 via restorable_runtimes | v1 | | v1 Disabled | ROOTFS on v2 | v2 | | v1 Disabled, actor moved to another template with UpdateActor | VOLUMES on v2 (template replacement takes precedence) | v2 | | version names swapped (bundles unchanged, with duplicate aliases) | MEMORY; worker lists the hash once | v1 | | assets edited under the same version name | ROOTFS | new hash | | Tag clone after v1 Disabled | ROOTFS on v2 | v2 | The fixture (`setupSuspendedV1Actor`, `updateVersions`, `waitForWorker`, `resumeAndCheck`) is what the follow-up PRs build on: paused actor across version changes, scheduler preferring the compatible pool, WorkerPool configRef changes and bad asset SHA256. The controlapi unit test change seeds a worker with a different default so the snapshot-stamp assertion actually discriminates. AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Shrutiya <[email protected]> (cherry picked from commit 24482d5) Signed-off-by: Eitan Yarmush <[email protected]>
… changes (agent-substrate#3370) `EnsureAPIServerPrerequisites` created `ConfigMap/ate-api-authentication` and `ConfigMap/ate-api-config` only when missing (unless `--ateapi-log-level` was passed), ignoring `--ateapi-expected-jwt-issuer` on re-deploy and leaving legacy or unknown `1.x` rollback fields in place where `ate-api-server`'s strict YAML/prototext loaders would `CrashLoopBackOff`. Additionally, `Deployment/ate-api-server`'s `ate.dev/env-hash` annotation only digested environment variables rather than the mounted `ate-api-authentication` and `ate-api-config` ConfigMaps. Fixes agent-substrate#3271 ## Summary ```text EnsureAPIServerPrerequisites ensureAPIAuthenticationConfig -> ateapiauth.ParseAuthenticationConfig (rewrites when missing, when --ateapi-expected-jwt-issuer is set, or when stored YAML fails strict schema validation; preserves valid custom issuers and extra jwtProviders) ensureAPIConfig -> apiconfig.ParseTextProto (rewrites when missing, when --ateapi-log-level is set, or when stored prototext fails strict schema/level validation; preserves valid customized log levels) DeployAPIServer / UpdateAPIAuthenticationConfig / UpdateAPIConfig apiServerEnvHash(env, ate-api-authentication, ate-api-config) -> annotates Deployment/ate-api-server pod template (ate.dev/env-hash) ``` ## Evidence - **Before:** Re-running `ate-setup deploy ate-system --ateapi-expected-jwt-issuer <issuer>` skipped `ConfigMap/ate-api-authentication`, and upgrading across schema removals or rolling back `1.x -> 1.0.0` with unknown fields in `authentication.yaml` or `apiconfig.textproto` left `ate-api-server` in `CrashLoopBackOff`. - **After:** Valid customized ConfigMaps (custom JWT issuers, extra external `jwtProviders`, hand-edited log levels) survive `ate-setup deploy` untouched when flags are unset; explicit flags or invalid/unknown rollback fields rewrite the ConfigMaps cleanly and update `ate.dev/env-hash`, rolling `Deployment/ate-api-server` to `Ready`. ## Merge Danger **Door:** Two-way **Blast Radius:** Small (`internal/ateapiauth`, `internal/apiconfig`, and `cmd/ate-setup` prerequisite reconciliation and `ate.dev/env-hash` computation) ## Rationale `api-server` crashes on any config field it doesn't recognize, and `ate-setup` never touched those config maps once they existed. the PR makes `ate-setup` rewrite the config maps when the server can't parse it, and keeps valid custom ones. However this need to be in `1.0,0` otherwise if `1.0.x` adds a config field, then someone rollback from `1.0.x` to `1.0.0`, the old server will fail the rollback as it cannot parse the yaml. AI use: AI-assisted implementation and unit tests; reviewed and verified by @moficodes on Kind and GKE. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Mofi Rahman <[email protected]> (cherry picked from commit 671cb69) Signed-off-by: Eitan Yarmush <[email protected]>
…substrate#1607) ateapi stores resources as binary protobuf in `bytea` columns. Binary protobuf identifies a field only by its number. If a field is renumbered, existing rows lose its value. If a retired number is reused, existing rows decode into the wrong field. `atepg.go` unmarshals with `DiscardUnknown: true`, so the original bytes are dropped on the next read-modify-write. Nothing reports this at runtime. - `hack/tools/buf`: adds buf v1.72.0 as a tool module, run through `hack/run-tool.sh`. - `buf.yaml`: checks the protos under `api/`, `internal/proto/`, and `pkg/proto/` with buf's `PACKAGE` breaking rules. Within a proto package, declarations may move between `.proto` files, but messages, enums, services, RPCs, fields, enum values, and reserved ranges cannot be deleted, renamed, renumbered, or retyped. - `hack/verify/proto-breaking.sh`: runs `buf breaking` on the working tree against the merge base of `HEAD` and `origin/main` (or `main`). CI checks out with `fetch-depth: 0`, so the merge base is available there. It passes `--against-config buf.yaml` so a baseline commit without `buf.yaml` still builds. - `AGENTS.md`: removes the Backward Compatibility section. Breaking changes are no longer acceptable once this check is in. - `hack/verify-all.sh`: runs every verifier and lists the failures at the end, instead of stopping at the first one. This is to make sure the CI runs every verifier on a PR including breaking proto changes. I can split this into its own PR if you prefer. I used Claude Code to help write the change and this description. --------- Signed-off-by: Haiyan Meng <[email protected]> (cherry picked from commit ae3ac05) Signed-off-by: Eitan Yarmush <[email protected]>
Why is this needed? The existing threat model diagram was out of date and the security guide lacked a high-level overview diagram illustrating Substrate's core isolation boundaries. This PR: 1. Adds `docs/assets/security-boundaries.svg` and a new "Security Boundaries" section to `docs/security.md`, defining the four core isolation boundaries (sandbox, gateways, control plane, storage) held by Substrate. 2. Replaces the single threat model diagram with two coordinated SVGs: - `docs/assets/actor-template-diagram.svg`: Details actor template creation (T1–T10), from image pull and golden actor warm-up to suspend, snapshot upload, and template readiness. - `docs/assets/threat-model-diagram.svg`: Details actor creation (A1–A2) and the lifecycle of an incoming request through outbound tool invocation (B1–B11). 3. Adds numbered step tables (`Step | What happens | What's checked`) below each diagram in `docs/threat-model.md`. 4. Refreshes the Key Components list in `docs/threat-model.md` to reflect current components (including `ate-idp-server`, `k8s-credential-provider`, and `atunnel`), sorted alphabetically. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Greg Castle <[email protected]> (cherry picked from commit dc039f1) Signed-off-by: Eitan Yarmush <[email protected]>
…upper back from disk (agent-substrate#3485) Why: In the microVM burn-in (GKE downstream testing), wake p50 went from ~350ms to ~1.1s (!), almost all of it in upper_join. Reported to me by @annapendleton though I'm not sure who originally found it. agent-substrate#3353 started writing out snapshot tars as they're written, so their dirty pages can't run a tight worker pod out of memory. It also dropped the written pages from the page cache, so a resume on the same node read the rootfs upper tar back from disk ... which is slow. It also made every suspend write the tars at disk speed. Fixes by commit order: 1. Pull summing memory.stat keys out of memoryUsed, no behavior change. 2. Clean pages are reclaimable, so dropping them is irrelevant for OOM safety. - The tars now stay cached. Write-out waits on the previous chunk instead of the one just written, so dirty pages stay at about two chunks without stalling on each one. 4. Start snapshot writeback after the OnDemand delta merge, which replaces memory-ranges, so it tracks the merged file instead of the unlinked delta. 5. Write snapshot files back in the background instead of with a blocking fdatasync. - Starting write-out can block for a long time on a busy disk (https://github.com/torvalds/linux/blob/v6.12/fs/sync.c#L315-L317), and holding deleted snapshot files open meanwhile leaked disk under load, so it starts write-out once, 64MiB at a time, and closes a file as soon as it's unlinked, which drops its dirty pages. 7. What snapshot writes can borrow leaves clean page cache out, so cached tars don't make a pod look "full". - A worker pod without a memory limit (e.g. the deploy_locust.sh default) has nothing to protect, so it lends all of it instead of capping every suspend as if the pod were full. 9. When the pod can lend memory for the whole archive, the tar skips the write-behind and goes to the page cache at full speed, and the loan comes back once it's on disk. Otherwise the write-behind still bounds its dirty pages. That said, with respect to workers without limits: We should probably consider recommending memory limits on worker pods. I will think about it overnight and propose docs updates tomorrow. Tested with locust burn-in tests (see comments in the PR). AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Benjamin Elder <[email protected]> (cherry picked from commit a6442f9) Signed-off-by: Eitan Yarmush <[email protected]>
…gent-substrate#3447) Part of agent-substrate#3095. The objectstore-plugin exchange read actor JWT keys from a file, a stopgap from before ate-idp-server. agent-substrate#3358 needs an issuer instead, so its `pkg/` API takes no key file ([thread](agent-substrate#3358 (comment))). - The exchange fetches keys through `<issuer>/.well-known/openid-configuration`. `--actor-jwt-jwks-file`, `oidcjwt.FileKeySource` and the `actor-id-jwt-jwks` volume are removed. - The sidecar verifies the issuer's certificate against the system roots plus `--actor-jwt-issuer-ca-file`, which is new and defaults to the service DNS trust bundle. The default ate-idp-server issuer verifies with or without an explicit `:443`, and a custom issuer can use a public certificate. ate-setup writes no CA setting for it. The sidecar re-reads the bundle when it changes and keeps the last good pool if a re-read fails. ate-api-server's `JWTProvider` CA files use the same reloading client and still replace the system roots there. - Nothing reads the pool Secret's `jwks` key now, so its writers are removed: the `admin` commands of kubectl-ate, ate-setup, `PublicJWKS` and `ConcretePool.JWKS`. Existing Secrets keep the stale key. - The sidecar fetches keys once at start. It exits on a certificate error and only warns on other errors, so an unreachable issuer fails exchanges with `UNAVAILABLE` until a fetch succeeds. Cached keys refresh after 5m, a failed refresh retries after 1m, and a token with an unknown key ID triggers at most one fetch per 1m. Until a fetch completes, including one the caller canceled, such a token gets `UNAVAILABLE`, so a newly rotated key ID is retried. Upgrade: drop `--actor-jwt-jwks-file` from custom sidecar configs. Set `--actor-jwt-issuer-ca-file` only if the issuer uses a private CA other than the service DNS one. Tests pass for `oidcjwt`, `credbundle`, `objectstoreplugin`, `cmd/objectstore-plugin`, `cmd/ate-setup` and `apiauthn`. The snapshotcredentials e2e needed no change. It compiles with `-tags=e2e` but has not run, as it needs GCS. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Anish Ramasekar <[email protected]> (cherry picked from commit 92040fe) Signed-off-by: Eitan Yarmush <[email protected]>
…bstrate#3533) Part of agent-substrate#3226. `gvisor` and `microvm` previously mounted different filesystems at `/dev/shm`, and `/tmp`: - `/dev/shm` on gVisor was a `0755` directory on `runsc`'s internal `dev` filesystem, blocking non-root shared memory writes. - `/tmp` was part of the host-backed rootfs upper layer on microVM and a conditional `tmpfs` on gVisor when the image's `/tmp` was empty. - Dropped the `/run` tmpfs from both runtimes, so `/run` comes from the image's rootfs, as on Kubernetes. Move the standard OCI system mounts (`/proc`, `/dev`, `/dev/pts`, `/dev/shm`, `/dev/mqueue`, `/sys`) into `ocispec.Build` and add an explicit `mode=1777` `tmpfs` mount at `/tmp`. If a container mounts a volume at a system mount destination such as `/tmp`, `ocispec.Build` replaces the default `tmpfs` entry there. `ShapeMicroVM` now rewrites `bind` mounts in place. AI use: AI-assisted, I reviewed and tested it. Breaking change: - Old gVisor MEMORY snapshots, including goldens, don't restore. Recut golden snapshots after upgrading. - MicroVM MEMORY snapshots and goldens still resume, but keep the old mounts until they are recut. - MicroVM `/tmp`: now a tmpfs in guest memory instead of the host-backed rootfs. - Its files count against the actor's memory limit. - It is capped at half of the guest's memory. - ROOTFS snapshots no longer keep it, and files older ROOTFS snapshots saved there are hidden. - Files the image ships under `/tmp` are hidden. - gVisor `/dev/shm`: now a 64 MiB tmpfs (`mode=1777`) instead of a root-only directory. Non-root processes can now use it, but it is capped at 64 MiB. - gVisor `/tmp`: now always a tmpfs. runsc used to mount one only when the image's `/tmp` was empty, so files the image ships under `/tmp` are now hidden. If an actor needs more space in `/dev/shm` or `/tmp`, or needs files there kept in ROOTFS snapshots, mount a `durableDir` volume at that path. It replaces the default tmpfs. Images that ship files under `/tmp` should create them at startup instead. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Nayana Bidari <[email protected]> (cherry picked from commit fd576d3) Signed-off-by: Eitan Yarmush <[email protected]>
…e#3450) Concurrency tests in agent-substrate#3255 showed that a request authorized against a deleted atespace can apply to a new atespace with the same name. A previous owner can regain `owner` on it or delete it. We decided not to fix this in code. Atespace names must not be reused, and nothing enforces it. This documents the limitation in the authorization guide and qualifies the claim that a recreated atespace starts with no policy. I used an AI assistant to analyze the issue and draft the doc text, and reviewed the result. --------- Signed-off-by: Shruti Nair <[email protected]> (cherry picked from commit 9d16b69) Signed-off-by: Eitan Yarmush <[email protected]>
…ate#3410) Part of agent-substrate#1661. - Documents the JSON array form of ate-setup's `--credential-provider` in `docs/guides/egress-credential-injection.md` and `cmd/ate-setup/commands.md`, and that the gateway sends each `ate-secret://` URI to the provider its host names. - Updates comments that still described one provider: the egress manifest's marker comment and `cmd/atenet/internal/router/egress/provider.go`. Docs and comments only, so it can merge during the freeze. AI use: AI-assisted draft, reviewed by me. - [x] Tests pass (no code changes) - [x] Docs updated Signed-off-by: Max Thompson <[email protected]> (cherry picked from commit ac8c621) Signed-off-by: Eitan Yarmush <[email protected]>
…ubstrate#3486) Follow-up to agent-substrate#3323, as @SergeyKanzhelev suggested there. kubelet expects a device plugin to switch to a new socket when it re-registers. Since 1.37 it rejects a Register for a socket it still has connected, and two overlapping Registers for one socket can wedge the resource as "already connected" until kubelet restarts: kubernetes/kubernetes#139338 - the device plugin listens on a new `ate.dev-kvm-<random>.sock` for each registration, and removes sockets left over from earlier atelets - the device plugin stays up until atelet exits again. agent-substrate#3323 stopped it on SIGTERM to avoid overlapping with the replacement pod, but the DaemonSet only starts the replacement once the old pod is gone, so that just left `ate.dev/kvm` unhealthy for the whole drain <!-- e.g. "AI-assisted, I reviewed and tested it." No AI co-author trailers. --> AI use: I used Claude to draft this and reviewed it. <!-- Breaking change? Describe what users must do on upgrade and ask for the `breaking-change` label. --> - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Benjamin Elder <[email protected]> (cherry picked from commit 38e9ef6) Signed-off-by: Eitan Yarmush <[email protected]>
…e#3553) `delete demo` returned while the demo namespace was still terminating, so a `deploy demo` right after it failed: the API server refuses new objects in a terminating namespace. Delete now waits for the namespace to be gone, as `delete ate-system` does. Why is this needed? (Or link the issue. Open one first for features, API, install or CI changes.) Demo Fix. Fixes agent-substrate#3525 AI use: I wrote, claude polished. - [ ] Tests pass - [x] Docs updated Signed-off-by: Haven Xia <[email protected]> (cherry picked from commit e99dffb) Signed-off-by: Eitan Yarmush <[email protected]>
The claude-code-multiplex demo doesn't work as written: 1. `deploy demo claude-code-multiplex` creates the three agent templates but no actors, so the dashboard has no agents and "Give a task" fails with `no agents available in namespace`. 2. Even with actors created by hand, the agents can't reach the Anthropic API. Every actor starts with **no egress policy**, and the egress gateway blocks all traffic. 3. The WorkerPool sets no resource limits, so each pod advertises the whole node and all three agents run at once. The demo is about a third agent waiting for a free pod, and that never happens. How this PR fixes it: - It builds on Adam's change from agent-substrate#2982 is folded in, with him as co-author. Deploy creates the three actors, the dashboard resumes an agent when its task starts and suspends it when the task ends, and each pod is sized for one agent. - When both pods are busy, a new task stays queued until one finishes, instead of suspending an agent mid-task. - Gives each agent an egress policy that lets TLS to api.anthropic.com:443 pass through the gateway. - The pod memory is 2560Mi (request = limit), not 2Gi. The worker keeps part of it for itself, so a 2Gi agent didn't fit. - Update the README to match the code. agent-substrate#2488 by @Ziy1-Tan documents the same setup as manual steps. With this PR those steps aren't needed. Testing: tested on kind. AI use: Claude assisted and I reviewed - [x] Tests pass - [x] Docs updated Signed-off-by: Haven Xia <[email protected]> Co-authored-by: Adam Wathieu <[email protected]> (cherry picked from commit 43dc887) Signed-off-by: Eitan Yarmush <[email protected]>
…-substrate#3517) 1. Removes the `status: planned` banner so the page no longer claims to be aspirational. 2. Rewrites the trade-offs section into present-tense implementation context (data locality, control-plane boundaries, and OpenTelemetry observability) and removes the outdated "haven't started to tackle" list. 3. Documents `atenet-egress`, DNS relay filtering, and `EgressPolicy` credential injection under the Networking Stack. 4. Replaces the inline Security & Isolation bullet list with links to the main Security Overview and Threat Model documents. AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated --------- Signed-off-by: Greg Castle <[email protected]> (cherry picked from commit 5a5b192) Signed-off-by: Eitan Yarmush <[email protected]>
agent-substrate#3391) Fixes agent-substrate#3287 `WorkerPool/parking` in `demos/parking/parking.yaml.tmpl` did not set container resource limits, so the Kubernetes Downward API exposed the node's allocatable capacity in `/run/worker-capacity`. As a result, 2 workers had room for all 4 `1000m` actors at once and request parking was never triggered. Set per-worker container limits to `1` CPU and `1Gi` memory so each worker hosts at most one actor. Also add a `wakeupProbe` on `/readyz` port `80` to `demos/parking/parking-template.yaml.tmpl` (matching `demos/counter/counter-template.yaml.tmpl`) so golden memory snapshots are captured after the counter HTTP server binds port 80. In `demos/parking/load.sh`: - Write `kubectl ate proxy actor` output to `.proxy` files instead of `.proxy.log` so `cat "${TMP}"/*.log` does not count proxy startup lines as failed requests. - Wait for each local proxy to start listening before generating load, and fail fast with the proxy output if a proxy exits early. - Exit `1` when any request fails with a non-200/503 status (`other > 0`) or when no requests succeed (`ok == 0`). Signed-off-by: Bo-Cheng Chu <[email protected]> (cherry picked from commit a9fe809) Signed-off-by: Eitan Yarmush <[email protected]>
gvisor-default gains v7, release 20261102.0, as its default version. v6 stays enabled, so actors with memory snapshots taken on v6 still restore with their memory. The new gVisor release includes some reliability fixes such as: - Systrap no longer crashes sandboxes at high density. It now recognizes host stalls, such as a cgroup freeze, and stops killing stub threads that only look stuck (google/gvisor@a5b7936e84dd). - Resumed actors no longer fire all their pending timers at once, because CLOCK_MONOTONIC stops counting checkpointed time. Clock reads are fast right after a restore (google/gvisor@25172e526f45). - Apps no longer freeze while a large memory region loads during a background restore, which lowers resume latency for big actors (google/gvisor@ffc048011e1b, google/gvisor@ae286a28ef26). - Checkpoint no longer stats every cached file. For actors with many files, such as node_modules, the pause drops from seconds to milliseconds (google/gvisor@8229bcb7a7b6). - Disk-backed overlay pages no longer count as memory after a restore, and long-lived sandboxes no longer grow memory for each file they touch (google/gvisor@aa767e4c43d0, google/gvisor@b24cc9724102). AI use: AI-assisted, I reviewed and tested it. - [x] Tests pass - [x] Docs updated Signed-off-by: Nayana Bidari <[email protected]> (cherry picked from commit fc931c4) Signed-off-by: Eitan Yarmush <[email protected]>
Dmitry Berkovich (dberkov)
merged commit Nov 3, 2026
3e03cbf
into
agent-substrate:release-1.0
9 checks passed
This was referenced Nov 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Restore the 19 backports from #3571 after its squash merge was reverted in #3593, and add the parking-demo fixes in #3391 and gVisor update in #3574. Each source change is a separate commit on top of the revert.
Please use Rebase and merge to preserve the individual commits and author attribution.
All 21 applied cleanly in upstream order. Original authors, author dates, co-author and sign-off trailers, and
-xsource references are retained. The final tree matches upstreammainat fc931c4 exactly.Upgrade notes:
/tmpis now always tmpfs; ROOTFS snapshots no longer preserve its contents, and image-provided files there are hidden. MicroVM/tmpcounts against actor memory and is capped at half the guest's memory. gVisor/dev/shmis now a 64 MiB tmpfs. Mount adurableDirat either path when persistence or more space is needed./runnow comes from the image rootfs.--actor-jwt-jwks-filefrom custom credential-exchange sidecar configs. Set--actor-jwt-issuer-ca-fileonly when the issuer uses a private CA other than the service DNS CA.Includes #3505 and #3574, which overlap the open backports in #3547 and #3576, respectively. GKE provisioning and teardown now live in substrate-gke rather than this repository.
Included PRs, in cherry-pick order
Validation: all 21 author records and attribution trailers are preserved, every intermediate tree matches its upstream source, and
git diff --checkpasses. The original 19 commits on this branch are unchanged.AI use: AI-assisted cherry-picking, attribution checks, and review.