v1.0.0-rc.3 is the third release candidate for Substrate 1.0. It brings release-1.0 up to main as of November 4. On top of rc.2 it moves atelet to the system-cluster-critical priority class so disk pressure can no longer evict the one component that frees node disk, makes a gVisor durableDir root 0777 as documented, adds the pkg/objectstore/actorauthz package so out-of-tree object-store providers can run the actor checks, publishes CRD types in the API reference, and adds an upgrade e2e test that runs on every push to a release branch.

It also carries everything in rc.2: actors run as the image's USER, capabilities.add is limited to a short allowlist, /tmp and /dev/shm are tmpfs, mounts are nodev on both runtimes, and the worker outage from microVM memory overshoot, the 15 s atenet-router start stall, drifting actor and worker assignments, and the device plugin wedge after an atelet restart are fixed.

Upgrading from rc.2 breaks gVisor MEMORY snapshots of actors with a durableDir. Upgrading from rc.1 additionally breaks all gVisor snapshots and needs an image cache reset. See Upgrading from v1.0.0-rc.2 and Upgrading from v1.0.0-rc.1 at the end. Upgrading from v0.6.0 is not supported; reinstall.

What's Changed

⚠️ Breaking Changes

Settings to change before upgrading:

• Snapshot location path segments are validated (#2784 by Anish Ramasekar (@aramase)). Each segment of --snapshot-location (SNAPSHOT_LOCATION) must match ^[A-Za-z0-9_-]+$ and cannot equal atespaces in any case. Fix the path before upgrading or ate-api-server refuses to start.
• --cordon-control-plane is now --cordon-system-infra-components (#3210 by haiyanmeng). The env var is ATE_INSTALL_CORDON_SYSTEM_INFRA_COMPONENTS, the config key cordonSystemInfraComponents, and the shared pool label and taint ate.dev/workloadType=ate-system-infra. The old flag and config key fail; the old env var is ignored silently and the install stops cordoning. Relabel and retaint the pool, then redeploy with the new flag; the PR has the gcloud commands.
• ActorTemplates may only add 12 capabilities (#3580 by Greg Castle (@destijl), #3487 by Benjamin Elder (@BenTheElder)). securityContext.capabilities.add accepts only AUDIT_WRITE, CHOWN, DAC_OVERRIDE, FOWNER, FSETID, KILL, NET_BIND_SERVICE, SETFCAP, SETGID, SETPCAP, SETUID and SYS_CHROOT; MKNOD, SYS_ADMIN, SYS_RAWIO and the rest are rejected at admission. Unknown names in add or drop are rejected too. Check your templates for typos and for capabilities outside the list. Unprivileged ping keeps working without NET_RAW.
• Custom credential-exchange sidecars lose --actor-jwt-jwks-file (#3447 by Anish Ramasekar (@aramase)). The exchange now fetches keys through OIDC discovery. Drop the flag from custom sidecar configs, and set --actor-jwt-issuer-ca-file only when the issuer uses a private CA other than the service DNS one.
• The public MintActorCertificate RPC is removed (#3427 by Eitan Yarmush (@EItanya)). Calls return UNIMPLEMENTED. Remove references to its client method and message types before upgrading. Actor JWTs and the worker certificates atunnel uses are unchanged.
• Worker pods need the new worker image before the new ate-controller (#2428 by Tom O'Rourke (@tjorourke)). Workers get runtime-specific capability sets and the pod template changed. Move every WorkerPool to the new worker image first; a microVM pool left on the old image stops working once ate-controller rolls it. The PR has the upgrade and rollback steps.
• A .svc actor JWT issuer is trusted only through IssuerCAFile (#3358 by Anish Ramasekar (@aramase)). The credential-exchange sidecar no longer accepts a .svc issuer certificate that verifies only through the system roots. Set --actor-jwt-issuer-ca-file to the issuer's CA, or the sidecar fails at start.
• Pools hold a little less (#3104 by Benjamin Elder (@BenTheElder)). A worker's capacity is its memory limit minus 64 MiB, and each microVM actor books its limit plus 128 MiB + 8 MiB/GiB. A pool sized to exactly fit its actors needs limits.memory raised by about that much. MicroVM guests now see their whole memory limit.

Behavior changes after upgrading:

• Containers run as the image's USER and WORKDIR, and the image cache layout changed (#2392 by Benjamin Elder (@BenTheElder)). Images with a USER stop running as root and capabilities.add no longer helps a non-root user. After an upgrade or rollback, atelet refuses the old image cache: delete /var/lib/ate/image-cache on each node once no actor runs there, or reboot the node. A custom atelet securityContext needs CHOWN, DAC_OVERRIDE, FOWNER and FSETID.
• /tmp and /dev/shm are tmpfs on both runtimes (#3533 by nybidari). ROOTFS snapshots no longer keep /tmp, and image-provided files there are hidden. MicroVM /tmp counts against actor memory, capped at half the guest's memory; gVisor /dev/shm is 64 MiB. Mount a durableDir at either path for persistence or more space. /run now comes from the image rootfs.
• gVisor MEMORY snapshots from before this release don't restore (#3533, #3580, #2392). runsc restore rejects a checkpoint whose mounts or user differ from the new spec, so suspended gVisor actors crash on resume and templates created before the upgrade crash on first resume. RevertActor doesn't help. Recreate the templates and replace the actors. MicroVM actors are unaffected.
• A gVisor durableDir root is now mode 0777, and its old MEMORY snapshots don't restore (#3457 by nybidari). The API guide always said 0777; gVisor gave the tmpfs default 01777, whose sticky bit stopped one user deleting another's files in the volume root. The mount now carries a mode=0777 option, and runsc restore rejects a MEMORY checkpoint whose mount options differ, so a gVisor actor whose template has a durableDir crashes on resume from a snapshot taken before this release, including the template's golden. Recreate those templates and replace actors holding an old MEMORY snapshot. Rollback breaks the same way. Dirs restored from VOLUMES or ROOTFS snapshots keep 01777.
• MicroVM assets move to kata 4.3.1, Envoy to v1.39.5 (#3181 by Davanum Srinivas (@dims)). A microVM MEMORY snapshot taken on kata 4.1.0 restores at ROOTFS fidelity on its next resume: a fresh guest with its rootfs writes and volumes, no memory state. Recreate templates whose golden you want warm again.
• A microVM actor that overshoots its booking is killed, not its worker (#3429 by Benjamin Elder (@BenTheElder)). Each actor's cgroup gets memory.high at what it books; one stuck there is killed and reported crashed with "killed for exceeding its memory limit". Other actors keep running. Without PSI the worker sets no limit and warns at startup.
• MicroVM guests get containerd's default masked and read-only paths (#3249 by Huy Pham (@hdp617)). A uid 0 entrypoint can no longer write guest sysctls such as vm.max_map_count, matching gVisor. Goldens and checkpoints taken before this release keep a writable /proc/sys until re-taken.
• WorkerPool NetworkPolicies are named substrate-<pool> (#3130 by Yufan Su (@yufan-su)). Pools with similar long names no longer share a policy. On an in-place upgrade the old substrate-<truncated>-<hash> policy stays until the pool is deleted; once the new one exists, delete the old ones with the command in the PR.
• An actor and its worker assignment are written in one transaction (#3223 by Scott Funkenhauser (@sfunkenhauser)). Earlier releases could leave them out of sync after an interrupted write, and this release no longer repairs that. A worker that can't be deleted or an actor stuck in RESUMING is the symptom.
• Atespace names must not be reused (#3450 by Shruti Nair (@SHRUTI6991)). A request authorized against a deleted atespace can apply to a new one with the same name, and nothing enforces uniqueness. Pick a new name instead.

Bug Fixes

• atelet runs at system-cluster-critical so DiskPressure can't evict the component that frees node disk; upgraded installs can delete the old ate-node-critical PriorityClass by igooch in #3328
• atenet-router no longer stalls 15s on every start by Chuang Wang (@chuangw6) in #3346
• An actor and worker assignment that drift apart no longer wedge the worker; see Breaking Changes by Scott Funkenhauser (@sfunkenhauser) in #3223
• atelet: register the device plugin on a new socket each time, fixing the wedge after a restart by Benjamin Elder (@BenTheElder) in #3486 and https://github.com/agent-substrate/substrat...

Now witness the power of this fully armed and operational ... system for managing agents and agent-like workloads securely at scale.

Thank you to our MANY contributors who helped get us here! We couldn't have done it without you. ❤️

What's Changed

⚠️ Breaking Changes

• Envoy moves to v1.39.5, micro-VM assets to kata 4.3.1, kind and CSI images pinned by digest by Davanum Srinivas (@dims) in #3181
• WorkerPools with similar long names no longer share a NetworkPolicy by Yufan Su (@yufan-su) in #3130
• MicroVM guests get their whole memory limit, and placement books each actor's runtime overhead by Benjamin Elder (@BenTheElder) in #3104
• Rename --cordon-control-plane to --cordon-system-infra-components and its node pool label to ate-system-infra by haiyanmeng in #3210
• A microVM actor using more host memory than it booked no longer takes down its worker by Benjamin Elder (@BenTheElder) in #3429
• Remove the unused public MintActorCertificate API by Eitan Yarmush (@EItanya) in #3427
• [release-1.0] Cherry-pick #2392: Run containers as the image's USER and WORKDIR, with layer file owners kept by Benjamin Elder (@BenTheElder) in #3473
• [release-1.0] Cherry-pick #3487: Unknown Linux capability names in an ActorTemplate are rejected at admission by Benjamin Elder (@BenTheElder) in #3514
• [release-1.0] Cherry-pick 23 main changes through 4488a66 by Eitan Yarmush (@EItanya) in #3536
• [release-1.0] Cherry-pick 19 main changes through 5a5b192 by Eitan Yarmush (@EItanya) in #3571
• [release-1.0] Cherry-pick 21 main changes through fc931c4 by Eitan Yarmush (@EItanya) in #3596
• [release-1.0] Cherry-pick #3580: Restrict actor capabilities.add and make mounts nodev by Dmitry Berkovich (@dberkov) in #3611
• [release-1.0] Cherry-pick 5 main changes through 2289ade by Dmitry Berkovich (@dberkov) in #3645

Bug Fixes

• atelet: detach Checkpoint from caller cancellation once underway by Chenyi Wang (@chw120) in #2602
• Actors can reach the network while they boot, before their wakeup probe passes by Quentin Bisson (@QuentinBisson) in #2117
• atenet: classify a 403 route outcome as denied, not resume_error by Timo Derstappen (@teemow) in #2534
• A gVisor actor going over its memory limit no longer takes down every actor on its worker by Benjamin Elder (@BenTheElder) in #3106
• benchmarking/nighthawk-egress: create namespaces in both deploy scripts by Yuan Gao (@ygao-g) in #3315
• microvm: forward sysctls and lift the guest's privileged-port limit to match gVisor by Davanum Srinivas (@dims) in #1904
• Releases publish the envoy-dataplane image by Grant McCloskey (@MushuEE) in #3252
• atelet, ateworker: finish a checkpoint retry after atelet restarts by Chenyi Wang (@chw120) in #3218
• ateworker: reuse a completed checkpoint only on the same boot by Chenyi Wang (@chw120) in #3213
• atelet: keep the worker's Unavailable when a checkpoint fails by Aditya Shantanu (@aditya-shantanu) in #3123
• fix atelet deviceplugin wedge by Benjamin Elder (@BenTheElder) in #3323
• [release-1.0] Cherry-pick #3476: leave page cache out of the memory guard's overshoot check by Benjamin Elder (@BenTheElder) in #3513

Features: Networking and Egress

• e2e: use a port-forward per router CONNECT tunnel by Benjamin Elder (@BenTheElder) in #3051
• envoy: cross compile instead of compiling under slow emulation by Benjamin Elder (@BenTheElder) in #3152
• actornet: pin the sysctls a sandbox namespace inherits from the node by Benjamin Elder (@BenTheElder) in #3310
• Add kubectl ate proxy actor command. by Julian Gutierrez Oschmann (@juli4n) in #3191
• Allow the egress gateway to use more than one credential provider by Max Thompson (@thompsonmax) in #3409

Features: Security and Identity

• manifests: Run control plane containers as non-root by Mike Dickey (@mikedickey) in #3028
• Enable runsc core tagging when supported by the kernel by Fabricio Voznika (@fvoznika) in #2887

Features: Workers and Actors

• require workers to run in their own cgroup namespace by Benjamin Elder (@BenTheElder) in #3110
• Update gVisor to release 20261026.0 by nybidari in #3364
• Update a deleting tag returns FAILED_PRECONDITION by Luiz Oliveira (@laoj2) in #3340
• Suspending actors no longer OOM-kills tight worker pods by Benjamin Elder (@BenTheElder) in #3353
• ateapi: carry the golden actor's crash message into the template status by Davanum Srinivas (@dims) in #2644
• [release-1.0] Cherry-pick #3456: microvm: skip software bounce buffer (SWIOTLB) by default by jiayimeow in #3515

Features: Install and Operations

• ci: automate release artifact build and publish by Steven Shriver (@stshrive) in #2112
• On size10 clusters, the bundled Postgres accepts 500 connections instead of 300 by haiyanmeng in #3141
• benchmarking: add the egress-tunnels workload by Yuan Gao (@ygao-g) in #2716
• Add documentation for SLOs and alerting by Adrian Ludwin (@adrianludwin) in #2476
• [release-1.0] Cherry-pick #3446: ate-setup: retry each ko release image with backoff by Anna Pendleton (@annapendleton) in #3459

Documentation

• document that PRs are squash-merged, only HEAD needs to build, optimize for reviews by Benjamin Elder (@BenTheElder) in #3109
• correct the minimum kernel for the image cache: Linux 6.7, or 6.6.23 by Benjamin Elder (@BenTheElder) in #3128
• docs: use kvm-arm.vgic_v3_group1_trap=1 for Lima microVM setup on arm64 by Benjamin Elder (@BenTheElder) in #3149
• microvm consistency: standardize docs and agent instructions by Benjamin Elder (@BenTheElder) in #3105
• atelet: say snapshot cache hits fall back to copies across mounts by Haowei Cai (Roy) (@roycaihw) in #3064
• docs: require DCO sign-off instead of the Google CLA by Benjamin Elder (@BenTheElder) in #3237
• docs: fix the README note on enabling beta APIs by Haven Xia (@HavenXia) in #3266
• [bugbash] hack/teardown.sh: add --cluster-scoped to tear down one cluster by Haven Xia (@HavenXia) in #3296
• benchmarking: what to set and watch before a dense run by Davanum Srinivas (@dims) in #2499
• docs: add extensibility guide for gateways by Greg Castle (@destijl) in #3317
• AGENTS.md: add the version skew rules by Haven Xia (@HavenXia) in #3151
• Document running the ate-api-server cold resume tests from a workstation by haiyanmeng in #3394
• Add Hugo Docsite by Mofi Rahman (@moficodes) in #1974

Other Changes

• Prevent storing actors/tags with malformed durable snapshot URIs by Luiz Oliveira (@laoj2) in #3082
• Remove outdated roadmap and update security vulnerability reporting guidelines by Greg Castle (@destijl) in #3122
• glutton: run the RunScript timing tests on a synctest bubble by Max Smythe (@maxsmythe) in #3090
• atecontroller: Limit the Secrets read to the egress MITM CA pool by Mike Dickey (@mikedickey) in #3036
• manifests: Give podcertificate-controller its own ServiceAccount by Mike Dickey (@mikedickey) in #3059
• atelet: return Canceled when Run or Restore fails to reset actor dirs by Zoe Zhao (@zoez7) in #3063
• Resume restores an uploaded snapshot when its node is gone by igooch in #3094
• Move public APIs under api/<api-name>/v1. by Julian Gutierrez Oschmann (@juli4n) in #2996
• Fix statusz port in k8s credential provider by Yufan Su (@yufan-su) in #3067
• benchmarking/locust: cap trial_summary actor density by live actors by Chenyi Wang (@chw120) in https://github.com/ag...

v1.0.0-rc.2 is the second release candidate for Substrate 1.0. It brings release-1.0 up to main as of November 3 and hardens the sandboxes before GA: actors run as the image's USER, capabilities.add is limited to a short allowlist, /tmp and /dev/shm are tmpfs, and mounts are nodev on both runtimes. It also fixes a worker outage from microVM memory overshoot, a 15 s stall on every atenet-router start, an actor and worker assignment that could drift apart, and the device plugin wedging after an atelet restart.

Upgrading from rc.1 is possible but breaks gVisor snapshots and needs an image cache reset. See Upgrading from v1.0.0-rc.1 at the end. Upgrading from v0.6.0 is not supported; reinstall.

What's Changed

⚠️ Breaking Changes

Settings to change before upgrading:

• Snapshot location path segments are validated (#2784 by Anish Ramasekar (@aramase)). Each segment of --snapshot-location (SNAPSHOT_LOCATION) must match ^[A-Za-z0-9_-]+$ and cannot equal atespaces in any case. Fix the path before upgrading or ate-api-server refuses to start.
• --cordon-control-plane is now --cordon-system-infra-components (#3210 by haiyanmeng). The env var is ATE_INSTALL_CORDON_SYSTEM_INFRA_COMPONENTS, the config key cordonSystemInfraComponents, and the shared pool label and taint ate.dev/workloadType=ate-system-infra. The old flag and config key fail; the old env var is ignored silently and the install stops cordoning. Relabel and retaint the pool, then redeploy with the new flag; the PR has the gcloud commands.
• ActorTemplates may only add 12 capabilities (#3580 by Greg Castle (@destijl), #3487 by Benjamin Elder (@BenTheElder)). securityContext.capabilities.add accepts only AUDIT_WRITE, CHOWN, DAC_OVERRIDE, FOWNER, FSETID, KILL, NET_BIND_SERVICE, SETFCAP, SETGID, SETPCAP, SETUID and SYS_CHROOT; MKNOD, SYS_ADMIN, SYS_RAWIO and the rest are rejected at admission. Unknown names in add or drop are rejected too. Check your templates for typos and for capabilities outside the list. Unprivileged ping keeps working without NET_RAW.
• Custom credential-exchange sidecars lose --actor-jwt-jwks-file (#3447 by Anish Ramasekar (@aramase)). The exchange now fetches keys through OIDC discovery. Drop the flag from custom sidecar configs, and set --actor-jwt-issuer-ca-file only when the issuer uses a private CA other than the service DNS one.
• The public MintActorCertificate RPC is removed (#3427 by Eitan Yarmush (@EItanya)). Calls return UNIMPLEMENTED. Remove references to its client method and message types before upgrading. Actor JWTs and the worker certificates atunnel uses are unchanged.
• Worker pods need the new worker image before the new ate-controller (#2428 by Tom O'Rourke (@tjorourke)). Workers get runtime-specific capability sets and the pod template changed. Move every WorkerPool to the new worker image first; a microVM pool left on the old image stops working once ate-controller rolls it. The PR has the upgrade and rollback steps.
• Pools hold a little less (#3104 by Benjamin Elder (@BenTheElder)). A worker's capacity is its memory limit minus 64 MiB, and each microVM actor books its limit plus 128 MiB + 8 MiB/GiB. A pool sized to exactly fit its actors needs limits.memory raised by about that much. MicroVM guests now see their whole memory limit.

Behavior changes after upgrading:

• Containers run as the image's USER and WORKDIR, and the image cache layout changed (#2392 by Benjamin Elder (@BenTheElder)). Images with a USER stop running as root and capabilities.add no longer helps a non-root user. After an upgrade or rollback, atelet refuses the old image cache: delete /var/lib/ate/image-cache on each node once no actor runs there, or reboot the node. A custom atelet securityContext needs CHOWN, DAC_OVERRIDE, FOWNER and FSETID.
• /tmp and /dev/shm are tmpfs on both runtimes (#3533 by nybidari). ROOTFS snapshots no longer keep /tmp, and image-provided files there are hidden. MicroVM /tmp counts against actor memory, capped at half the guest's memory; gVisor /dev/shm is 64 MiB. Mount a durableDir at either path for persistence or more space. /run now comes from the image rootfs.
• gVisor MEMORY snapshots from before this release don't restore (#3533, #3580, #2392). runsc restore rejects a checkpoint whose mounts or user differ from the new spec, so suspended gVisor actors crash on resume and templates created before the upgrade crash on first resume. RevertActor doesn't help. Recreate the templates and replace the actors. MicroVM actors are unaffected.
• MicroVM assets move to kata 4.3.1, Envoy to v1.39.5 (#3181 by Davanum Srinivas (@dims)). A microVM MEMORY snapshot taken on kata 4.1.0 restores at ROOTFS fidelity on its next resume: a fresh guest with its rootfs writes and volumes, no memory state. Recreate templates whose golden you want warm again.
• A microVM actor that overshoots its booking is killed, not its worker (#3429 by Benjamin Elder (@BenTheElder)). Each actor's cgroup gets memory.high at what it books; one stuck there is killed and reported crashed with "killed for exceeding its memory limit". Other actors keep running. Without PSI the worker sets no limit and warns at startup.
• MicroVM guests get containerd's default masked and read-only paths (#3249 by Huy Pham (@hdp617)). A uid 0 entrypoint can no longer write guest sysctls such as vm.max_map_count, matching gVisor. Goldens and checkpoints taken before this release keep a writable /proc/sys until re-taken.
• WorkerPool NetworkPolicies are named substrate-<pool> (#3130 by Yufan Su (@yufan-su)). Pools with similar long names no longer share a policy. On an in-place upgrade the old substrate-<truncated>-<hash> policy stays until the pool is deleted; once the new one exists, delete the old ones with the command in the PR.
• An actor and its worker assignment are written in one transaction (#3223 by Scott Funkenhauser (@sfunkenhauser)). Earlier releases could leave them out of sync after an interrupted write, and this release no longer repairs that. A worker that can't be deleted or an actor stuck in RESUMING is the symptom.
• Atespace names must not be reused (#3450 by Shruti Nair (@SHRUTI6991)). A request authorized against a deleted atespace can apply to a new one with the same name, and nothing enforces uniqueness. Pick a new name instead.

Bug Fixes

• atenet-router no longer stalls 15s on every start by Chuang Wang (@chuangw6) in #3346
• An actor and worker assignment that drift apart no longer wedge the worker; see Breaking Changes by Scott Funkenhauser (@sfunkenhauser) in #3223
• atelet: register the device plugin on a new socket each time, fixing the wedge after a restart by Benjamin Elder (@BenTheElder) in #3486 and #3323
• Resuming a microVM actor on the same node no longer reads its rootfs upper back from disk by Benjamin Elder (@BenTheElder) in #3485
• Leave page cache out of the memory guard's overshoot check by Benjamin Elder (@BenTheElder) in #3476
• A gVisor actor going over its memory limit no longer takes down every actor on its worker by Benjamin Elder (@BenTheElder) in #3106
• Suspending actors no longer OOM-kills tight worker pods by Benjamin Elder (@BenTheElder) in #3353
• ate-setup: reconcile ate-api-server ConfigMaps and roll out on config changes by Mofi Rahman (@moficodes) in #3370
• An actor's new egress connections no longer fail with 403 an hour after it starts by Chenyi Wang (@chw120) in #3144
• Resume restores an uploaded snapshot when its node is gone by igooch in #3094
• atelet: clear node-local caches after a node reboot by igooch in #3376
• Actors can reach the network while they boot, before their wakeup probe passes by Quentin Bisson (@QuentinBisson) in #2117
• atenet: classify a 403 route outcome as denied, not resume_error by Timo Derstappen (@teemow) in #2534
• Export atenet-egress traces and metrics with service.name=atenet-egress by Bo-Cheng Chu (@bochengchu) in #3417
• atelet: detach Checkpoint from caller cancellation once underway by Chenyi Wang (@chw120) in #2602
• atelet, ateworker: finish a checkpoint retry after atelet restarts by Chenyi Wang (@chw120) in #3218
• ateworker: reuse a co...

Container Images

Container images for v1.0.0-rc.1 are published to GitHub Container Registry (GHCR).

All images are tagged with v1.0.0-rc.1.

Available images: Agent Substrate GitHub Packages

To pull an image:

docker pull ghcr.io/agent-substrate/substrate/<image-name>:v1.0.0-rc.1

What's Changed

⚠️ Breaking Changes

• Envoy moves to v1.39.5, micro-VM assets to kata 4.3.1, kind and CSI images pinned by digest by Davanum Srinivas (@dims) in #3181
• WorkerPools with similar long names no longer share a NetworkPolicy by Yufan Su (@yufan-su) in #3130
• MicroVM guests get their whole memory limit, and placement books each actor's runtime overhead by Benjamin Elder (@BenTheElder) in #3104
• Rename --cordon-control-plane to --cordon-system-infra-components and its node pool label to ate-system-infra by haiyanmeng in #3210
• A microVM actor using more host memory than it booked no longer takes down its worker by Benjamin Elder (@BenTheElder) in #3429
• Remove the unused public MintActorCertificate API by Eitan Yarmush (@EItanya) in #3427

Bug Fixes

• atelet: detach Checkpoint from caller cancellation once underway by Chenyi Wang (@chw120) in #2602
• Actors can reach the network while they boot, before their wakeup probe passes by Quentin Bisson (@QuentinBisson) in #2117
• atenet: classify a 403 route outcome as denied, not resume_error by Timo Derstappen (@teemow) in #2534
• A gVisor actor going over its memory limit no longer takes down every actor on its worker by Benjamin Elder (@BenTheElder) in #3106
• benchmarking/nighthawk-egress: create namespaces in both deploy scripts by Yuan Gao (@ygao-g) in #3315
• microvm: forward sysctls and lift the guest's privileged-port limit to match gVisor by Davanum Srinivas (@dims) in #1904
• Releases publish the envoy-dataplane image by Grant McCloskey (@MushuEE) in #3252
• atelet, ateworker: finish a checkpoint retry after atelet restarts by Chenyi Wang (@chw120) in #3218
• ateworker: reuse a completed checkpoint only on the same boot by Chenyi Wang (@chw120) in #3213
• atelet: keep the worker's Unavailable when a checkpoint fails by Aditya Shantanu (@aditya-shantanu) in #3123
• fix atelet deviceplugin wedge by Benjamin Elder (@BenTheElder) in #3323

Features: Networking and Egress

• e2e: use a port-forward per router CONNECT tunnel by Benjamin Elder (@BenTheElder) in #3051
• envoy: cross compile instead of compiling under slow emulation by Benjamin Elder (@BenTheElder) in #3152
• actornet: pin the sysctls a sandbox namespace inherits from the node by Benjamin Elder (@BenTheElder) in #3310
• Add kubectl ate proxy actor command. by Julian Gutierrez Oschmann (@juli4n) in #3191
• Allow the egress gateway to use more than one credential provider by Max Thompson (@thompsonmax) in #3409

Features: Security and Identity

• manifests: Run control plane containers as non-root by Mike Dickey (@mikedickey) in #3028
• Enable runsc core tagging when supported by the kernel by Fabricio Voznika (@fvoznika) in #2887
• Generate the actor identity CA with ECDSA P-256 so external relying parties accept it by Greg Castle (@destijl) in #3251

Features: Workers and Actors

• require workers to run in their own cgroup namespace by Benjamin Elder (@BenTheElder) in #3110
• Update gVisor to release 20261026.0 by nybidari in #3364
• Update a deleting tag returns FAILED_PRECONDITION by Luiz Oliveira (@laoj2) in #3340
• Suspending actors no longer OOM-kills tight worker pods by Benjamin Elder (@BenTheElder) in #3353
• ateapi: carry the golden actor's crash message into the template status by Davanum Srinivas (@dims) in #2644

Features: Install and Operations

• ci: automate release artifact build and publish by Steven Shriver (@stshrive) in #2112
• On size10 clusters, the bundled Postgres accepts 500 connections instead of 300 by haiyanmeng in #3141
• benchmarking: add the egress-tunnels workload by Yuan Gao (@ygao-g) in #2716
• Add documentation for SLOs and alerting by Adrian Ludwin (@adrianludwin) in #2476
• [release-1.0] Cherry-pick #3446: ate-setup: retry each ko release image with backoff by Anna Pendleton (@annapendleton) in #3459

Documentation

• document that PRs are squash-merged, only HEAD needs to build, optimize for reviews by Benjamin Elder (@BenTheElder) in #3109
• correct the minimum kernel for the image cache: Linux 6.7, or 6.6.23 by Benjamin Elder (@BenTheElder) in #3128
• docs: use kvm-arm.vgic_v3_group1_trap=1 for Lima microVM setup on arm64 by Benjamin Elder (@BenTheElder) in #3149
• microvm consistency: standardize docs and agent instructions by Benjamin Elder (@BenTheElder) in #3105
• atelet: say snapshot cache hits fall back to copies across mounts by Haowei Cai (Roy) (@roycaihw) in #3064
• docs: require DCO sign-off instead of the Google CLA by Benjamin Elder (@BenTheElder) in #3237
• docs: fix the README note on enabling beta APIs by Haven Xia (@HavenXia) in #3266
• [bugbash] hack/teardown.sh: add --cluster-scoped to tear down one cluster by Haven Xia (@HavenXia) in #3296
• benchmarking: what to set and watch before a dense run by Davanum Srinivas (@dims) in #2499
• docs: add extensibility guide for gateways by Greg Castle (@destijl) in #3317
• AGENTS.md: add the version skew rules by Haven Xia (@HavenXia) in #3151
• Document running the ate-api-server cold resume tests from a workstation by haiyanmeng in #3394
• Add Hugo Docsite by Mofi Rahman (@moficodes) in #1974

Other Changes

• Prevent storing actors/tags with malformed durable snapshot URIs by Luiz Oliveira (@laoj2) in #3082
• Remove outdated roadmap and update security vulnerability reporting guidelines by Greg Castle (@destijl) in #3122
• glutton: run the RunScript timing tests on a synctest bubble by Max Smythe (@maxsmythe) in #3090
• atecontroller: Limit the Secrets read to the egress MITM CA pool by Mike Dickey (@mikedickey) in #3036
• manifests: Give podcertificate-controller its own ServiceAccount by Mike Dickey (@mikedickey) in #3059
• atelet: return Canceled when Run or Restore fails to reset actor dirs by Zoe Zhao (@zoez7) in #3063
• Resume restores an uploaded snapshot when its node is gone by igooch in #3094
• Move public APIs under api/<api-name>/v1. by Julian Gutierrez Oschmann (@juli4n) in #2996
• Fix statusz port in k8s credential provider by Yufan Su (@yufan-su) in #3067
• benchmarking/locust: cap trial_summary actor density by live actors by Chenyi Wang (@chw120) in #3140
• ateapi: squash PostgreSQL migrations and lock them once merged by Jeremy Alvis (@iplay88keys) in #2991
• e2e: fix the TestKillMatching flake, wait for the child's command line by Davanum Srinivas (@dims) in #3155
• ateapi accepts snapshot report from atelet for actors in CRASHED state by Zoe Zhao (@zoez7) in #3148
• ateworker: reuse completed checkpoint when workload is already stopped by Chenyi Wang (@chw120) in #3015
• gVisor sandboxes move to release 20261019.0 from the 2026-10-17 nightly by Davanum Srinivas (@dims) in #3177
• Reject WorkerPool names longer than 63 characters by haiyanmeng in #3136
• atelet: verify microvm SandboxConfig assets before writing sandbox-runtimes.json by Shrutiya Mohan (@shrutiyam-glitch) in #3143
• benchmarking: dial atenet-router over SPIFFE mTLS on port 443 by Sairaj Pokale (@sairajp-rewind) in #3089
• Base images drop the distrusted eMudhra roots and the kind Jaeger moves to v2.22.0 by Davanum Srinivas (@dims) in #3196
• UpdateActor fails with Aborted while another o...

What's Changed

⚠️ Breaking Changes

• require actor cpu and memory limits by Benjamin Elder (@BenTheElder) in #2285
• rename ateom -> ateworker by Benjamin Elder (@BenTheElder) in #2254
• gvisor SandboxConfig no longer accepts a bare runsc asset by Eric Curtin (@ericcurtin) in #2798
• ateapi: Remove the PauseActor RPC by Zoe Zhao (@zoez7) in #2813
• ateworker no longer exports to localhost:4317 without a relay or endpoint, and takes its own exporter settings by Quentin Bisson (@QuentinBisson) in #2189
• Remove any_state and allow deleting actors in any state by Luiz Oliveira (@laoj2) in #2727
• [bug] fix microVM actors with one character container names by Benjamin Elder (@BenTheElder) in #2801
• Add a default cluster snapshot storage location with snapshotConfig.storageLocation override in preview mode by Sneha-at (@Sneha-at) in #2544
• Renumber proto fields consecutively, with ResourceMetadata as field 1 by Luiz Oliveira (@laoj2) in #2888
• resource limits and volume capacity are integers now: cpuMillicores, memoryBytes, capacityBytes by Benjamin Elder (@BenTheElder) in #2769
• imagecache: Verify layer content against its diffID before it enters the pool by igooch in #2279
• atelet, nodepath: separate per-actor snapshot directories under /var/lib/ate/snapshots by Haowei Cai (Roy) (@roycaihw) in #2746
• imagecache: Refuse a layout version 1 pool and log diffID mismatches by igooch in #2979
• On GKE, micro-VM assets move to their own bucket and nodes lose access to the snapshot bucket by ILLUM1N0X in #3037
• On GCS, snapshot transfers run as a dedicated service account and their tokens no longer run short near the end of each hour by ILLUM1N0X in #3093

Bug Fixes

• Reduce memory utilization in Envoy egress gateway by deduplicating policy object by yanavlasov in #2651
• atenet-egress: refuse an actor's own CONNECT with 403 by Bo-Cheng Chu (@bochengchu) in #2721
• fix(ateom-gvisor): isolate and evict checkpoint page cache to prevent actor cgroup OOM by Chenyi Wang (@chw120) in #1917
• atelet: make durable-dir volumes writable by non-root containers by Davanum Srinivas (@dims) in #1906
• skip device nodes when unpacking image layers by Benjamin Elder (@BenTheElder) in #2739
• setup-gcp: raise node ARP table and inotify sysctls for high sandbox density by Chenyi Wang (@chw120) in #2444
• egress: return a retryable 503 instead of a 504 when a credential provider is slow by Benjamin Elder (@BenTheElder) in #2787
• ateinterceptors: elide List responses from the RPC log by Davanum Srinivas (@dims) in #2494
• atelet: only ate-api-server may call AteomHerder by Greg Castle (@destijl) in #2516
• Fake data plane installs no longer fail at startup on the --preview flag by haiyanmeng in #2884
• setup-gcp no longer disables Filestore on an existing cluster by Eric Curtin (@ericcurtin) in #2485
• Large GCS uploads no longer leave scratch objects behind by Eric Curtin (@ericcurtin) in #2178
• ateapi: crash the actor when atelet's returned snapshot state conflicts by igooch in #2794
• Snapshot uploads reserve only the memory their upload path uses by Haowei Cai (Roy) (@roycaihw) in #2848
• imagecache: Pin a pull's layers so eviction cannot retire them mid-pull by igooch in #1509
• Grant fake-workersync the controller authz role so it can manage Workers by haiyanmeng in #2885
• atelet: Bound background snapshot uploads with a timeout by Michelle Au (@msau42) in #2831
• ateworker-microvm: give the guest CRNG reseed 60s on restore by Haowei Cai (Roy) (@roycaihw) in #2743
• atecontroller: deregister workers when their WorkerPool is deleted by Eric Curtin (@ericcurtin) in #2093
• TestSyncer_Lifecycle no longer flakes on a worker that is still UNAVAILABLE by Eric Curtin (@ericcurtin) in #2824
• ateapi: check mounted volumes are attachable and test assigned_node detach on crash by Vu Le (@lekhoavu2909) in #2611
• Revert and delete no longer hang when an actor's node is deleted by Eren Akbulut (@eren23) in #2486
• ateworker: clear partial checkpoint files and preserve microVM base on merge by Chenyi Wang (@chw120) in #2623
• atelet: Wait for superseded uploads to stop by Michelle Au (@msau42) in #2833
• Retry GCS rate-limit errors when deleting and copying snapshots by Sairaj Pokale (@sairajp-rewind) in #2143
• atelet: Answer a retried Checkpoint from the snapshot it committed by Michelle Au (@msau42) in #2849
• Components no longer warn about a partial telemetry resource at startup by Tim Bai (@baizhenyu) in #2998
• resources: mask debug_redact fields in validation error text by Da Huang (@git286) in #3071
• ateworker-microvm, atelet: classify pre-serving restore failures as Unavailable by Haowei Cai (Roy) (@roycaihw) in #2744
• atelet: Clean up whole actor directory on detach by Sairaj Pokale (@sairajp-rewind) in #2871
• ateapi,atelet: reclaim actor node state when a worker's pod is gone by Sairaj Pokale (@sairajp-rewind) in #2797
• objectstorage: fail a ranged read cut short by cancellation by Walter Fender (@cheftako) in #3049
• setup-gcp: say which node pools need replacing after enabling the beta APIs by Gari Singh (@mastersingh24) in #1852

Features: Networking and Egress

• Router authn/z part 2: ate-setup: add --ingress-auth-mode by Taahir Ahmed (@ahmedtd) in #2388
• Router authn/z part 2.5: podcertcontroller: Don't load pod in podidentity signer by Taahir Ahmed (@ahmedtd) in #2389
• atenet-egress: log every failed request, sample successes at 1% by Yuan Gao (@ygao-g) in #2693
• write /etc/hosts for actors so localhost resolves by Benjamin Elder (@BenTheElder) in #2738
• benchmarking/atunnel: atunnel CPU and memory benchmark by Chuang Wang (@chuangw6) in #2761
• E2E checks the actor JWTs the egress gateway injects by Max Thompson (@thompsonmax) in #2256
• router authn/z part 3: Convert all e2e tests so that they pass for static-mtls ingress auth mode by Taahir Ahmed (@ahmedtd) in #2390
• e2e: fix the ingressauth flake: retry a refused handshake that ends without the router's alert by Davanum Srinivas (@dims) in #2961
• atenet-egress: raise the envoy config for a million tunnels case by Yufan Su (@yufan-su) in #2732
• Use crypto/rand for upstream DNS transaction IDs by Bo-Cheng Chu (@bochengchu) in #3020
• Configure preview golden egress policies on actor templates by Eitan Yarmush (@EItanya) in #2159
• atenet-egress: log access log paths without the query string by Da Huang (@git286) in #3077
• atenet: Router checks the routed worker's SPIFFE ID on atunnel's cert by Lior Lieberman (@LiorLieberman) in #2678
• Fix the DNS to return NXDOMAIN by Bowei Du (@bowei) in #2774
• atenet-egress: reload actor-identity CA bundle in atenet-egress when it rotates by Yufan Su (@yufan-su) in #3081
• atenet-router: raise the tunnel circuit breaker to 1m by Yufan Su (@yufan-su) in #3087

Features: Security and Identity

• ate-api-server's objectstore plugin serves per-actor snapshot credentials on GCS by ILLUM1N0X in #2762
• hardening: fix submounts of read-only image volumes being writable by Benjamin Elder (@BenTheElder) in #2944
• Upgrade Go to 1.27.3 by Benjamin Elder (@BenTheElder) in #2984
• ate-api-server accepts actor JWT issuers with a path by Max Thompson (@thompsonmax) in #2781
• router authn/z part 4: Remove deprecated-insecure mode by Taahir Ahmed (@ahmedtd) in #2977

Features: Observabil...

What's Changed

⚠️ Breaking Changes

• SandboxConfig now holds multiple named versions with a default by Zoe Zhao (@zoez7) in #2333
• Replace SnapshotContentScope with the SnapshotFidelity ladder and preferred_fidelity by Dmitry Berkovich (@dberkov) in #2364
• Add sandboxClasses with an optional SandboxConfig ref to WorkerPool by Zoe Zhao (@zoez7) in #2332
• gvisor: move the sandbox asset to release 20260907.0 by Yuan Gao (@ygao-g) in #2413
• Health probes move to their own server on :9091, and :9090 serves only /metrics by Jeff Luo (@JeffLuoo) in #2424
• Rename ContainerWakeupProbe to WakeupProbe by Luiz Oliveira (@laoj2) in #2575
• Remove the substrate version label from the install by Haven Xia (@HavenXia) in #1812
• Block link local egress in atunnel by Keith Mattix II (@keithmattix) in #2319
• stop naming every cached sandbox asset runsc- by Benjamin Elder (@BenTheElder) in #2627
• Changed SandboxClass from enum to string to allow extensibility by Luiz Oliveira (@laoj2) in #2586

Bug Fixes

• objectstore: finish snapshot cleanup when the bucket is missing by Luomu (@cloudonly) in #1973
• Components honor OTEL_TRACES_EXPORTER=none and treat OTEL_METRICS_EXPORTER=prometheus as none where they serve /metrics by Quentin Bisson (@QuentinBisson) in #2190
• Return the caller's context error when an atelet call fails by Eren Akbulut (@eren23) in #2236
• ateom-gvisor: attribute runsc output for the pause container by Luomu (@cloudonly) in #2330
• Benchmark automation runners can reach ate-api-server again under ate-system's default-deny ingress by haiyanmeng in #2637
• ateapi: fail /readyz while the store is unreachable by Eric Curtin (@ericcurtin) in #1989
• run ateom under minimal init by Benjamin Elder (@BenTheElder) in #2626
• atunnel: bound the tunnel open by a deadline by Yuan Gao (@ygao-g) in #2654

Features: Networking and Egress

• atenet: keep the router pod unready until its Envoy is serving by Lior Lieberman (@LiorLieberman) in #2325
• Egress gateway can mint and cache actor JWTs by Max Thompson (@thompsonmax) in #2135
• Egress policies can inject actor JWTs by Max Thompson (@thompsonmax) in #2141
• More robust Actor DNS Relay by Bowei Du (@bowei) in #2370
• Add E2E HTTPS egress tests and reject WebSocket upgrades by Adhita Selvaraj (@swiftdiaries) in #1634
• Add actor egress benchmarking tool by yanavlasov in #2505
• Reduce number of egress atunnels by multiplexing CONNECTs over H/2 by yanavlasov in #2425
• atenet-router: reach atunnel through mTLS CONNECT tunnels by default by Lior Lieberman (@LiorLieberman) in #2618
• Router status private by Lior Lieberman (@LiorLieberman) in #2573
• Block DNS if there is no egress policy configured by Bowei Du (@bowei) in #2524

Features: Security and Identity

• atespace isolation tests by Siyan Shen (@siyanshen) in #1966
• security: prevent path traversal in snapshot storage locations by Benjamin Elder (@BenTheElder) in #2372
• atelet: introduce declarative validation for the AteomHerder RPCs by Shrutiya Mohan (@shrutiyam-glitch) in #1997
• ate-system now denies ingress by default; in-cluster API clients must label their namespace ate.dev/api-access=true by Greg Castle (@destijl) in #2522
• add support for connect from atenet (envoy) to atunnel by Lior Lieberman (@LiorLieberman) in #2531

Features: Observability

• ateom-gvisor: emit actor usage records by Tim Bai (@baizhenyu) in #2194
• ateom-microvm: emit actor usage records by Tim Bai (@baizhenyu) in #2195
• atelet: drop its actor usage records and count each activation's CPU once by Tim Bai (@baizhenyu) in #1984
• benchmarking/analysis: aggregate suspend/resume phase breakdown log r… by Lucky Abolorunke (@Oneimu) in #2009
• Drop per-pod server.address label from rpc.client.call.duration metrics by Jeff Luo (@JeffLuoo) in #2576

Features: Workers and Actors

• fake-dataplane: let the fake data plane host actors by haiyanmeng in #2300
• Build the OCI spec in ateom instead of atelet by Haven Xia (@HavenXia) in #2293
• Allow ActorTemplates to disable golden snapshot creation by Luomu (@cloudonly) in #2328
• Benchmark external volumes with the DurDir load by anishgang in #2057
• ateom-microvm: boot the guest image over virtio-pmem with DAX by jiayimeow in #2218
• microvm: evict snapshot page cache on restore to prevent cgroup memory pressure by Chenyi Wang (@chw120) in #2030
• Move static runsc flags to configuration file by Fabricio Voznika (@fvoznika) in #2281
• drop wall-clock TestWaitForSocketReturnsPromptly, it flakes on loaded ci runners by Benjamin Elder (@BenTheElder) in #2625
• fix kind cluster delete hanging when nfs volumes are mounted by Benjamin Elder (@BenTheElder) in #2621

Features: Install and Operations

• benchmarking: pace boomer to a target rate and add the resumepaused user class by haiyanmeng in #2350
• ate-setup: have EnsureCRDs create only the missing CRDs by Haven Xia (@HavenXia) in #2585
• Add --fake-data-plane to install a fake data plane for control-plane benchmarks by haiyanmeng in #2469

Dependencies

• Envoy, Alpine and dev-tool dependency bumps clear pre-GA CVEs by Anna Pendleton (@annapendleton) in #2568

Documentation

• Lifecycle metrics omit empty template labels, and a crash during a delete reports the delete operation by Jeff Luo (@JeffLuoo) in #2356
• Upgrade Runbook GA by Haven Xia (@HavenXia) in #1844

Other Changes

• Build the JWK set that publishes actor JWT verification keys by Max Thompson (@thompsonmax) in #2132
• ate-setup: do not record demo and benchmark deploys by Haven Xia (@HavenXia) in #2320
• ate-setup should record the Substrate version an install used by Haven Xia (@HavenXia) in #2321
• Volume E2E tests wait for actor restoration before checking mounts by Luomu (@cloudonly) in #2294
• docs(router): fix the request parking guide link by ktsoator (@ktsoator) in #2327
• Authenticate bundled PostgreSQL logins with separate pod certificates by Jeremy Alvis (@iplay88keys) in #2266
• ateapi: record one scheduler assignment for each resume by Jeff Luo (@JeffLuoo) in #2244
• Delete unused actor_template_ref from ActorAssigment sub-resource by Julian Gutierrez Oschmann (@juli4n) in #2322
• benchmark(ateom-gvisor): log per-actor checkpoint and restore phase b… by Lucky Abolorunke (@Oneimu) in #2302
• Don't default name in store layer. by Julian Gutierrez Oschmann (@juli4n) in #2359
• kubectl ate admin can rotate the actor JWT signing key by Max Thompson (@thompsonmax) in #2140
• cdiinject: confine device paths and host mounts to allowed roots by Benjamin Elder (@BenTheElder) in #2315
• Add sections about sub-resources and singletons to the API style guide. by Julian Gutierrez Oschmann (@juli4n) in #2354
• ate-setup: choose the StorageClass of the bundled PostgreSQL by Eric Curtin (@ericcurtin) in #2314
• Move snapshot transfers behind the object-store snapshot plugin by ILLUM1N0X in #2224
• Fail snapshot plugin calls with Unavailable after 30s when the plugin is down by Anish Ramasekar (@aramase) in #2380
• docs(api): fix systeminfo volume examples by ktsoator (@ktsoator) in #2346
• ateom-gvisor logs a warning when a running actor's sandbox dies by Da Huang (@git286) in #2258
• Update Snapshots API to support Actor Lifecycle...

What's Changed

⚠️ Breaking Changes

• snapshot rootfs uppers as one tar per container by Benjamin Elder (@BenTheElder) in #2110
• actorevent: write each actor event to one place, with a shared OTEL_*_EXPORTER parser by Tim Bai (@baizhenyu) in #2193
• ateom-microvm: snapshot durable-dir volumes as one tar per volume by Benjamin Elder (@BenTheElder) in #2283

Bug Fixes

• Clean up actors when ateom terminates by Scott Funkenhauser (@sfunkenhauser) in #1888
• Select kvm-clock for amd64 micro-VM guests by Adhita Selvaraj (@swiftdiaries) in #1945
• Reject repointing an actor that owns a snapshot to a different storage location by Luiz Oliveira (@laoj2) in #2014
• ateapi: stop logging the raw PostgreSQL connection string (which might contain db password) at startup by Da Huang (@git286) in #1833
• atecontroller: add a zero-count bucket to empty exponential histograms by Eric Curtin (@ericcurtin) in #1933
• Support workers with multiple IPs by Scott Funkenhauser (@sfunkenhauser) in #2052
• router: report no_capacity only for a ResourceExhausted cause by Eric Curtin (@ericcurtin) in #1929
• fix(serverboot): honor OTEL_METRICS_EXPORTER=none by Quentin Bisson (@QuentinBisson) in #1878
• atelet: try each matching image pull credential in turn by Benjamin Elder (@BenTheElder) in #2104
• Support --platform build flag in orchestrator image by Max Smythe (@maxsmythe) in #1901
• ateapi: reach an atelet again after its node restarts by Eric Curtin (@ericcurtin) in #2118
• atelet: measure what a snapshot image costs, not how long it is by Chenyi Wang (@chw120) in #931
• Only require a pause image for gVisor sandboxes by Benjamin Elder (@BenTheElder) in #2144
• fix: reload the pod-identity CA pool for client-cert verification by Rashad Sirajudeen (@rashadism) in #1930
• fix: reload frozen CA pools in atelet, ateapiauth, and the worker-side atelet dialer by Rashad Sirajudeen (@rashadism) in #1932
• Preserve system-info registrations during stale lifecycle cleanup by Adhita Selvaraj (@swiftdiaries) in #1944
• [Flaky test fix] TestSandboxAssetPrewarmDownloads - atelet: make the sandbox prewarm worker joinable, fixing the race by Aditya Shantanu (@aditya-shantanu) in #1940
• fix(objectstorage): stop initial range downloads on close by ktsoator (@ktsoator) in #2176
• contextlogging: redact protobuf attributes in the shared slog handler by Da Huang (@git286) in #1942
• WorkerPool can set the ServiceAccount of its worker pods by Danielle Mustillo (@danielle-mustillo) in #2219
• router: report a shed request as no_capacity after a full fleet by Jeff Luo (@JeffLuoo) in #2138

Features: Networking and Egress

• Add gcp secret manager as a credential provider in its own go module by Yufan Su (@yufan-su) in #1991
• egress policy: add an actor JWT value source to CredentialHeader by Max Thompson (@thompsonmax) in #1960
• Implement passthrough egress policy by yanavlasov in #2019
• docs: how an actor exports telemetry under the egress lockdown by Da Huang (@git286) in #2120
• Passthrough dial address resolved by the gateway by Lior Lieberman (@LiorLieberman) in #2045
• Update agentgateway and enable ingress and egress test coverage by Eitan Yarmush (@EItanya) in #2158
• router authn/z: Prefactor; extract router flag binding to function by Taahir Ahmed (@ahmedtd) in #2161
• remove non mitm dataplanes by Lior Lieberman (@LiorLieberman) in #2015
• Router no longer logs request credentials or query strings by Da Huang (@git286) in #2206
• kubectl-ate: share the missing-actor check across egress-policy commands by Eric Curtin (@ericcurtin) in #2191
• graduate credential injection from experimental by Lior Lieberman (@LiorLieberman) in #2101
• ateapi: describe no_capacity without "free worker" by Jeff Luo (@JeffLuoo) in #2137
• SystemInfo: Implement system-roots.ate.dev trust bundle by Taahir Ahmed (@ahmedtd) in #2049

Features: Security and Identity

• feat(credentials): narrow a namespace grant by secret name or label by Jonathan Jamroga (@jjamroga) in #1859
• ClusterTrustBundle v1 support by John Howard (@howardjohn) in #1924
• atelet: pull image credentials from the kubelet credential provider by Benjamin Elder (@BenTheElder) in #917
• ateapi: Implement Policy Governance CRUD Access Policy APIs by Shruti Nair (@SHRUTI6991) in #1965
• oidcdiscovery: build the OpenID discovery document by Max Thompson (@thompsonmax) in #1963
• kubectl-ate: add --validity flag to admin make-ca-pool by Pedro Nunes (@pintonunes) in #2058
• localjwtauthority: thumbprint key IDs and a configurable signing algorithm by Max Thompson (@thompsonmax) in #1961
• security: only run sandbox runtime assets from the static files dir by Benjamin Elder (@BenTheElder) in #2269
• only enable guest debug console with --guest-debug by Benjamin Elder (@BenTheElder) in #2291
• ateapi: [Part 2] Runtime Authorization check for Actor and ActorTemplate CRUD by Shruti Nair (@SHRUTI6991) in #1980

Features: Observability

• benchmarking/locust: harvest server-side telemetry from Prometheus by Nishanth Kotla (@Nishanth29) in #1725
• ateomstats: add per-activation usage state and the usage emitter by Tim Bai (@baizhenyu) in #2005
• otlprelay: use a short temp dir for the test socket by Benjamin Elder (@BenTheElder) in #2270

Features: Workers and Actors

• workerservice: validate RequestActorSuspend requests by Shrutiya Mohan (@shrutiyam-glitch) in #2023
• atelet: use a glibc base image to support credential providers by Benjamin Elder (@BenTheElder) in #2108
• ateapi: prefer less-loaded workers via power of two choices by Chenyi Wang (@chw120) in #1915
• atelet,ateom: restore local pause checkpoints directly from LocalSnap by Chenyi Wang (@chw120) in #1876
• csi: add publishContext to actor volumes by hajiler in #1578
• Fix grpc status error when creating actors and tags by Julian Gutierrez Oschmann (@juli4n) in #2263
• fake-dataplane: add a fake dataplane skeleton by haiyanmeng in #2272
• ocispec: raise actor RLIMIT_NOFILE from 1024 to 1048576 by Yuan Gao (@ygao-g) in #2242

Features: Install and Operations

• benchmarking: agent-session workload — a scripted coding agent that suspends while the LLM thinks by Aditya Shantanu (@aditya-shantanu) in #1934
• ci: add automated release notes generation by Steven Shriver (@stshrive) in #2018
• cordon-control-plane: share one pool, give postgres its own by haiyanmeng in #2056
• feat(benchmarking): add substrate spawn benchmark by Sairaj Pokale (@sairajp-rewind) in #1969
• Let glutton actors consume CPU, driven from boomer dynconfig by Max Smythe (@maxsmythe) in #1875
• ate-setup: add the uniform configuration package by Steven Shriver (@stshrive) in #2012

Documentation

• Add Yolo the Octopus logo and quick navigation links to README by Brandon Royal (@brandonroyal) in #2239

Other Changes

• atepg: name the row in list decode errors by Eric Curtin (@ericcurtin) in #2000
• ategcs: rename and move to pkg/ for reuse outside atelet by ILLUM1N0X in #2013
• Remove reserved fields from proto and update back-compat instruction in AGENTS.md by Luiz Oliveira (@laoj2) in #1985
• remove ssl_cert_dir and add clearer guidance for actor trust store by Lior Lieberman (@LiorLieberman) in #2044
• ateom: share atunnel wiring in ateomtunnel by Eric Curtin (@ericcurtin) in #1959
• Delete worker pods in terminal states by Scott Funkenhauser (@sfunkenhauser) in #2051
• ...

What's Changed

Bug Fixes

• Update agentgateway for split actor and ateom identities by Eitan Yarmush (@EItanya) in #1923
• atelet: pin the multi-actor stats fold and keep CPU baselines across pending sweeps by Tim Bai (@baizhenyu) in #1885
• Snapshot path validation by Benjamin Elder (@BenTheElder) in #1950
• fix(benchmarking): make the SWE-Perf ActorTemplate deployable by Haowei Cai (Roy) (@roycaihw) in #1889
• nighthawk-ingress: let run-dev.sh run more actors than workers by haiyanmeng in #1982
• ateom-microvm: reseed the guest CRNG on restore by eliranw in #1524
• Pin envoy-dataplane from the release on a pre-built install by Anna Pendleton (@annapendleton) in #1964

Features: Networking and Egress

• ateom/actor ID split: Ateom uses MintAteomActorCertificate by Taahir Ahmed (@ahmedtd) in #1809
• egress policy enhancements for GA by Lior Lieberman (@LiorLieberman) in #1751
• e2e: add test for egress credential injection cujs by Yufan Su (@yufan-su) in #1795
• Egress policy plumbing by yanavlasov in #1978
• kubectl-ate: add delete egress-policy by Yuan Gao (@ygao-g) in #1808
• envoy dyn module followup by Lior Lieberman (@LiorLieberman) in #1995

Features: Security and Identity

• ateapi: make the actor JWT issuer configurable and add typ to the header by Max Thompson (@thompsonmax) in #1834
• ateapi: [Part 1] Runtime Authorization check for Atespace CRUD by Shruti Nair (@SHRUTI6991) in #1702
• ateapi: require an actor JWT lifetime and align the subject with the SPIFFE ID by Max Thompson (@thompsonmax) in #1902

Features: Observability

• actorevent: define the actor usage event and the activation epoch by Tim Bai (@baizhenyu) in #1881
• benchmarking: actor telemetry for the sweperf workload by Nishanth Kotla (@Nishanth29) in #1848

Features: Install and Operations

• CI: create JUnit artifacts and gate on all tests passing by Steven Shriver (@stshrive) in #1873
• benchmark(atelet, ateom-microvm): log per-actor checkpoint and restor… by Lucky Abolorunke (@Oneimu) in #1941
• ate-setup: add deploy subcommands for podcertificate-controller and SandboxConfig by Haven Xia (@HavenXia) in #2010

Features: Workers and Actors

• Add ActorDirs message to carry per-actor directories and and split the path packages by owner by Haven Xia (@HavenXia) in #1730
• ateapi: release a suspend's replaced snapshot after committing SUSPENDED by haiyanmeng in #2006

Documentation

• docs: add a star history chart to the README by botengyao in #1894
• doc: egress credential injection with sample k8s secret credential pr… by Yufan Su (@yufan-su) in #1805
• Fix several broken links to CODE_OF_CONDUCT.md by Tim Flannagan (@timflannagan) in #1987

Other Changes

• ateom/actor ID split: Clean up purpose from MintActorCertificate by Taahir Ahmed (@ahmedtd) in #1810
• Remove unnecessary fsync during ResumeActor by Michelle Au (@msau42) in #1883
• serverboot.Fatal: Allow additional slog arguments by Taahir Ahmed (@ahmedtd) in #1908
• ate-api: Move authn package into server-specific internal package by Taahir Ahmed (@ahmedtd) in #1909
• ateletpb: carry the Restore base snapshot as a typed source, dropping golden_snapshot_uri by Dmitry Berkovich (@dberkov) in #1602
• Clean up packages in ateomnet by Bowei Du (@bowei) in #1897
• nodepath: move the node state root to /var/lib/ate by Davanum Srinivas (@dims) in #1926
• move agw tests to post-merge and release-* branches by Lior Lieberman (@LiorLieberman) in #1949
• upgrade otel log to 0.21 by Benjamin Elder (@BenTheElder) in #1958
• ateom-gvisor: take the actor directories from the request by Haven Xia (@HavenXia) in #1865
• ateom-microvm: take the actor directories from the request and remove ateompath package by Haven Xia (@HavenXia) in #1872
• atelet: introduce declarative validation for the AteomSupport RPCs by Shrutiya Mohan (@shrutiyam-glitch) in #1903
• atelet: add CAP_DAC_OVERRIDE to reset dirs a non-root container wrote by Davanum Srinivas (@dims) in #1910
• ateapi: rename --egress-gateway-address to --default-egress-gateway-address by Lior Lieberman (@LiorLieberman) in #1858
• Clean up vestigial metadata setting in atepg by Tim Hockin (@thockin) in #1992
• Move DV validators under apivalidations. by Julian Gutierrez Oschmann (@juli4n) in #2007

New Contributors

• Haowei Cai (Roy) (@roycaihw) made their first contribution in #1889
• Tim Flannagan (@timflannagan) made their first contribution in #1987

Full Changelog: v0.2.0...v0.3.0

Committers in this release

• Taahir Ahmed (@ahmedtd)
• Anna Pendleton (@annapendleton)
• Tim Bai (@baizhenyu)
• Benjamin Elder (@BenTheElder)
• botengyao
• Bowei Du (@bowei)
• Dmitry Berkovich (@dberkov)
• Dependabot (@dependabot)[bot]
• Davanum Srinivas (@dims)
• Du Bin (@dubin555)
• Eitan Yarmush (@EItanya)
• eliranw
• haiyanmeng
• Haven Xia (@HavenXia)
• Jeffrey Ying (@Jefftree)
• Julian Gutierrez Oschmann (@juli4n)
• Lior Lieberman (@LiorLieberman)
• Michelle Au (@msau42)
• Nishanth Kotla (@Nishanth29)
• Lucky Abolorunke (@Oneimu)
• Haowei Cai (Roy) (@roycaihw)
• Shruti Nair (@SHRUTI6991)
• Shrutiya Mohan (@shrutiyam-glitch)
• Steven Shriver (@stshrive)
• Tim Hockin (@thockin)
• Max Thompson (@thompsonmax)
• Tim Flannagan (@timflannagan)
• yanavlasov
• Yuan Gao (@ygao-g)
• Yufan Su (@yufan-su)

v0.2.0 adds multi-actor workers, per-sandbox networking, and egress policy enforcement, along with many fixes and API cleanups.

Substrate is still pre-1.0 and changes quickly. This release contains breaking changes to the API and the wire protocol. Please read them below before you upgrade.

Please try it out and file issues for anything you find.

Breaking Changes

• Requests are now routed with an explicit ate-target-actor: <atespace>/<actor> header instead of the Host header. (#1333)
• ActorTemplate.containers.readyz is renamed to wakeupProbe. (#1794)
• The top-level boot field is removed from ResumeActorRequest. (#1548)
• IPBlockRule is renamed to CIDRRule. (#1597)
• SnapshotsConfig is renamed to SnapshotConfig. (#1791)
• LocalSnapshotInfo is renamed to LocalSnapshot. (#1856)
• A worker's sandbox_class is now immutable. (#1257)
• Every Delete<Resource> method now takes DeletePreconditions. (#1807)
• The egress gateway now enforces EgressPolicy. An actor without a policy has no egress, and traffic that no rule allows is denied. (#1545)
• Actors can no longer send UDP to any port except DNS. (#1623)
• Removed the unused TagStatus.source_actor_uid field (#1772), other unused messages (#1814), and reserved field numbers (#1793).
• Removed the ate.scheduler.eligible_workers metric (#1804) and the ateerrors failure-reason taxonomy (#1817).
• Snapshot storage paths now use Tag UIDs. (#1521)
• kubectl ate now prints a single resource as a bare object, not a list. (#1627)

Features

Workers and Actors

• A worker can now run more than one actor at a time. (#1836)
• Each actor sandbox gets its own network namespace. (#1689)
• New RevertActor lifecycle RPC. (#1675, #1711)
• ActorStatus now reports the crash reason and when it happened. (#1867)
• An actor is marked crashed when an ateom checkpoint or restore fails with a non-retriable error. (#1220)
• ateom can now suspend actors itself. (#1779)
• ActorTemplate golden snapshots are published as tags. (#1523)
• ExternalSnapshot records the actor_template_uid of each snapshot. (#1713)
• The same volume can be mounted at more than one path. (#1611)
• The actor template resync interval is configurable. (#1522)
• The actor termination grace period is now 30 minutes. (#1565)
• API fields can have defaults. (#1630, #1674)

Networking and Egress

• The egress gateway enforces each actor's EgressPolicy. (#1545)
• An egress credential injector (#1360), plus an example credential provider that reads Kubernetes Secrets (#1335, #1811).
• kubectl ate can get, create, and update egress policies. (#1659, #1813)
• atunnel supports looking up the original destination over IPv6. (#753)
• The parking lot takes its slots when a request parks, not when it is admitted. (#1229)
• Initial Envoy Dynamic Modules for the dataplane. (#1535)
• agentgateway is tested in CI. (#1598)

Security and Identity

• Groundwork for authorization: an OpenFGA model for atespaces, served from the API server. Checks are not enforced yet. (#1233, #1670, #1839)
• Actor JWT and certificate minting moved into the API server. (#1315, #1626)
• Running actors receive rotated projected trust bundles without a restart. (#1231)
• Support for PodCertificateRequest v1, falling back to v1beta1. Needed on clusters that no longer serve v1beta1. (#1829)
• Secret-bearing proto fields are marked debug_redact and redacted from RPC logs. (#1803, #1822)
• The CSI plugin caches its CA certificate pool. (#1344)

Observability

• Actor lifecycle events are emitted over OTLP. (#1658, #1638, #1771)
• atelet emits per-actor usage events. (#1206, #1559)
• The OTLP relay forwards log records, and ateoms get a LoggerProvider over it. (#1800, #1857)
• ate.worker.state reports worker occupancy in actor slots. (#1854)
• ate.sandbox.class is bounded by one rule for every emitter. (#1484)
• Each component logs its build version at startup. (#1569)

Install and Operations

• The namespace, Service names, and ServiceAccount names are configurable. (#350)
• A large-cluster install option, and an option to run each control-plane component on a dedicated machine. (#1632)
• Tolerations to isolate workload types on tainted nodes. (#1874)
• atelet sets resource requests and a priority class. (#1651)
• The worker Deployment has an explicit rollout strategy. (#1783)
• setup-gcp gains --enable-nested-virtualization and other improvements. (#1669, #1616)
• ko base images are pinned by digest. (#1500)
• Micro-VM kata assets are bumped to 4.1.0 (#1708), and the kata-config asset is dropped (#1704).
• kubectl ate fixes, including better get workers output. (#1536, #1577, #1733)

Bug Fixes

• Fixed the kind install on arm64 hosts. (#1869)
• In-progress snapshots no longer leak after a worker is deleted. (#1541)
• atelet removes an actor's directory when it terminates the actor (#1654), and ateom removes its directory on graceful shutdown (#1678).
• ateom-gvisor deletes an actor's containers before its sandbox (#1847), treats a container runsc has no record of as already destroyed (#1666), and keys cgroup leaves by actor UID (#1667).
• ateom-gvisor names its root container _pause so no actor container can collide with it. (#1496)
• ateom-gvisor no longer passes -direct to runsc restore. (#1549)
• ateom-microvm finds leftover VMM processes by their staged path. (#1843)
• Stale named network namespaces are replaced. (#1682)
• Stale system-info volume registrations are superseded in atelet. (#1684)
• atelet caches pool resolutions so a list flap cannot split the CPU counter. (#1479)
• The API server always resolves the atelet IP from the node name. (#1742)
• An API server replica applies its own worker writes to its cache at commit. (#1583)
• Worker pool changes now propagate to workers. (#1636)
• atenet reports "unknown" when a resume does not complete (#1482), and has a longer default route timeout (#1529).
• ListActorTemplates maps pagination errors correctly. (#1595)
• Teardown can now delete crashed actors (#942) and waits for the ate-system namespace to terminate (#1851).
• More validation at template creation: snapshot storage location (#1493), system-info projection paths (#1439), and container image (#1614). Added declarative validation for TagStatus and GoldenSnapshotStatus (#1772, #1831).

Documentation

• The upgrade runbook is reorganized. (#1624)
• The supported egress traffic for GA (#1749), how the gateway enforces an EgressPolicy (#1545), and the atunnel to PEP contract (#1726).
• A metrics best-practices guide for component authors. (#1553)
• Warnings that worker node pools must not auto-upgrade or use spot nodes (#1528), and about the 30-minute eviction window (#1618).
• Added MAINTAINERS.md (#1499), and welcomed Keith Mattix as a maintainer (#1806).

Thank you to all of our contributors.

Committers in this release include:

• Aditya Shantanu (@aditya-shantanu)
• Taahir Ahmed (@ahmedtd)
• Anna Pendleton (@annapendleton)
• Alex Zakonov (@azakonovg)
• Tim Bai (@baizhenyu)
• Benjamin Elder (@BenTheElder)
• Bowei Du (@bowei)
• Chuang Wang (@chuangw6)
• Chenyi Wang (@chw120)
• Dmitry Berkovich (@dberkov)
• Eitan Yarmush (@EItanya)
• eliranw
• Da Huang (@git286)
• haiyanmeng
• hajiler
• Haven Xia (@HavenXia)
• Huy Pham (@hdp617)
• John Howard (@howardjohn)
• Jeff Luo (@JeffLuoo)
• Jonathan Jamroga (@jjamroga)
• Joe Betz (@jpbetz)
• Julian Gutierrez Oschmann (@juli4n)
• Keith Mattix II (@keithmattix)
• Kevin Steuer (@kjsteuer)
• Krisztian F (@krisztianfekete)
• Luiz Oliveira (@laoj2)
• Lev Rado (@levrado)
• Lior Lieberman (@LiorLieberman)
• Bingtan Lu (@lubingtan)
• Max Smythe (@maxsmythe)
• Maya Wang (@mayawang)
• Michelle Au (@msau42)
• Grant McCloskey (@MushuEE)
• Nishanth Kotla (@Nishanth29)
• Omer Yahud (@omeryahud)
• NekoPunch (@orangeCatDeveloper)
• Parnika Mandewalkar (@pmandewalkar)
• Jaana Dogan (@rakyll)
• Ron Lev (@ronlv10)
• Sairaj Pokale (@sairajp-rewind)
• Scott Funkenhauser (@sfunkenhauser)
• Shruti Nair (@SHRUTI6991)
• Shrutiya Mohan (@shrutiyam-glitch)
• Sneha-at (@Sneha-at)
• Steven Shriver (@stshrive)
• Adhita Selvaraj (@swiftdiaries)
• Tim Hockin (@thockin)
• Max Thompson (@thompsonmax)
• yanavlasov
• Youssuf Elshall (@yelshall)
• Yuan Gao (@ygao-g)
• Yufan Su (@yufan-su)
• Zoe Zhao (@zoez7)

v0.1.0 Marks our first "real" release.

Substrate is still evolving rapidly and will continue to make breaking changes for the immediate future, but we think the system is ready for early testing.

Please try it out and submit issues for the things you find!

• If something you thought should work, doesn't. Let us know.
• If something doesn't work the way you think it should, please file a bug.
• If you find a missing feature, check if we're already tracking it and if not please file a feature request.

Say hi to Yolo the Octopus, our logo and mascot:

Thank you to all of our contributors.

Committers in this release include:

• Aditya Shantanu (@aditya-shantanu)
• Taahir Ahmed (@ahmedtd)
• Alex Bulankou (@AlexBulankou)
• Arnaud M. (@ameukam)
• Angela (@Angelawork)
• Anna Pendleton (@annapendleton)
• Anish Ramasekar (@aramase)
• Alex Zakonov (@azakonovg)
• Adrien Delorme (@azr)
• Tim Bai (@baizhenyu)
• Benjamin Elder (@BenTheElder)
• Bo-Cheng Chu (@bochengchu)
• botengyao
• Bowei Du (@bowei)
• Brandon Jacobs (@brandonrjacobs)
• Brendan Burns (@brendandburns)
• Walter Fender (@cheftako)
• Chuang Wang (@chuangw6)
• Chenyi Wang (@chw120)
• Dmitry Berkovich (@dberkov)
• Dependabot (@dependabot)[bot]
• Davanum Srinivas (@dims)
• Eitan Yarmush (@EItanya)
• eliranw
• Eric Bishop (@ericdbishop)
• Etienne Perot (@EtiennePerot)
• Francisco Cabrera (@fcabrera23)
• Frederick F. Kautz IV (@fkautz)
• Fabricio Voznika (@fvoznika)
• Da Huang (@git286)
• gooleg-gh
• haiyanmeng
• hajiler
• han2ni3bal-pixel
• xu0o0 (@haoqixu)
• Haven Xia (@HavenXia)
• John Howard (@howardjohn)
• igooch
• ILLUM1N0X
• Jeremy Alvis (@iplay88keys)
• jbpratt
• Jeff Luo (@JeffLuoo)
• Jeffrey Ying (@Jefftree)
• Jonathan Jamroga (@jjamroga)
• J.M. Huibonhoa (@jmhbh)
• Joe Betz (@jpbetz)
• Julian Gutierrez Oschmann (@juli4n)
• Keith Mattix II (@keithmattix)
• Kevin Steuer (@kjsteuer)
• Kyosuke Konishi (@konippi)
• Krisztian F (@krisztianfekete)
• Luiz Oliveira (@laoj2)
• Steven Linde (@linde)
• Lior Lieberman (@LiorLieberman)
• Max Smythe (@maxsmythe)
• Maya Wang (@mayawang)
• Tommy McCormick (@mccormickt)
• Mark Chmarny (@mchmarny)
• Mesut Oezdil (moved to @moezdil) (@mesutoezdil)
• Michael Aspinwall (@michaelasp)
• Jing Chen (@milantracy)
• geyingqi (@mowangdk)
• Michelle Au (@msau42)
• Michael Taufen (@mtaufen)
• Grant McCloskey (@MushuEE)
• Noureldin (@nmn3m)
• Nina Polshakova (@npolshakova)
• nybidari
• Omer Yahud (@omeryahud)
• Lucky Abolorunke (@Oneimu)
• NekoPunch (@orangeCatDeveloper)
• Peter Jausovec (@peterj)
• Parnika Mandewalkar (@pmandewalkar)
• Michał Kuliński (@Qqkyu)
• Jaana Dogan (@rakyll)
• Ron Lev (@ronlv10)
• Sairaj Pokale (@sairajp-rewind)
• Scott Funkenhauser (@sfunkenhauser)
• Shruti Nair (@SHRUTI6991)
• Shrutiya Mohan (@shrutiyam-glitch)
• skamboj
• skvikas-work
• Sneha-at (@Sneha-at)
• Shane O'Donnell (@Sodman)
• Jet Chiang (@supreme-gg-gg)
• Tim Hockin (@thockin)
• Max Thompson (@thompsonmax)
• Troy Chiu (@troychiu)
• Youssuf Elshall (@yelshall)
• Yuan Gao (@ygao-g)
• Yiqing Wang (@YQ-Wang)
• Yufan Su (@yufan-su)
• Zheng Lammerts (@zlammerts-svg)
• Zoe Zhao (@zoez7)