Fix multiple issues in claude-code-multiplex demo by HavenXia · Pull Request #3560 · agent-substrate/substrate

Haven Xia (HavenXia) · 2026-11-02T18:34:40Z

The claude-code-multiplex demo doesn't work as written:

deploy demo claude-code-multiplex creates the three agent templates but no actors, so the dashboard has no agents and "Give a task" fails with no agents available in namespace.
Even with actors created by hand, the agents can't reach the Anthropic API. Every actor starts with no egress policy, and the egress gateway blocks all traffic.
The WorkerPool sets no resource limits, so each pod advertises the whole node and all three agents run at once. The demo is about a third agent waiting for a free pod, and that never happens.

How this PR fixes it:

It builds on Adam's change from demo: add actors by default in claude multiplex #2982 is folded in, with him as co-author. Deploy creates the three actors, the dashboard resumes an agent when its task starts and suspends it when the task ends, and each pod is sized for one agent.
When both pods are busy, a new task stays queued until one finishes, instead of suspending an agent mid-task.
Gives each agent an egress policy that lets TLS to api.anthropic.com:443 pass through the gateway.
The pod memory is 2560Mi (request = limit), not 2Gi. The worker keeps part of it for itself, so a 2Gi agent didn't fit.
Update the README to match the code.

#2488 by Ziyi Tan (@Ziy1-Tan) documents the same setup as manual steps. With this PR those steps aren't needed.

Testing: tested on kind.

AI use: Claude assisted and I reviewed

Tests pass
Docs updated

Dmitry Berkovich (dberkov) · 2026-11-03T04:53:26Z

+        memory: 3Gi
+      requests:
+        cpu: 250m
+        memory: 512Mi


🤖 should-fix 🟡 – The memory request is 512Mi under a 3Gi limit, so the kube-scheduler can pack the node well past what two 2Gi agents need, and the worker pod gets OOM-killed or evicted mid-demo. The counter pool keeps the memory request equal to the limit for this reason (memory is not compressible). Set memory: 3Gi here too.

I set the request a bit smaller, at 2560Mi instead of 3Gi: two 3Gi requests don't fit plus a ~2.2Gi ate-system on a default 8GB kind node, and 2560Mi still leaves 512Mi over the agent's 2Gi.

Dmitry Berkovich (dberkov) · 2026-11-03T04:53:27Z

 ### 3. Drive the demo

-Click "Give a task". The UI picks a random idle agent and creates a task for it. Watch:
+Click "Give a task". The UI picks a random idle agent and gives it a task for a fixed time. The task and its timing live only in the dashboard; a resumed agent runs `claude --print` in a loop (see [`workload/run.sh`](workload/run.sh)), and its log shows the replies. Watch:


🤖 question 🟢 – run.sh runs claude --print once and then sleeps 45s, while a task resumes the agent for only 9–16s. A memory snapshot carries the sleep across suspend and resume, so after the first task most resume windows end before the next Claude call and the log shows nothing new. Should INTERVAL_SECONDS in the agent templates drop to a few seconds so each task produces a reply?

Dropped it to 5s. On kind each task now gets 2–3 Claude calls.

Dmitry Berkovich (dberkov) · 2026-11-03T04:53:27Z

 	}
 	names := make([]string, 0, len(resp.GetActors()))
 	for _, a := range resp.GetActors() {
+		if namespace != "" && a.GetMetadata().GetAtespace() != "" && a.GetMetadata().GetAtespace() != namespace {


🤖 nit 🟢 – The request now carries Atespace: namespace, so the server already filters and this check never trips. The same goes for the template-atespace check in handleActors. Drop both.

I remove the redundant checks.

Dmitry Berkovich (dberkov) · 2026-11-03T04:53:27Z

+		ticker := time.NewTicker(time.Second)
+		defer ticker.Stop()
+		for range ticker.C {
+			applyComputedStates()


🤖 nit 🟢 – This goroutine now writes State, StartedAt and CompletedAt under mu, but handleTaskStatus marshals the same *assignment values after releasing the lock. That is a data race. Copy the structs by value while holding mu, or marshal under the lock.

Dmitry Berkovich (dberkov) · 2026-11-03T04:53:27Z

 	"context"
 	"fmt"

+	"go.ate.dev/substrate/api/ateapi/v1"


🤖 nit 🟢 – The package comment at line 18 still calls the workload a "Python and Claude Code wrapper". The README now says node:20-slim, which matches the Dockerfile. Update the comment.

…a task waits for a free pod Co-authored-by: Adam Wathieu <[email protected]> Signed-off-by: Haven Xia <[email protected]>

@Ziy1-Tan

The claude-code-multiplex demo doesn't work as written: 1. `deploy demo claude-code-multiplex` creates the three agent templates but no actors, so the dashboard has no agents and "Give a task" fails with `no agents available in namespace`. 2. Even with actors created by hand, the agents can't reach the Anthropic API. Every actor starts with **no egress policy**, and the egress gateway blocks all traffic. 3. The WorkerPool sets no resource limits, so each pod advertises the whole node and all three agents run at once. The demo is about a third agent waiting for a free pod, and that never happens. How this PR fixes it: - It builds on Adam's change from agent-substrate#2982 is folded in, with him as co-author. Deploy creates the three actors, the dashboard resumes an agent when its task starts and suspends it when the task ends, and each pod is sized for one agent. - When both pods are busy, a new task stays queued until one finishes, instead of suspending an agent mid-task. - Gives each agent an egress policy that lets TLS to api.anthropic.com:443 pass through the gateway. - The pod memory is 2560Mi (request = limit), not 2Gi. The worker keeps part of it for itself, so a 2Gi agent didn't fit. - Update the README to match the code. agent-substrate#2488 by @Ziy1-Tan documents the same setup as manual steps. With this PR those steps aren't needed. Testing: tested on kind. AI use: Claude assisted and I reviewed - [x] Tests pass - [x] Docs updated Signed-off-by: Haven Xia <[email protected]> Co-authored-by: Adam Wathieu <[email protected]> (cherry picked from commit 43dc887) Signed-off-by: Eitan Yarmush <[email protected]>

Bring `release-1.0` up to `main` at 5a5b192 with the 19 changes merged since the previous batch in #3536. The final tree matches that commit exactly. **Please use Rebase and merge to preserve the individual commits and author attribution.** All 19 applied cleanly in upstream order. Original authors, author dates, co-author and sign-off trailers, and `-x` source references are retained. Upgrade notes: - #3533: old gVisor MEMORY snapshots, including goldens, no longer restore. Recreate them after upgrading. MicroVM MEMORY snapshots still resume with their old mounts until recreated. `/tmp` is now always tmpfs; ROOTFS snapshots no longer preserve its contents, and image-provided files there are hidden. MicroVM `/tmp` counts against actor memory and is capped at half the guest's memory. gVisor `/dev/shm` is now a 64 MiB tmpfs. Mount a `durableDir` at either path when persistence or more space is needed. `/run` now comes from the image rootfs. - #3447: remove `--actor-jwt-jwks-file` from custom credential-exchange sidecar configs. Set `--actor-jwt-issuer-ca-file` only when the issuer uses a private CA other than the service DNS CA. - #3450: do not reuse a deleted atespace name. See the updated atespace name-reuse guidance. Includes #3505, which overlaps the open backport in #3547. GKE provisioning and teardown now live in [substrate-gke](https://github.com/ai-on-gke/substrate-gke) rather than this repository. <details> <summary>Included PRs, in cherry-pick order</summary> 1. #3500: demos/{sandbox,jupyter,multi-template}: fix tutorial prerequisites, walkthroughs, and counter volume 2. #3498: demos/counter: inline gVisor walkthrough and fix setup commands 3. #3511: e2e tests: Extend the existing tests to cover tag actor. 4. #2775: Generate API reference documentation and publish it on Substrate website 5. #3119: Document how people, pods and CI jobs should authenticate to ate-api 6. #3505: tools: remove setup-gcp and hack/teardown.sh 7. #3147: e2e: resume fidelity after SandboxConfig version edits 8. #3370: ate-setup: reconcile ate-api-server ConfigMaps and roll out on config changes 9. #1607: hack: reject breaking proto changes across each proto package 10. #3415: docs: update and add security diagrams 11. #3485: Resuming a microVM actor on the same node no longer reads its rootfs upper back from disk 12. #3447: Verify actor JWTs in the credential exchange through OIDC discovery 13. #3533: ocispec: mount tmpfs at /dev/shm, and /tmp on both runtimes 14. #3450: docs: Document that atespace names must not be reused 15. #3410: Document using more than one egress credential provider 16. #3486: atelet: register the device plugin on a new socket each time 17. #3553: ate-setup: wait for the demo namespace in delete demo 18. #3560: Fix multiple issues in claude-code-multiplex demo 19. #3517: docs: remove inaccurate early-stage architecture documentation </details> Validation: every intermediate tree matches its upstream source commit, all 19 authorship records and original attribution trailers are preserved, and `git diff --check` passes. The original 18 backport commits are unchanged. AI use: AI-assisted cherry-picking, attribution checks, and review. - [ ] Tests pass - [x] Docs updated --------- Signed-off-by: Mofi Rahman <[email protected]> Signed-off-by: Eitan Yarmush <[email protected]> Signed-off-by: Nayana Bidari <[email protected]> Signed-off-by: Julian Gutierrez Oschmann <[email protected]> Signed-off-by: Max Thompson <[email protected]> Signed-off-by: Steven Shriver <[email protected]> Signed-off-by: Grant McCloskey <[email protected]> Signed-off-by: Shrutiya <[email protected]> Signed-off-by: Haiyan Meng <[email protected]> Signed-off-by: Greg Castle <[email protected]> Signed-off-by: Benjamin Elder <[email protected]> Signed-off-by: Anish Ramasekar <[email protected]> Signed-off-by: Shruti Nair <[email protected]> Signed-off-by: Haven Xia <[email protected]> Co-authored-by: Mofi Rahman <[email protected]> Co-authored-by: nybidari <[email protected]> Co-authored-by: Julian Gutierrez Oschmann <[email protected]> Co-authored-by: Max Thompson <[email protected]> Co-authored-by: Grant McCloskey <[email protected]> Co-authored-by: Steven Shriver <[email protected]> Co-authored-by: shrutiyam-glitch <[email protected]> Co-authored-by: haiyanmeng <[email protected]> Co-authored-by: Greg Castle <[email protected]> Co-authored-by: Benjamin Elder <[email protected]> Co-authored-by: Anish Ramasekar <[email protected]> Co-authored-by: Shruti Nair <[email protected]> Co-authored-by: Haven Xia <[email protected]> Co-authored-by: Adam Wathieu <[email protected]>

@Ziy1-Tan

The claude-code-multiplex demo doesn't work as written: 1. `deploy demo claude-code-multiplex` creates the three agent templates but no actors, so the dashboard has no agents and "Give a task" fails with `no agents available in namespace`. 2. Even with actors created by hand, the agents can't reach the Anthropic API. Every actor starts with **no egress policy**, and the egress gateway blocks all traffic. 3. The WorkerPool sets no resource limits, so each pod advertises the whole node and all three agents run at once. The demo is about a third agent waiting for a free pod, and that never happens. How this PR fixes it: - It builds on Adam's change from #2982 is folded in, with him as co-author. Deploy creates the three actors, the dashboard resumes an agent when its task starts and suspends it when the task ends, and each pod is sized for one agent. - When both pods are busy, a new task stays queued until one finishes, instead of suspending an agent mid-task. - Gives each agent an egress policy that lets TLS to api.anthropic.com:443 pass through the gateway. - The pod memory is 2560Mi (request = limit), not 2Gi. The worker keeps part of it for itself, so a 2Gi agent didn't fit. - Update the README to match the code. #2488 by @Ziy1-Tan documents the same setup as manual steps. With this PR those steps aren't needed. Testing: tested on kind. AI use: Claude assisted and I reviewed - [x] Tests pass - [x] Docs updated Signed-off-by: Haven Xia <[email protected]> Co-authored-by: Adam Wathieu <[email protected]> (cherry picked from commit 43dc887) Signed-off-by: Eitan Yarmush <[email protected]>

Haven Xia (HavenXia) force-pushed the multiplex-takeover branch from 6d16e64 to c8fff09 Compare November 2, 2026 19:00

Haven Xia (HavenXia) requested a review from Dmitry Berkovich (dberkov) November 2, 2026 21:01

Haven Xia (HavenXia) assigned Dmitry Berkovich (dberkov) Nov 2, 2026

Dmitry Berkovich (dberkov) reviewed Nov 3, 2026

View reviewed changes

claude-code-multiplex: deploy gives each agent an egress policy, and …

2f7c0a3

…a task waits for a free pod Co-authored-by: Adam Wathieu <[email protected]> Signed-off-by: Haven Xia <[email protected]>

Haven Xia (HavenXia) force-pushed the multiplex-takeover branch from c8fff09 to 2f7c0a3 Compare November 3, 2026 05:10

Haven Xia (HavenXia) requested a review from Dmitry Berkovich (dberkov) November 3, 2026 05:10

Dmitry Berkovich (dberkov) approved these changes Nov 3, 2026

View reviewed changes

Dmitry Berkovich (dberkov) added this pull request to the merge queue Nov 3, 2026

Merged via the queue into agent-substrate:main with commit 43dc887 Nov 3, 2026
7 checks passed

Eitan Yarmush (EItanya) mentioned this pull request Nov 3, 2026

[release-1.0] Cherry-pick 19 main changes through 5a5b19219 #3571

Merged

1 of 2 tasks

Eitan Yarmush (EItanya) mentioned this pull request Nov 3, 2026

[release-1.0] Cherry-pick 21 main changes through fc931c4e2 #3596

Merged

1 of 2 tasks

a (aalzo) mentioned this pull request Nov 5, 2026

demos/claude-code-multiplex: wire UI tasks to real actor lifecycle, inject ANTHROPIC_API_KEY via EgressPolicy, and fix README #3503

Open

Navigation Menu

Repository navigation

Fix multiple issues in claude-code-multiplex demo - #3560

Fix multiple issues in claude-code-multiplex demo#3560
Dmitry Berkovich (dberkov) merged 1 commit into
agent-substrate:mainfrom
HavenXia:multiplex-takeover

Haven Xia (HavenXia) commented Nov 2, 2026 •

edited

Loading

Uh oh!

Dmitry Berkovich (dberkov) Nov 3, 2026

Uh oh!

Haven Xia (HavenXia) Nov 3, 2026

Uh oh!

Dmitry Berkovich (dberkov) Nov 3, 2026

Uh oh!

Haven Xia (HavenXia) Nov 3, 2026

Uh oh!

Dmitry Berkovich (dberkov) Nov 3, 2026

Uh oh!

Haven Xia (HavenXia) Nov 3, 2026

Uh oh!

Dmitry Berkovich (dberkov) Nov 3, 2026

Uh oh!

Haven Xia (HavenXia) Nov 3, 2026

Uh oh!

Dmitry Berkovich (dberkov) Nov 3, 2026

Uh oh!

Haven Xia (HavenXia) Nov 3, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Haven Xia (HavenXia) commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Haven Xia (HavenXia) commented Nov 2, 2026 •

edited

Loading