Actors can reach the network while they boot, before their wakeup probe passes by QuentinBisson · Pull Request #2117 · agent-substrate/substrate

Quentin Bisson (QuentinBisson) · 2026-10-02T10:19:12Z

An actor's egress was turned on only after every container answered its wakeup probe, so a workload that downloads what it needs to become ready (a model, a Git repository) could not start. atunnel closed the connection inside the sandbox before it reached the gateway. Egress is now turned on before the first container starts, on both sandbox classes and for both run and restore; ingress still waits for the probe. The atenet gateway admits RESUMING actors as well as RUNNING ones, and atunnel and the gateway now log the connections they refuse.

The actor's EgressPolicy still applies. The new e2e test gives the golden actor its policy through the template's goldenEgressPolicy (#2159), so it runs only with the GoldenEgressPolicy preview gate. A restored actor's wakeup probe also waits for a fresh fetch as itself, which covers the restore paths. agentgateway moves to v0.0.0-alpha.9d36620d, which admits RESUMING actors too (agentgateway/agentgateway#3882), so the test runs on both dataplanes.

Tested on kind with gVisor on both dataplanes, before the move to goldenEgressPolicy: the test passed with this change and failed without it (on agentgateway, against the previous pin). The rest of the gVisor e2e suite also passes on the agentgateway lane, apart from the external-volume tests, which could not mount NFS on my rootless setup.

Part of #1543

Tests pass
Appropriate changes to documentation are included in the PR

Eitan Yarmush (EItanya) · 2026-10-06T15:59:13Z

Quentin Bisson (@QuentinBisson) I think this doesn't do anything without #2159 allowing for policies to be applied to goldens

Quentin Bisson (QuentinBisson) · 2026-10-06T16:08:09Z

You're totally right. I forgot to link it here. I did not want to make the other PR bigger

Quentin Bisson (QuentinBisson) · 2026-10-08T08:40:14Z

To clarify my earlier reply: the two PRs are complementary. #2159 gives the golden actor a policy, and this PR turns egress on before readyz, so the golden's boot fetch needs both. This one also helps any actor that already has a policy and fetches while booting or restoring. I've linked #2159 in the description.

Allow substrate egress for `RESUMING` actors as well as `RUNNING` actors, so workloads can fetch what they need while booting or restoring, before the wakeup probe passes. This supports agent-substrate/substrate#2117. The actor's egress policy still applies. All 91 Substrate integration tests pass with `cargo test -p agentgateway --test integration tests::substrate::`, including coverage for resuming actors, rejected states, and policy enforcement. Formatting checks pass. AI assistance was used for the implementation, tests, and this description. - [ ] As required by the [Code of Conduct](https://github.com/agentgateway/agentgateway/blob/main/CODE_OF_CONDUCT.md#generative-ai-policy), the description, docs, and comments (words meant for humans) are written by a human, not by an LLM. Signed-off-by: Eitan Yarmush <[email protected]>

Eitan Yarmush (EItanya) · 2026-10-09T14:01:13Z

Quentin Bisson (@QuentinBisson) this release will have the fix for agw so you don't need conditional logic: https://github.com/agentgateway/agentgateway/actions/runs/37940656707

Quentin Bisson (QuentinBisson) · 2026-10-09T14:03:40Z

Thanks Eitan Yarmush (@EItanya) I'll take a look on monday

Eitan Yarmush (EItanya) · 2026-10-09T14:11:48Z

Thanks Eitan Yarmush (Eitan Yarmush (@EItanya)) I'll take a look on monday

sounds good, btw I brought this whole issue up in the community meeting and we're doing our best to get it done for GA

Quentin Bisson (QuentinBisson) · 2026-10-13T09:17:59Z

Eitan Yarmush (@EItanya) sorry, i got sick yesterday and had to postpone the fix. Do you know when GA is coming?

Quentin Bisson (QuentinBisson) · 2026-10-20T16:30:57Z

Eitan Yarmush (@EItanya) would you have some capacity to review this one ?

A memory restore resumes past the probe's boot fetch. The boot-fetch probe now fails its wakeup probe until the restored actor has fetched again as itself, so a restore that tunnels egress only after the wakeup probe fails the test.

Eitan Yarmush (EItanya) · 2026-10-22T13:25:21Z

🤖 AI-generated comment.

To clarify the remaining golden-policy comment: WithGoldenEgressPolicy still waits for the golden actor to appear and attaches its policy after boot may have started. Can we set goldenEgressPolicy on the fixture's ActorTemplate, using the API from #2159? The controller then creates the policy before starting the golden actor, so we can remove WithGoldenEgressPolicy/ensureGoldenEgressPolicy and the retries added for that race. This case needs the GoldenEgressPolicy preview gate and client opt-in.

The fixture template sets goldenEgressPolicy, so the controller creates the golden actor's EgressPolicy before it resumes the actor. This drops WithGoldenEgressPolicy and the boot fetch retries that covered a policy landing after the golden boot started. The test needs the GoldenEgressPolicy preview gate. Signed-off-by: QuentinBisson <[email protected]>

Quentin Bisson (QuentinBisson) · 2026-10-22T13:45:27Z

Thanks, addressed in 37d3df4: the template now sets goldenEgressPolicy, so WithGoldenEgressPolicy and the fetch retries are gone (the test needs the GoldenEgressPolicy gate). The placeholders were fixed in b8be927, and 3a06d83 adds the restore case.

On dialer == nil: not a new fix. On main, a connection that arrives before Activate is already dropped silently by the expiry check (zero expiresAt). Since this PR logs that branch as "certificate expired", the check keeps the log reason correct.

Eitan Yarmush (EItanya)

🤖 AI-generated review.

One remaining restore retry issue to fix before approval, plus a small, nonblocking test-command correction.

Eitan Yarmush (EItanya) · 2026-10-22T14:03:55Z

 	// processes.
+	// As in coldBootActor: egress before the VM restores, ingress after the
+	// wakeup probe.
+	if err := s.tunnel.ActivateEgress(p.attribution(), egress); err != nil {


With egress active before Resume, a restored background task can complete an external POST while dialAgentRetry or reseedGuestCRNG subsequently fails. Both still go through restoreGuestReadyError, which returns Unavailable. Atelet preserves that status, so a retry restores the same snapshot and can repeat the POST. The helper's retry rationale explicitly relies on outbound TCP being blocked until activation, which is no longer true.

Can we keep early egress but stop classifying these failures as safe to retry once the guest may have run, and add a regression test for that transition?

Fixed, a bit differently from what you suggested. A restore from a memory snapshot now gets egress after the CRNG reseed, just before the wakeup probe, instead of before Resume.

Why: before the reseed, every clone restored from the same snapshot has the same frozen "random" state. Opening egress then could let those identical values leave the host, which is what the reseed is there to prevent. It also means a failure during Resume, the agent dial or the reseed happens before any egress, so retrying it can't resend a request and the existing retry stays. Making those failures non-retryable instead would have crashed the actor on any short host stall. The restored workload is already past its boot, so it only needs egress once the probe starts.

A wake cut short during the probe can still be retried from the snapshot. The api-guide now says requests made while waking may be sent again. The run command now names the GoldenEgressPolicy gate.

Eitan Yarmush (EItanya) · 2026-10-22T14:03:55Z

+// policy before the controller resumes it. An actor restored from that snapshot then has to fetch again
+// as itself before its wakeup probe passes, which covers the restore paths.
+//
+//	hack/run-e2e-kind.sh ./internal/e2e/suites/egressmitm -run TestGoldenBootEgress -v -args --no-color


Small nit: can we add --preview=GoldenEgressPolicy to this command? RequirePreview skips the test without it, so copying the documented command doesn't exercise golden boot egress. The cluster also needs that gate enabled.

Quentin Bisson (QuentinBisson) · 2026-10-22T14:12:05Z

Sure I'm on it

Egress opened before Resume let a restored guest send traffic drawn from CRNG state it shares with the snapshot's other clones, and let a request complete before a failure that atelet retries from the same snapshot. A memory restore is past the workload's boot, so it gets egress after the reseed, just before the wakeup probe; cold boots and gVisor restores keep egress before the workload starts. The api-guide says a wake retried after egress was active can resend outbound requests. atunnel's per-connection drop logs move to Debug with the actor UID, since renewal already warns once. The fixture render test fails on a placeholder left unfilled, and the boot egress test's run command names the GoldenEgressPolicy preview gate.

Lior Lieberman (LiorLieberman)

LGTM overall, some nits

Lior Lieberman (LiorLieberman) · 2026-10-22T16:00:11Z

+// appRestore time the cold start that stands in for the restore. activate sums
+// the tunnel's egress half, armed before the containers restore, and its
+// ingress half, after the wakeup probe.


nit: can we humanize this comment just a bit?

What do you think about:

Suggested change

// appRestore time the cold start that stands in for the restore. activate sums

// the tunnel's egress half, armed before the containers restore, and its

// ingress half, after the wakeup probe.

// appRestore time the cold start that stands in for the restore. activate is

// the total time spent activating the tunnel: egress is activated before the

// containers restore and ingress after the wakeup probe, and both are counted.

Lior Lieberman (LiorLieberman) · 2026-10-22T16:01:20Z

+		// From inside the sandbox this is a connection that opens and closes at
+		// once, indistinguishable from a network failure. Debug, since the actor
+		// sets the rate and an actor without a gateway gets here on every


can we humanize this comment a bit?

Suggested change

// From inside the sandbox this is a connection that opens and closes at

// once, indistinguishable from a network failure. Debug, since the actor

// sets the rate and an actor without a gateway gets here on every

// The actor sees this as a connection that closes right away, like a

// network failure. This logs at Debug because an actor without an

// egress gateway hits this on every outbound connection.

Eitan Yarmush (EItanya) · 2026-10-22T16:36:49Z

cla is failing

Eitan Yarmush (EItanya) · 2026-10-22T17:07:53Z

Also:

Golden-boot e2e still fails. The latest completed preview job (https://github.com/agent-substrate/substrate/actions/runs/39797111141/job/119931050165) failed TestGoldenBootEgress with GoldenActorCrashed. Subsequent commits changed only comments. The crash needs diagnosis and a passing run.
The microVM regression test is still missing. Egress correctly activates after reseeding, but no test verifies that a failed reseed keeps outbound traffic blocked

Co-authored-by: Lior Lieberman <[email protected]>

Lior Lieberman (LiorLieberman) · 2026-10-22T17:11:31Z

fixed CLA

The origin's certificate covers only <name>.<namespace>.svc, which the actor resolves through its search path. The DNS relay answers only names the EgressPolicy lists, so the golden and restored actors' policies also list the .svc.cluster.local name the search path expands to.

Quentin Bisson (QuentinBisson) · 2026-10-22T19:56:36Z

Can you take another look?

Eitan Yarmush (EItanya) · 2026-10-22T20:06:11Z

+	if err := reseedThenActivateEgress(ctx, guestAC, func() error {
+		return s.tunnel.ActivateEgress(p.attribution(), egress)
+	}); err != nil {


nit: Why does this need to be a callback with this, feels like claude trying to inject a test seam

Eitan Yarmush (EItanya)

once you fix that test seam issue i think I'm good with this

Quentin Bisson (QuentinBisson) · 2026-10-22T20:40:25Z

It was added so we could test that a failed reseed leaves egress off in the unit test but it's a seam that only exists for tests so lets remove it

Quentin Bisson (QuentinBisson) · 2026-10-22T22:19:05Z

🎉 Thank you

Quentin Bisson (QuentinBisson) force-pushed the fix/actor-egress-during-boot branch 2 times, most recently from f788639 to 8151b8d Compare October 2, 2026 13:08

Quentin Bisson (QuentinBisson) marked this pull request as ready for review October 2, 2026 13:12

Quentin Bisson (QuentinBisson) force-pushed the fix/actor-egress-during-boot branch from 8151b8d to de5bd12 Compare October 2, 2026 13:23

Maya Wang (mayawang) added the kind/bug Something isn't working / bugfixes label Oct 2, 2026

Maya Wang (mayawang) requested a review from Lior Lieberman (LiorLieberman) October 2, 2026 17:46

Maya Wang (mayawang) added area/network area/security Security related issue/pr labels Oct 2, 2026

Quentin Bisson (QuentinBisson) force-pushed the fix/actor-egress-during-boot branch from de5bd12 to d689539 Compare October 8, 2026 08:29

Lior Lieberman (LiorLieberman) requested a review from Eitan Yarmush (EItanya) October 8, 2026 21:30

Quentin Bisson (QuentinBisson) force-pushed the fix/actor-egress-during-boot branch from d689539 to a45ab26 Compare October 8, 2026 21:34

Quentin Bisson (QuentinBisson) mentioned this pull request Oct 9, 2026

fix(substrate): admit a resuming actor's egress at CONNECT time agentgateway/agentgateway#3876

Closed

1 task

Eitan Yarmush (EItanya) mentioned this pull request Oct 9, 2026

Allow substrate egress while actors resume agentgateway/agentgateway#3882

Merged

1 task

Quentin Bisson (QuentinBisson) force-pushed the fix/actor-egress-during-boot branch from 60dd5ab to df525eb Compare October 13, 2026 09:13

Quentin Bisson (QuentinBisson) mentioned this pull request Oct 13, 2026

feat(skills): private Git skill sources via egress gateway credentials kagent-dev/kagent#3099

Draft

Quentin Bisson (QuentinBisson) force-pushed the fix/actor-egress-during-boot branch 3 times, most recently from eef5295 to 5bf56a7 Compare October 18, 2026 12:34

This was referenced Oct 19, 2026

docs(fork): ledger facts as of 2026-10-19 giantswarm/substrate#477

Merged

docs(fork): the ledger follows the upstream moves of 2026-W43 giantswarm/kagent-upstream#750

Merged

Quentin Bisson (QuentinBisson) force-pushed the fix/actor-egress-during-boot branch from 5bf56a7 to ddb320f Compare October 20, 2026 19:14

Cover egress during restore in the boot egress test

3a06d83

A memory restore resumes past the probe's boot fetch. The boot-fetch probe now fails its wakeup probe until the restored actor has fetched again as itself, so a restore that tunnels egress only after the wakeup probe fails the test.

Quentin Bisson (QuentinBisson) force-pushed the fix/actor-egress-during-boot branch from abb2ed5 to 3a06d83 Compare October 22, 2026 13:12

Eitan Yarmush (EItanya) reviewed Oct 22, 2026

View reviewed changes

Lior Lieberman (LiorLieberman) reviewed Oct 22, 2026

View reviewed changes

Quentin Bisson (QuentinBisson) and others added 2 commits October 22, 2026 10:10

Update cmd/ateworker-gvisor/main.go

9c1858f

Co-authored-by: Lior Lieberman <[email protected]>

Update cmd/ateworker-gvisor/main.go

089f0fa

Co-authored-by: Lior Lieberman <[email protected]>

Lior Lieberman (LiorLieberman) force-pushed the fix/actor-egress-during-boot branch from f948134 to 089f0fa Compare October 22, 2026 17:11

Quentin Bisson (QuentinBisson) added 3 commits October 22, 2026 19:33

Reword the egress activation and dropped-connection comments

54a5c60

Test that a failed CRNG reseed leaves a restored micro-VM without egress

702588f

Eitan Yarmush (EItanya) reviewed Oct 22, 2026

View reviewed changes

Activate a restored micro-VM's egress directly after the CRNG reseed

39be331

Eitan Yarmush (EItanya) approved these changes Oct 22, 2026

View reviewed changes

Eitan Yarmush (EItanya) enabled auto-merge October 22, 2026 21:49

Eitan Yarmush (EItanya) added this pull request to the merge queue Oct 22, 2026

Merged via the queue into agent-substrate:main with commit ddab01e Oct 22, 2026
15 checks passed

Quentin Bisson (QuentinBisson) mentioned this pull request Oct 26, 2026

docs(fork): record the upstream merges and closes of 2026-10-21 to 2026-10-23 on four rows giantswarm/substrate#543

Merged

Conversation

Quentin Bisson (QuentinBisson) commented Oct 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Eitan Yarmush (EItanya) commented Oct 6, 2026

Uh oh!

Quentin Bisson (QuentinBisson) commented Oct 6, 2026

Uh oh!

Quentin Bisson (QuentinBisson) commented Oct 8, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Eitan Yarmush (EItanya) commented Oct 9, 2026

Uh oh!

Quentin Bisson (QuentinBisson) commented Oct 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Eitan Yarmush (EItanya) commented Oct 9, 2026

Uh oh!

Quentin Bisson (QuentinBisson) commented Oct 13, 2026

Uh oh!

Quentin Bisson (QuentinBisson) commented Oct 20, 2026

Uh oh!

Eitan Yarmush (EItanya) commented Oct 22, 2026

Uh oh!

Quentin Bisson (QuentinBisson) commented Oct 22, 2026

Uh oh!

Eitan Yarmush (EItanya) left a comment

Choose a reason for hiding this comment

Uh oh!

Eitan Yarmush (EItanya) Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

Quentin Bisson (QuentinBisson) Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

Eitan Yarmush (EItanya) Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

Quentin Bisson (QuentinBisson) commented Oct 22, 2026

Uh oh!

Lior Lieberman (LiorLieberman) left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Lior Lieberman (LiorLieberman) Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

Quentin Bisson (QuentinBisson) Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Lior Lieberman (LiorLieberman) Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

Quentin Bisson (QuentinBisson) Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Eitan Yarmush (EItanya) commented Oct 22, 2026

Uh oh!

Eitan Yarmush (EItanya) commented Oct 22, 2026

Uh oh!

Lior Lieberman (LiorLieberman) commented Oct 22, 2026

Uh oh!

Quentin Bisson (QuentinBisson) commented Oct 22, 2026

Uh oh!

Eitan Yarmush (EItanya) Oct 22, 2026

Choose a reason for hiding this comment

Uh oh!

Eitan Yarmush (EItanya) left a comment

Choose a reason for hiding this comment

Uh oh!

Quentin Bisson (QuentinBisson) commented Oct 22, 2026

Uh oh!

Quentin Bisson (QuentinBisson) commented Oct 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Quentin Bisson (QuentinBisson) commented Oct 2, 2026 •

edited

Loading

Quentin Bisson (QuentinBisson) commented Oct 8, 2026 •

edited

Loading

Quentin Bisson (QuentinBisson) commented Oct 9, 2026 •

edited

Loading

Quentin Bisson (QuentinBisson) Oct 22, 2026 •

edited

Loading

Quentin Bisson (QuentinBisson) Oct 22, 2026 •

edited

Loading

Quentin Bisson (QuentinBisson) commented Oct 22, 2026 •

edited

Loading