Skip to content

Commit 05de980

Browse files
teemowQuentinBisson
authored andcommitted
fix(atenet,ateom): let an actor's egress through while it resumes — the golden boot fetches before readyz (#4)
* Let an actor's egress through while it resumes An actor's tunneled egress was armed only after every container served readyz, and atenet's egress gateway refused a CONNECT from any actor that was not RUNNING — a state the control plane commits only after the worker's Run returned, that is, after readyz. A workload that fetches what it needs to become ready — a model, a skill repository, packages — therefore could not: atunnel dropped the intercepted connection on the floor, the fetch died with "Broken pipe", readyz never came, and the actor was re-run every minute with one worker pinned to it. Nothing was logged at either hop; the workload's own error was the only trace. An ActorTemplate whose workload fetches at start-up never got its golden snapshot. The certificate the tunnel authenticates with is minted for exactly this placement before the workload starts (prepareActorEgress; the credential broker mints only for a worker's current assignment), so the identity is there from the first packet. Readiness gates ingress, not egress. * ateom (gVisor and micro-VM): arm egress right after the actor network is set up and the failure cleanup is registered, before the first container starts; ingress still waits for readyz. * atenet egress: a RESUMING actor is placed on a worker and may tunnel. SUSPENDED, PAUSED, CRASHED and DELETING are still refused: a certificate within its lifetime must not tunnel for an actor that has left its worker. The refusal is logged with the actor, its state and the destination. * atunnel: a dropped egress connection (no active egress, expired certificate) is logged with the peer address. * docs(fork): ledger row for the egress-while-resuming patch (#4)
1 parent b8a1e24 commit 05de980

9 files changed

Lines changed: 141 additions & 25 deletions

File tree

‎FORK.md‎

Lines changed: 5 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -37,10 +37,13 @@ Everything on `giantswarm` that is not in the pin (`git log v0.0.26..giantswarm`
3737
| Patch | Purpose | Fork commit | Upstream |
3838
|---|---|---|---|
3939
| Grant atelet cluster-wide read access to sandbox configs | atelet's sandbox-asset prewarm degraded on the second test cluster without the RBAC ([#37742](https://github.com/giantswarm/giantswarm/issues/37742) row 10) | `74b45f9e` (`git cherry-pick -x a7505e9c`) | [kagent-dev/substrate#33](https://github.com/kagent-dev/substrate/pull/33), merged 2026-09-08, not in v0.0.26 — falls away at the re-pin onto the first tag that contains it |
40+
| Let an actor's egress through while it resumes (ateom arms tunneled egress before the first container starts, atenet admits `RESUMING` actors, both hops log a refusal) | an actor whose workload fetches what it needs to become ready — kagent's Go ADK and Claude harnesses materialise git skills before readyz — never got its golden snapshot: atunnel dropped the fetch (`Broken pipe`), atenet would have refused a non-`RUNNING` actor, nothing was logged ([#37742](https://github.com/giantswarm/giantswarm/issues/37742) rows 8 and 13; acceptance test `agentlab skills-test`, [agentlab#137](https://github.com/giantswarm/agentlab/issues/137)) | [#4](https://github.com/giantswarm/substrate/pull/4) (squash; `git log --grep '(#4)'`) | to file: the upstream-shaped patch is branch [`upstream/atenet-egress-during-resume`](https://github.com/giantswarm/substrate/tree/upstream/atenet-egress-during-resume) here, rebased on the mirror `main`; a team member opens the kagent-dev/substrate pull request with DCO sign-off once #37742 has reviewed it |
4041
| Fork infrastructure: this file, the README pointer, `CODEOWNERS`, `.github/workflows/publish.yaml`, `.github/workflows/sync-upstream.yaml`, `.trivyignore`, and the branch triggers of `pr-workflow.yaml`, `helm-e2e.yaml`, `govulncheck.yaml` (`main` → `giantswarm`, govulncheck also on pull requests) | the line's CI, publishing and sync | the `giantswarm` branch history | not for upstream |
4142

42-
Nothing in the line changes Substrate's behaviour beyond what upstream has already merged. Giant Swarm
43-
specific wiring lives elsewhere: the CA/JWT pool bootstrap (`kubectl-ate admin make-ca-pool`/`make-jwt-pool`
43+
One patch changes Substrate's behaviour ahead of upstream — egress for an actor while it resumes, without
44+
which no skill-carrying agent of the platform boots; it is written for upstream and leaves at the first
45+
release that carries it. Everything else is what upstream has already merged. Giant Swarm specific wiring
46+
lives elsewhere: the CA/JWT pool bootstrap (`kubectl-ate admin make-ca-pool`/`make-jwt-pool`
4447
and the `ate-api-authentication` ConfigMap) is created by [agentlab](https://github.com/giantswarm/agentlab)
4548
and by meta chart 4.0; the `WorkerPool` the platform's Harnesses run on comes with the kagent chart
4649
(`kagent.substrateWorkerPool`); feature gates, Kyverno exceptions and network policies are cluster

‎cmd/atenet/internal/router/README.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,11 @@ Router has several responsibilities:
2626
* Authenticates actor identity on egress: on every CONNECT, the egress
2727
gateway's ext_proc handler re-verifies the actor's client certificate against
2828
the actor-identity CA, reads the `ActorIdentity` X.509 extension out of it,
29-
and checks the certified UID against the ATE API.
29+
and checks against the ATE API that the certified UID is the actor's and that
30+
the actor is placed on a worker (running, or resuming onto the worker that
31+
minted the certificate — what a workload fetches to become ready goes out
32+
before it serves readyz). Every denial is logged with the actor and the
33+
destination.
3034
* Authorizes egress against the actor's `EgressPolicy`: every request the
3135
gateway can read is decided on its `Host` and the address the actor dialed,
3236
rules in order, first match wins; what it cannot read is decided by address

‎cmd/atenet/internal/router/egress/egress.go‎

Lines changed: 29 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -171,7 +171,9 @@ func (h *Handler) handleConnect(ctx context.Context, md *extproc.RequestMetadata
171171
if err := validateIdentity(identity); err != nil {
172172
return extproc.Result{}, err
173173
}
174-
if err := h.validateActor(ctx, identity); err != nil {
174+
// For a CONNECT the :authority is the actor's original destination
175+
// (IP:port).
176+
if err := h.validateActor(ctx, identity, md.Host); err != nil {
175177
return extproc.Result{}, err
176178
}
177179

@@ -280,9 +282,11 @@ func validateIdentity(identity *substratex509.ActorIdentity) error {
280282

281283
// validateActor checks the identity a certificate certifies against the control
282284
// plane's current view of that actor: it still exists, it is the actor the
283-
// certificate was issued to, and it is running. Every error it returns is
284-
// already a client-facing ext_proc denial.
285-
func (h *Handler) validateActor(ctx context.Context, identity *substratex509.ActorIdentity) error {
285+
// certificate was issued to, and it is placed on a worker — running, or
286+
// resuming onto the worker that minted the certificate. Every error it returns
287+
// is already a client-facing ext_proc denial; destination is the CONNECT's
288+
// target, for the log.
289+
func (h *Handler) validateActor(ctx context.Context, identity *substratex509.ActorIdentity, destination string) error {
286290
atespace := identity.Atespace
287291
actorName := identity.ActorName
288292
actorUID := identity.ActorUid
@@ -309,12 +313,29 @@ func (h *Handler) validateActor(ctx context.Context, identity *substratex509.Act
309313
"egress denied: actor %q/%q is not the actor this certificate was issued to", atespace, actorName)
310314
}
311315

312-
// The actor performing egress must actually be running.
313-
if actor.GetStatus().GetState() != ateapipb.ActorState_ACTOR_STATE_RUNNING {
316+
// The actor performing egress must be placed on a worker. RUNNING is the
317+
// steady state; RESUMING is the workload booting or restoring on the
318+
// worker that minted this certificate for exactly that placement (the
319+
// credential broker mints only for a worker's current assignment), and a
320+
// workload that fetches what it needs to become ready — models, skills,
321+
// packages — does so before it serves readyz. Every other state means the
322+
// actor has left its worker or is leaving it, and a certificate still
323+
// within its lifetime must not open a tunnel on its behalf.
324+
switch state := actor.GetStatus().GetState(); state {
325+
case ateapipb.ActorState_ACTOR_STATE_RUNNING, ateapipb.ActorState_ACTOR_STATE_RESUMING:
326+
return nil
327+
default:
328+
// Logged with the destination: from inside the sandbox this denial
329+
// is indistinguishable from a network failure.
330+
slog.WarnContext(ctx, "egress denied: actor is not placed on a worker",
331+
slog.String("atespace", atespace),
332+
slog.String("actor", actorName),
333+
slog.String("actorUid", actorUID),
334+
slog.String("state", state.String()),
335+
slog.String("destination", destination))
314336
return extproc.NewReqError(envoy_type.StatusCode_Forbidden,
315-
"egress denied: actor %q/%q is %s, not running", atespace, actorName, actor.GetStatus().GetState())
337+
"egress denied: actor %q/%q is %s, not placed on a worker", atespace, actorName, state)
316338
}
317-
return nil
318339
}
319340

320341
// authenticateActorCertificate turns the mTLS peer certificate Envoy recorded

‎cmd/atenet/internal/router/egress/egress_test.go‎

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -416,6 +416,21 @@ func TestConnectLegWithoutRequestLegsFailsClosed(t *testing.T) {
416416
}
417417
}
418418

419+
// A resuming actor is booting or restoring on the worker that minted its
420+
// certificate; what it fetches to become ready goes out before it serves
421+
// readyz, so the gateway lets it through.
422+
func TestHandleRequestHeadersAllowsResumingActor(t *testing.T) {
423+
ca := newTestCA(t, "actor-identity-ca")
424+
leaf := ca.issueActorCert(t, actorCertOptions{})
425+
resuming := runningActor()
426+
resuming.Status.State = ateapipb.ActorState_ACTOR_STATE_RESUMING
427+
h := egressHandler(ca.roots(), resuming, nil)
428+
429+
if _, err := h.HandleRequestHeaders(context.Background(), egressMetadata(xfccHeader(leaf))); err != nil {
430+
t.Fatalf("HandleRequestHeaders() error = %v, want nil", err)
431+
}
432+
}
433+
419434
func TestHandleRequestHeadersAllowsAgentgatewayCertificateAttribute(t *testing.T) {
420435
ca := newTestCA(t, "actor-identity-ca")
421436
leaf := ca.issueActorCert(t, actorCertOptions{})
@@ -674,7 +689,10 @@ func TestHandleRequestHeadersAuthorization(t *testing.T) {
674689
want: envoy_type.StatusCode_Forbidden,
675690
},
676691
{
677-
name: "actor is not running",
692+
// Suspended, paused and crashed actors have left their worker;
693+
// a deleting one is leaving it. Their certificates are still
694+
// within their lifetime and must not open a tunnel.
695+
name: "actor is suspended",
678696
actor: &ateapipb.Actor{
679697
Metadata: &ateapipb.ResourceMetadata{
680698
Atespace: testEgressAtespace,
@@ -685,6 +703,18 @@ func TestHandleRequestHeadersAuthorization(t *testing.T) {
685703
},
686704
want: envoy_type.StatusCode_Forbidden,
687705
},
706+
{
707+
name: "actor is being deleted",
708+
actor: &ateapipb.Actor{
709+
Metadata: &ateapipb.ResourceMetadata{
710+
Atespace: testEgressAtespace,
711+
Name: testEgressActor,
712+
Uid: testEgressActorUID,
713+
},
714+
Status: &ateapipb.ActorStatus{State: ateapipb.ActorState_ACTOR_STATE_DELETING},
715+
},
716+
want: envoy_type.StatusCode_Forbidden,
717+
},
688718
{
689719
name: "actor no longer exists",
690720
err: status.Error(codes.NotFound, "no such actor"),

‎cmd/ateom-gvisor/main.go‎

Lines changed: 30 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -713,6 +713,11 @@ func (s *AteomService) RunWorkload(ctx context.Context, req *ateompb.RunWorkload
713713
}
714714
}
715715
}()
716+
// Egress is armed before the first container starts (see
717+
// activateActorEgress); ingress waits for readyz below.
718+
if err := s.activateActorEgress(egress); err != nil {
719+
return nil, err
720+
}
716721
// Create and start pause container. The bundle rootfs is composed here —
717722
// an overlay of the node's cached image layers plus the bundle's private
718723
// upper — because mounting is ateom's job (atelet runs with no
@@ -753,7 +758,7 @@ func (s *AteomService) RunWorkload(ctx context.Context, req *ateompb.RunWorkload
753758
if err := readyz.WaitAll(ctx, req.GetSpec().GetContainers(), ateomnet.ActorVethIP); err != nil {
754759
return nil, fmt.Errorf("while waiting for container readyz: %w", err)
755760
}
756-
if err := s.activateActorNetworking(req.GetAtespace(), req.GetActorName(), egress); err != nil {
761+
if err := s.activateActorIngress(req.GetAtespace(), req.GetActorName()); err != nil {
757762
return nil, err
758763
}
759764

@@ -1008,6 +1013,10 @@ func (s *AteomService) RestoreWorkload(ctx context.Context, req *ateompb.Restore
10081013
}
10091014
}
10101015
}()
1016+
// As in RunWorkload: egress from the first packet, ingress after readyz.
1017+
if err := s.activateActorEgress(egress); err != nil {
1018+
return nil, err
1019+
}
10111020
checkpointDir := ateompath.RestoreStateDir(req.GetActorUid())
10121021

10131022
if hasDurableVolumes(req.GetSpec().GetContainers()) {
@@ -1082,7 +1091,7 @@ func (s *AteomService) RestoreWorkload(ctx context.Context, req *ateompb.Restore
10821091
if err := readyz.WaitAll(ctx, req.GetSpec().GetContainers(), ateomnet.ActorVethIP); err != nil {
10831092
return nil, fmt.Errorf("while waiting for container readyz: %w", err)
10841093
}
1085-
if err := s.activateActorNetworking(req.GetAtespace(), req.GetActorName(), egress); err != nil {
1094+
if err := s.activateActorIngress(req.GetAtespace(), req.GetActorName()); err != nil {
10861095
return nil, err
10871096
}
10881097

@@ -1197,10 +1206,16 @@ func (s *AteomService) terminateWorkload(ctx context.Context, actorRef resources
11971206
return errors.Join(errs...)
11981207
}
11991208

1200-
func (s *AteomService) activateActorNetworking(atespace, actorName string, egress *actorEgress) error {
1201-
if err := s.atunnelIngress.Activate(atespace, actorName); err != nil {
1202-
return fmt.Errorf("while activating actor ingress: %w", err)
1203-
}
1209+
// activateActorEgress arms tunneled egress before the workload's first
1210+
// container starts. The certificate was minted for this placement in
1211+
// prepareActorEgress, so the tunnel carries the actor's identity from its
1212+
// first packet, and what a workload fetches to become ready — models, skills,
1213+
// packages — goes out before it serves readyz. Armed after the workload-failure
1214+
// cleanup is registered, so a boot that dies disarms it again. Ingress is the
1215+
// other way round (activateActorIngress): nothing is routed to a workload
1216+
// before it is ready. A nil egress is an activation without a gateway, whose
1217+
// actor egress stays on the masquerade path.
1218+
func (s *AteomService) activateActorEgress(egress *actorEgress) error {
12041219
if egress == nil {
12051220
return nil
12061221
}
@@ -1210,6 +1225,15 @@ func (s *AteomService) activateActorNetworking(atespace, actorName string, egres
12101225
return nil
12111226
}
12121227

1228+
// activateActorIngress admits inbound traffic once every readyz-enabled
1229+
// container of the workload reports ready.
1230+
func (s *AteomService) activateActorIngress(atespace, actorName string) error {
1231+
if err := s.atunnelIngress.Activate(atespace, actorName); err != nil {
1232+
return fmt.Errorf("while activating actor ingress: %w", err)
1233+
}
1234+
return nil
1235+
}
1236+
12131237
func deleteContainers(ctx context.Context, rcmd *runsc, containers []string, operation string) {
12141238
for _, container := range slices.Backward(containers) {
12151239
if err := rcmd.cmdDelete(ctx, container); err != nil {

‎cmd/ateom-microvm/main.go‎

Lines changed: 19 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -571,10 +571,16 @@ func (s *AteomService) prepareActorEgress(ctx context.Context, actorAtespace, ac
571571
return &actorEgress{client: gatewayClient, certificateSource: certificateSource, expiresAt: expiresAt}, nil
572572
}
573573

574-
func (s *AteomService) activateActorNetworking(atespace, actorName string, egress *actorEgress) error {
575-
if err := s.atunnelIngress.Activate(atespace, actorName); err != nil {
576-
return fmt.Errorf("while activating actor ingress: %w", err)
577-
}
574+
// activateActorEgress arms tunneled egress before the workload's first
575+
// container starts. The certificate was minted for this placement in
576+
// prepareActorEgress, so the tunnel carries the actor's identity from its
577+
// first packet, and what a workload fetches to become ready — models, skills,
578+
// packages — goes out before it serves readyz. Armed after the workload-failure
579+
// cleanup is registered, so a boot that dies disarms it again. Ingress is the
580+
// other way round (activateActorIngress): nothing is routed to a workload
581+
// before it is ready. A nil egress is an activation without a gateway, whose
582+
// actor egress stays on the masquerade path.
583+
func (s *AteomService) activateActorEgress(egress *actorEgress) error {
578584
if egress == nil {
579585
return nil
580586
}
@@ -584,6 +590,15 @@ func (s *AteomService) activateActorNetworking(atespace, actorName string, egres
584590
return nil
585591
}
586592

593+
// activateActorIngress admits inbound traffic once every readyz-enabled
594+
// container of the workload reports ready.
595+
func (s *AteomService) activateActorIngress(atespace, actorName string) error {
596+
if err := s.atunnelIngress.Activate(atespace, actorName); err != nil {
597+
return fmt.Errorf("while activating actor ingress: %w", err)
598+
}
599+
return nil
600+
}
601+
587602
func (s *AteomService) deactivateActorNetworking(ctx context.Context) error {
588603
// Stop admitting traffic and drain active streams before the Actor network
589604
// is torn down. Attempt both directions even if one fails to deactivate.

‎cmd/ateom-microvm/restore.go‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -291,6 +291,10 @@ func (s *AteomService) restoreFullScope(ctx context.Context, p actorBootParams,
291291
}
292292
}
293293
}()
294+
// As in coldBootActor: egress from the first packet, ingress after readyz.
295+
if err := s.activateActorEgress(egress); err != nil {
296+
return err
297+
}
294298
netDevs, err := ch.SnapshotNetDevices(restoreDir)
295299
if err != nil {
296300
return fmt.Errorf("while reading snapshot net devices: %w", err)
@@ -416,7 +420,7 @@ func (s *AteomService) restoreFullScope(ctx context.Context, p actorBootParams,
416420
}
417421
}
418422

419-
if err := s.activateActorNetworking(p.actorRef.Atespace, p.actorRef.Name, egress); err != nil {
423+
if err := s.activateActorIngress(p.actorRef.Atespace, p.actorRef.Name); err != nil {
420424
return err
421425
}
422426
s.running[actorUID] = ra

‎cmd/ateom-microvm/run.go‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -436,6 +436,11 @@ func (s *AteomService) coldBootActor(ctx context.Context, p actorBootParams) (re
436436
}
437437
}
438438
}()
439+
// Egress is armed before the guest boots (see activateActorEgress);
440+
// ingress waits for readyz below.
441+
if err := s.activateActorEgress(egress); err != nil {
442+
return err
443+
}
439444

440445
// Guest sizing + agent kernel params from the kata config.
441446
memMiB, vcpus, kparams, err := s.guestConfig(rr)
@@ -608,7 +613,7 @@ func (s *AteomService) coldBootActor(ctx context.Context, p actorBootParams) (re
608613
slog.Duration("since_boot", time.Since(tBooted)))
609614

610615
ra := &runningActor{chCmd: chCmd, vfsdCmd: vfsdCmd, apiSocket: apiSocket, baseID: actorUID, guestAgent: ac, workloadIDs: workloadIDs(ctrs)}
611-
if err := s.activateActorNetworking(p.actorRef.Atespace, p.actorRef.Name, egress); err != nil {
616+
if err := s.activateActorIngress(p.actorRef.Atespace, p.actorRef.Name); err != nil {
612617
return err
613618
}
614619
s.running[actorUID] = ra

‎internal/atunnel/egress.go‎

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -243,13 +243,23 @@ func (e *Egress) handle(downstream net.Conn) {
243243
active := e.active
244244
if active == nil {
245245
e.mu.Unlock()
246+
// Dropped, and said so: from inside the sandbox a dropped connection
247+
// looks like a network failure, and this is the only place that knows
248+
// why. The original destination is deliberately not resolved for a
249+
// connection nobody vouches for; the peer address ties the line to
250+
// the actor's own log.
251+
slog.Warn("atunnel dropped an egress connection: no actor egress is active",
252+
slog.String("from", downstream.RemoteAddr().String()))
246253
_ = downstream.Close()
247254
return
248255
}
249-
if time.Now().Compare(active.expiresAt) >= 0 {
256+
if expiresAt := active.expiresAt; time.Now().Compare(expiresAt) >= 0 {
250257
// Expiry blocks only new tunnels. Connections admitted with a valid
251258
// certificate have completed mTLS and are allowed to drain normally.
252259
e.mu.Unlock()
260+
slog.WarnContext(active.ctx, "atunnel dropped an egress connection: the actor certificate has expired",
261+
slog.String("from", downstream.RemoteAddr().String()),
262+
slog.Time("expiredAt", expiresAt))
253263
_ = downstream.Close()
254264
return
255265
}

0 commit comments

Comments
 (0)