Repository navigation
[Bug]: Actor's worker IP is not revalidated after a node reboot — stays RUNNING, unreachable #1837
Description
Title: [Bug]: Actor's worker IP is not revalidated after a node reboot — stays RUNNING, unreachable
Relatives: #1526 (worker pod disappears → CRASHED, unrecoverable), #1665/#1518/#50 (stuck in a transitional state after a failed lifecycle op). This is the step before either: the actor never leaves
RUNNING, silently pointing at a dead address.
What happened
After a full host reboot, worker pods come back Running with the same name but a new pod IP (fresh network sandbox). The actor controller never re-resolves it: kubectl ate get actor / ax describe task keep reporting RUNNING/Ready at the old IP. First sign of trouble is a request failing:
upstream reset: ... delayed connect error: No route to host
(also seen via ax ssh: rpc error: code = Unavailable ... remote connection failure)
Trying to fix it by hand makes it worse — suspend actor fails (runsc checkpoint: exit status 128) and strands the actor in SUSPENDING with no way out (#1665/#50). Only recovery: delete and recreate.
Repro
- Single-node k3s,
--kindprofile, WorkerPool with 2 replicas. - Create an actor (or an AX Task — same mechanism), confirm it's reachable, note its worker IP.
- Reboot the host.
- Worker pod: same name, new IP (
kubectl get pod <worker> -o widebefore/after). Actor: stillRUNNINGat the old IP → unreachable, andsuspendstrands it inSUSPENDING.
Reproduced identically for a plain counter demo actor and for an AX Task.
Expected
The controller should revalidate/re-resolve a RUNNING actor's worker IP (or treat a recreated sandbox like a disappeared worker, #1526) — not report healthy indefinitely with no health signal.
Environment
Substrate 672533541dbf (main), --kind profile, single-node k3s 1.36.4, also reproduced through AX v0.3.0.
Activity
BenTheElder commentedon Sep 23, 2026
If the node is rebooted under the actor that actor is no longer running and we need to treat is as crashed. That's a bug for sure.
Actors / their sandboxes are interior processes that won't automatically be recreated (and shouldn't as they'd have lost memory), but should be possible to resume from the last snapshot, so ~#1526 / #1675 .
The IP part is a red-herring.
thpham commentedon Sep 24, 2026
Thanks — agreed, the IP change is just a symptom of the sandbox being gone; the missing crash detection is the real issue.
RevertActor(#1675) looks like exactly the recovery path we need. We're pinned to672533541dbf(2026-09-11 — not 09-22 as I wrote above, sorry), which predates it, because that's what google/ax v0.3.0 requires. We'll pick it up once AX bumps.sfunkenhauser commentedon Sep 24, 2026
We mark an actor
CRASHEDif the pod gets deleted, which is what happens during a normal eviction flow. I'm guessing eviction didn't happen (Because the reboot was sudden?), the container was terminated and restarted by theReplicaSetcontroller. We need to mark an actor asCRASHEDif we notice the ateom has terminated and restarted.thpham commentedon Sep 24, 2026
Reproduced it on purpose to check the eviction theory — Scott Funkenhauser (@sfunkenhauser) is right, and there's a second half.
Single-node k3s, counter demo. Actor A suspended before the reboot, actor B left RUNNING. Plain
systemctl reboot, no drain.After reboot:
RESTARTS 1, new IPs — no eviction, containers restarted in place.RUNNINGat its old IP, never marked CRASHED.TODO: I don't think this is possiblebranch incmd/atecontroller/internal/workersync/syncer.go— here's a deterministic repro. Sinceipis immutable on a Worker, handling that mismatch like a pod deletion (drop the worker, CRASH its actor) would cover both halves.Workaround that works today: suspend actors → reboot → delete the worker pods (fresh pods re-register with correct IPs) → resume. A came back with its state intact (counter 3 → 4); B went CRASHED as expected.
sfunkenhauser commentedon Sep 25, 2026
That's a good point, there are 2 separate bugs here.
Worker IPs are never updated, which results in new actors being unreachable after they are scheduled. When an actor is assigned to a worker, the IP is copied to the actor object and used by atenet to route requests to the correct worker (ingress.go#123).
When ateom restarts, any running actors aren't restarted with it. These actors should be marked
CRASHED. This is being tracked by [Bug]: An ateom that restarts strands all running actors #1882BenTheElder commentedon Oct 14, 2026
x-ref: #2692 ... I think we need a slightly different approach, where again, the IP isn't so much the issue as correctly tracking worker liveness.
I'm also working on reporting actor failure when the pod doesn't crash, just the actor.
Detect worker restarts by an epoch the worker process registers (agen…