After a node restart, ate-api-server keeps dialing the old atelet IP until it restarts #2116

What happens: After a node restart, ate-api-server keeps dialing the old IP of that node's atelet. Every atelet call for the node fails with Unavailable: restore (resume), terminate, and delete --any-state. An actor that was on that node cannot be resumed or deleted. It stays ACTOR_STATE_CRASHED or ACTOR_STATE_DELETING and keeps its worker. The only recovery we found is a restart of ate-api-server.

Cause: DialForAteletOnNode looks up the current atelet pod for the node (#1742). But it checks the connection cache first, and the cache key is the atelet pod UID (dialer.go#L123-L148):

ateletKey := string(selectedAtelet.ObjectMeta.UID)
ateletConnAny, ok := d.ateletConns.Get(ateletKey)
if ok {
    return ateletConnAny.(*grpc.ClientConn), nil
}
...
grpc.NewClient(net.JoinHostPort(selectedAtelet.Status.PodIPs[0].IP, ...), ...)

A node restart without a drain restarts the atelet pod in place. The pod keeps its UID and gets a new IP. So the lookup finds the right pod, the cache returns the old connection, and that connection targets the old IP literal. The LRU (size 1024) evicts only on capacity, so the stale entry stays until ate-api-server restarts.

Evidence (kind, 1 control-plane node and 2 worker nodes, Substrate 1d7ca8c):

1. Atelet pod on kind-worker2, before and after docker stop kind-worker2 && docker start kind-worker2: UID 3c6bf412-… both times, IP 10.244.1.7 then 10.244.1.4. The worker pods on the node did the same ([Bug]: Actor's worker IP is not revalidated after a node reboot — stays RUNNING, unreachable #1837).
2. In an earlier run, an actor was mid-resume on that node during the restart. Then:

   ResumeActor: workflow failed at step CallAteletRestore: while loading actor to crash: ... context canceled
   DeleteActor (any_state=true): while terminating atelet: workflow failed at step CallAteletTerminate: while terminating actor on atelet: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial tcp 10.244.1.2:8085: i/o timeout"
   

   The actor stayed ACTOR_STATE_DELETING for more than 5 minutes. The atelet pod on the node (same pod, RESTARTS 1) had IP 10.244.1.3 at that time, and a worker pod had 10.244.1.2.
3. kubectl -n ate-system rollout restart deploy/ate-api-server, then kubectl ate delete actor <name> -a default --any-state: the delete worked at once, and the worker was free again.
4. A second run, 6 actors, node restarted while they were suspended: the 2 actors placed on the restarted node failed with dial tcp 10.244.2.2:8085: connect: no route to host and stayed ACTOR_STATE_CRASHED. After an ate-api-server restart (and new worker pods for [Bug]: Actor's worker IP is not revalidated after a node reboot — stays RUNNING, unreachable #1837), all 6 resumed with their files intact.

The error changes with what now has the old IP: i/o timeout, connection refused, or no route to host.

Suggested fix: Make the cache key the pod UID and its IP, for example uid + "/" + ip. Or compare the cached connection's target with Status.PodIPs and close and replace it when they differ. Either way, the lookup that #1742 added then takes effect.

Related: #1837 is the worker-side half of the same restart: the worker pod keeps its UID and Substrate keeps the old worker IP (fix in review as #1983, which does not touch dialer.go). #710 changes the error codes while an atelet pod is replaced. This issue is about the same pod with a new IP. #1259 closes connections on eviction.

Environment: Agent Substrate 1d7ca8c on a local kind cluster (Docker Desktop on Apple Silicon), gVisor sandbox class, nodes restarted with docker stop and docker start. Upstream main (6a35150) has the same code in dialer.go.