Shutdown sequence never finishes when a service does not stop: stopServices ends with an unbounded wait, so the node never powers off #14404

Activity

1. daixtrose commented on Sep 17, 2026

   

   daixtrose

   on Sep 17, 2026

   Author

   More actions

   [answer from Claude:] The reproducer is attached on the pull request rather than here: talos-shutdown-repro.zip on #14405.

   It is a deterministic Go unit test inside the Talos tree — no cluster, no Kubernetes, no KubeVirt, no root, and nothing to power off. run.sh runs the test twice against a clean checkout, first unpatched and then with fix.patch applied, and reverts the patch afterwards.
2. smira commented on Sep 18, 2026

   

   smira

   on Sep 18, 2026

   Member

   More actions

   Sorry, but this is AI slop, not an issue. Please submit real bug report with logs. We are not interested in code analysis (we can do it ourselves), we are interested in bugs and reproducers.
3. daixtrose commented on Sep 19, 2026

   

   daixtrose

   on Sep 19, 2026

   Author

   More actions

   Thanks for the feedback. I will investigate further and correct the flaws in the report. The issue is real, though, but I understand the provided repro only shows the indefinite wait, not the complete problem.
4. smira commented on Sep 19, 2026

   

   smira

   on Sep 19, 2026

   Member

   More actions

   I have no doubt that the issue is real, but can you provide the logs? A way to reproduce?
5. daixtrose commented on Sep 19, 2026

   

   daixtrose

   on Sep 19, 2026

   Author

   More actions

   I owe you an apology. The incident was real: on 2026-08-30 at 22:33Z and again at 01:04Z, talosctl shutdown --force on a v1.13.9 node logged Failed to remove cgroup /sys/fs/cgroup/podruntime/runtime: still contains running tasks and gave up on while stopping "cri", then went silent with apid still answering, and only the power button ended it. Two orphaned containerd-shim-runc-v2 processes with no children were parked in that cgroup.

   But the conclusions I drew from my experiments were wrong. I have since reproduced the orphaned shim on demand in a lab guest (node-level memory pressure through the OOM controller, cause containerd#13849, goroutine dump in hand), and with two such orphans present the same shutdown --force still powers the node off in about 60 s: Talos names the holders, kills them and completes. So the orphan alone does not hang the shutdown, and the reproducer I offered does not show the problem. I am sorry for the noise.

   One last thing, which I can show rather than argue. If the shim that hosts a Talos service stops responding, the shutdown never ends. The script at the bottom of this comment does exactly that on a disposable single-node cluster: it starts a privileged hostPID busybox pod, finds the containerd-shim-runc-v2 that hosts the kubelet service in the system namespace, freezes it with SIGSTOP (the kubelet process itself is left running), issues talosctl shutdown --force, and prints talosctl services every 30 s. This is its unedited output against a KubeVirt guest running Talos v1.13.9 with Kubernetes v1.36.4, with the identical middle readings cut:

   kubectl context : admin@lab-talos
   kubectl nodes   : node/lab-talos
   talosctl target : 192.168.177.50 ->     Tag:         v1.13.9
   == 2026-09-19T14:42:49Z 1. privileged hostPID pod
   namespace/freeze-demo created
   namespace/freeze-demo labeled
   pod/freezer created
   pod/freezer condition met
   == 2026-09-19T14:42:51Z 2. the shim that hosts the kubelet
   shim pid: 2500
   Name:    containerd-shim
   State:    S (sleeping)
   PPid:    1
   children:
   Name:    kubelet
   State:    S (sleeping)
   == 2026-09-19T14:42:51Z 3. SIGSTOP
   State:    T (stopped)
   == 2026-09-19T14:42:52Z 4. talosctl shutdown --force
   == 2026-09-19T14:42:52Z 5. watching for 15 minutes
   -- 2026-09-19T14:43:22Z
   NODE             SERVICE      STATE      HEALTH   LAST CHANGE   LAST EVENT
   192.168.177.50   apid         Running    OK       2m36s ago     Health check successful
   192.168.177.50   cri          Running    OK       2m36s ago     Health check successful
   192.168.177.50   kubelet      Stopping   Fail     19s ago       Sending SIGTERM to task kubelet (PID 2540, container kubelet)
   -- 2026-09-19T14:43:52Z
   NODE             SERVICE      STATE      HEALTH   LAST CHANGE   LAST EVENT
   192.168.177.50   apid         Running    OK       3m6s ago      Health check successful
   192.168.177.50   cri          Running    OK       3m6s ago      Health check successful
   192.168.177.50   kubelet      Stopping   Fail     49s ago       Sending SIGTERM to task kubelet (PID 2540, container kubelet)
   [... 27 further readings, every 30 s, identical apart from the ages ...]
   -- 2026-09-19T14:57:55Z
   NODE             SERVICE      STATE      HEALTH   LAST CHANGE   LAST EVENT
   192.168.177.50   apid         Running    OK       17m9s ago     Health check successful
   192.168.177.50   cri          Running    OK       17m9s ago     Health check successful
   192.168.177.50   kubelet      Stopping   Fail     14m52s ago    Sending SIGTERM to task kubelet (PID 2540, container kubelet)
   == 2026-09-19T14:57:55Z done; the node is still up if the last table shows kubelet Stopping. Hard-reset it.
   

   The node was still up at the end and needed a hard reset. On the console the sequence reaches phase cleanup (2/10), task stopAllPods (1/1): shutting down kubelet gracefully, prints service[kubelet](Stopping): Sending SIGTERM to task kubelet, and then nothing more; in an earlier run I watched it for 31 minutes. A second shutdown --force cancels the first (task stopAllPods (1/1): failed: context canceled, shutdown sequence: failed), starts a new sequence, and stops in the same task again; the emergency power-off armed in phase 1 did not fire either time.

   A frozen shim is an artificial way to get there, but an unresponsive shim is what containerd#13849 leaves behind in the wild, and it is what my node had. Is it intended that stopping a service has no deadline, so that a node in this state stays half-up forever instead of powering off? If that is worth a look, I am glad to help test a fix.

   freeze-kubelet-shim-repro.sh

   #!/usr/bin/env bash
   # Reproducer: a Talos shutdown that never ends when the kubelet's shim does not respond.
   #
   # Run it against a DISPOSABLE single-node Talos v1.13.9 cluster (a VM is fine). The node does not
   # power off afterwards; end it with a hard reset or the VM's power button, because the shim stays
   # frozen and the kubelet's own API is gone with it.
   #
   # Usage:
   #   freeze-kubelet-shim-repro.sh --node ADDR --talosconfig FILE --kubeconfig FILE [--watch-minutes N]
   #
   #   --node ADDR          the node's address; used as Talos node and endpoint
   #   --talosconfig FILE   talosctl configuration for that cluster
   #   --kubeconfig FILE    kubectl configuration for that cluster
   #   --watch-minutes N    how long to print the service table afterwards (default 15)
   #
   # Needs talosctl and kubectl. What it does:
   #   1. starts a privileged hostPID busybox pod so that a signal can be sent to a host process;
   #   2. finds the containerd-shim-runc-v2 that hosts the `kubelet` service in the system namespace;
   #   3. SIGSTOPs it, and shows that the kubelet child process is untouched;
   #   4. issues `talosctl shutdown --force`;
   #   5. prints `talosctl services` every 30 s, so you can watch the node stay up with apid
   #      answering and the kubelet in `Stopping`.
   set -uo pipefail
   
   NODE="" ; TALOSCONFIG_FILE="" ; KUBECONFIG_FILE="" ; WATCH_MINUTES=15
   usage() { sed -n '2,22p' "$0" | sed 's/^# \{0,1\}//'; exit "${1:-0}"; }
   while [ $# -gt 0 ]; do
       case "$1" in
           --node)          NODE="$2"; shift 2 ;;
           --talosconfig)   TALOSCONFIG_FILE="$2"; shift 2 ;;
           --kubeconfig)    KUBECONFIG_FILE="$2"; shift 2 ;;
           --watch-minutes) WATCH_MINUTES="$2"; shift 2 ;;
           -h|--help)       usage 0 ;;
           *) echo "unknown argument: $1" >&2; usage 64 ;;
       esac
   done
   [ -n "$NODE" ] && [ -n "$TALOSCONFIG_FILE" ] && [ -n "$KUBECONFIG_FILE" ] || { echo "--node, --talosconfig and --kubeconfig are required" >&2; usage 64; }
   [ -r "$TALOSCONFIG_FILE" ] || { echo "cannot read $TALOSCONFIG_FILE" >&2; exit 66; }
   [ -r "$KUBECONFIG_FILE" ]  || { echo "cannot read $KUBECONFIG_FILE" >&2; exit 66; }
   export TALOSCONFIG="$TALOSCONFIG_FILE" KUBECONFIG="$KUBECONFIG_FILE"
   
   NS=freeze-demo
   T="talosctl -n $NODE -e $NODE"
   ts() { date -u +%Y-%m-%dT%H:%M:%SZ; }
   
   # Guard: this leaves the node hung until a hard reset. Show what the two configurations point at
   # and require the node address to be typed back.
   echo "kubectl context : $(kubectl config current-context 2>&1)"
   echo "kubectl nodes   : $(kubectl get nodes -o name 2>&1 | tr '\n' ' ')"
   echo "talosctl target : $NODE -> $($T version --short 2>&1 | grep -m1 -E 'Tag|rror')"
   read -r -p "This node will NOT power off afterwards. Type the node address to continue: " CONFIRM
   [ "$CONFIRM" = "$NODE" ] || { echo "aborted"; exit 1; }
   
   echo "== $(ts) 1. privileged hostPID pod"
   kubectl create namespace "$NS" --dry-run=client -o yaml | kubectl apply -f -
   kubectl label namespace "$NS" pod-security.kubernetes.io/enforce=privileged pod-security.kubernetes.io/warn=privileged --overwrite
   kubectl -n "$NS" delete pod freezer --ignore-not-found --wait=true
   kubectl -n "$NS" apply -f - <<'EOF'
   apiVersion: v1
   kind: Pod
   metadata:
     name: freezer
   spec:
     hostPID: true
     restartPolicy: Never
     tolerations:
       - operator: Exists
     containers:
       - name: freezer
         image: docker.io/library/busybox:1.37
         command: ["sh", "-c", "sleep 100000"]
         securityContext:
           privileged: true
   EOF
   kubectl -n "$NS" wait --for=condition=Ready pod/freezer --timeout=120s
   
   echo "== $(ts) 2. the shim that hosts the kubelet"
   SHIM=$($T processes | grep 'containerd-shim-runc-v2 -namespace system -id kubelet ' | awk '{print $2}')
   [ -n "$SHIM" ] || { echo "no kubelet shim found"; exit 1; }
   echo "shim pid: $SHIM"
   kubectl -n "$NS" exec freezer -- sh -c "grep -E '^(Name|State|PPid)' /proc/$SHIM/status; echo children:; for c in \$(cat /proc/$SHIM/task/$SHIM/children); do grep -E '^(Name|State)' /proc/\$c/status; done"
   
   echo "== $(ts) 3. SIGSTOP"
   kubectl -n "$NS" exec freezer -- kill -STOP "$SHIM"
   kubectl -n "$NS" exec freezer -- grep '^State' "/proc/$SHIM/status"
   
   echo "== $(ts) 4. talosctl shutdown --force"
   $T shutdown --force --timeout 2m >/dev/null 2>&1 &
   
   echo "== $(ts) 5. watching for $WATCH_MINUTES minutes"
   END=$(( $(date +%s) + WATCH_MINUTES * 60 ))
   while [ "$(date +%s)" -lt "$END" ]; do
       sleep 30
       echo "-- $(ts)"
       if out=$(timeout 10 $T services 2>&1); then
           echo "$out" | grep -E 'SERVICE|apid|kubelet|cri '
       else
           echo "apid did not answer: $out"
       fi
   done
   echo "== $(ts) done; the node is still up if the last table shows kubelet Stopping. Hard-reset it."
6. smira commented on Sep 21, 2026

   

   smira

   on Sep 21, 2026

   Member

   More actions

   Talos 1.14+ moves shims out to a different cgroup. If you can reproduce the issue and finally submit a log instead of making LLM post here, it would be great.
7. daixtrose commented on Sep 26, 2026

   

   daixtrose

   on Sep 26, 2026

   Author

   More actions

   Closing this. I cannot reproduce the hang on Talos 1.14, and the artificial reproduction on 1.13.9 (a frozen shim) does not prove the original incident's cause. The node now runs 1.14.1. If the shutdown wedges again, I will capture a support bundle and the console log before anything else and reopen with those attached. Thanks for your time.
8. daixtrose

   closed this as completedon Sep 26, 2026