Skip to content

Commit b825c31

Browse files
test(e2e): self-heal Cilium orphaned-endpoint leak across install and apps
Run the Cilium endpoint-leak watchdog as an in-cluster Job instead of a background process started in the install bats setup_file. bats *_file hooks fire per file, so the old process was killed at the install file's teardown_file — before any app test ran — and the leak recurred unhealed in the app phase (e.g. the tenant-Kubernetes CVMI importer). The Job is bound to the cluster, not to one bats file, so it covers install and the whole app suite. The heal logic stays in hack/e2e-cilium-endpoint-leak-healer.sh (single source of truth) and is shipped to the pod via a ConfigMap built from it at apply time. The pod uses hostNetwork so it needs no Cilium endpoint and cannot be wedged by the very leak it repairs, with broad tolerations so it schedules before nodes are Ready. RBAC is scoped to read events/pods and exec into the cilium-agent. Image reuses the in-tree digest-pinned alpine/k8s already shipped by the installer. Interim CI band-aid for cilium/cilium#38313; remove with a fixed Cilium. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> Signed-off-by: Myasnikov Daniil <[email protected]>
1 parent e69d268 commit b825c31

3 files changed

Lines changed: 255 additions & 0 deletions

File tree

Lines changed: 120 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,120 @@
1+
#!/bin/sh
2+
# Interim CI self-heal for the Cilium in-memory endpoint leak.
3+
#
4+
# This is the heal loop; it runs as an in-cluster Job (see
5+
# hack/e2e-cilium-leak-healer.yaml), deployed during e2e install and left
6+
# running for the whole cluster lifetime. That is deliberate: an earlier
7+
# version ran this as a background process started in the install bats
8+
# setup_file and killed in teardown_file, but bats *_file hooks fire per file,
9+
# so it died when the install file finished — before any app test ran — and the
10+
# leak recurred in the app phase (e.g. the tenant-Kubernetes CVMI importer)
11+
# unhealed. An in-cluster Job is bound to the cluster, not to one bats file, so
12+
# it covers install AND every app test.
13+
#
14+
# Symptom: on a churn-heavy run a pod sticks in ContainerCreating/Init because
15+
# the cilium-agent rejects its sandbox with
16+
# [PUT /endpoint/{id}][400] putEndpointIdInvalid "IP ipv4:<X> is already in use"
17+
# repeated until the pod is deleted or the agent is restarted. The IP has no
18+
# live owner: IPAM and the ipcache released it, but the agent's in-memory
19+
# endpointManager still holds a stale Endpoint for it (the previous pod's CNI
20+
# DEL was skipped or raced, so unexpose()->removeReferencesLocked() never ran).
21+
# Neither the operator's CiliumEndpoint-CRD GC nor the agent's veth-based
22+
# endpoint GC reaps it, so it wedges the new pod and fails the run. Tracked
23+
# upstream at cilium/cilium#38313 (closed stale); no released or pre-release
24+
# Cilium fixes the leak and no flag disables it.
25+
#
26+
# This watchdog applies the least-disruptive known remedy: instead of
27+
# restarting the agent (which drops the whole node's in-memory registry), it
28+
# evicts only the orphaned endpoint via
29+
# cilium-dbg endpoint disconnect ipv4:<X>
30+
# which maps to DELETE /endpoint/{id} -> RemoveEndpoint -> unexpose ->
31+
# removeReferencesLocked, clearing the stale IP-keyed entry. The wedged pod's
32+
# next CNI ADD retry then succeeds. Blast radius is one dead endpoint.
33+
#
34+
# It is READ-ONLY until it has positively confirmed an orphan: an endpoint in
35+
# the owning node's registry holds the disputed IP, that endpoint does not back
36+
# a live Running pod with that IP, and a different pod is currently wedged
37+
# requesting the same IP. If anything is ambiguous it logs and does nothing —
38+
# it never disconnects a live endpoint.
39+
#
40+
# REMOVE this script, hack/e2e-cilium-leak-healer.yaml, and the setup_file/
41+
# teardown_file hooks in hack/e2e-install-cozystack.bats once a fixed Cilium
42+
# ships. This is a CI band-aid, not a product fix.
43+
set -u
44+
45+
CILIUM_NS="${CILIUM_NS:-cozy-cilium}"
46+
POLL_INTERVAL="${CILIUM_HEALER_POLL_SEC:-15}"
47+
48+
log() { echo "[cilium-leak-healer $(date -u +%H:%M:%S)] $*"; }
49+
50+
# cilium-agent pod running on a given node (DaemonSet selector k8s-app=cilium).
51+
agent_on_node() {
52+
kubectl get pod -n "$CILIUM_NS" -l k8s-app=cilium \
53+
--field-selector "spec.nodeName=$1" \
54+
-o jsonpath='{.items[0].metadata.name}' 2>/dev/null
55+
}
56+
57+
log "started (ns=$CILIUM_NS poll=${POLL_INTERVAL}s, in-cluster, runs for the cluster lifetime)"
58+
59+
# Endless loop: the pod is torn down with the ephemeral e2e cluster. Before
60+
# Cilium is serving (early install) the kubectl/exec calls simply fail and the
61+
# loop idles; once the CNI is up it begins healing. Host networking (see the
62+
# manifest) keeps this pod itself immune to the very leak it repairs.
63+
while true; do
64+
# Collect every "is already in use" rejection still attached to a pod sandbox.
65+
# The field-selector narrows to the relevant event reason; grep pins the exact
66+
# agent signature so unrelated sandbox failures are ignored.
67+
events=$(kubectl get events -A --field-selector reason=FailedCreatePodSandBox \
68+
-o jsonpath='{range .items[*]}{.involvedObject.namespace}{"|"}{.involvedObject.name}{"|"}{.message}{"\n"}{end}' 2>/dev/null \
69+
| grep -i "is already in use" | sort -u)
70+
71+
[ -n "$events" ] && while IFS='|' read -r ns pod msg; do
72+
[ -n "$ns" ] && [ -n "$pod" ] || continue
73+
ip=$(printf '%s' "$msg" | grep -oE 'ipv4:[0-9]{1,3}(\.[0-9]{1,3}){3}' | head -1 | cut -d: -f2)
74+
[ -n "$ip" ] || continue
75+
76+
# Only act on a pod that is still wedged (re-checked here = freshness gate).
77+
phase=$(kubectl get pod -n "$ns" "$pod" -o jsonpath='{.status.phase}' 2>/dev/null)
78+
case "$phase" in Running|Succeeded|"") continue ;; esac
79+
80+
node=$(kubectl get pod -n "$ns" "$pod" -o jsonpath='{.spec.nodeName}' 2>/dev/null)
81+
[ -n "$node" ] || continue
82+
agent=$(agent_on_node "$node")
83+
[ -n "$agent" ] || { log "no cilium-agent on node=$node for $ns/$pod ip=$ip"; continue; }
84+
85+
# Does an endpoint in this agent's registry hold the disputed IP at all?
86+
# (The wedged pod itself was never registered — CreateEndpoint rejected it
87+
# before exposing — so any endpoint holding the IP is the stale one.)
88+
ep_json=$(kubectl exec -n "$CILIUM_NS" "$agent" -c cilium-agent -- \
89+
cilium-dbg endpoint get "ipv4:$ip" -o json 2>/dev/null)
90+
[ -n "$ep_json" ] && [ "$ep_json" != "[]" ] || \
91+
{ log "no endpoint holds ip=$ip on node=$node (already cleared?) for $ns/$pod"; continue; }
92+
93+
epid=$(printf '%s' "$ep_json" | jq -r '.[0].id // empty' 2>/dev/null)
94+
epns=$(printf '%s' "$ep_json" | jq -r '.[0].status."external-identifiers"."k8s-namespace" // empty' 2>/dev/null)
95+
eppod=$(printf '%s' "$ep_json" | jq -r '.[0].status."external-identifiers"."k8s-pod-name" // empty' 2>/dev/null)
96+
97+
# If that endpoint claims a pod that is alive on the cluster with this IP,
98+
# it is a LIVE endpoint -> a genuine duplicate-IP, NOT the leak. Refuse and
99+
# shout so the real problem is visible instead of silently broken.
100+
if [ -n "$epns" ] && [ -n "$eppod" ]; then
101+
owner_ip=$(kubectl get pod -n "$epns" "$eppod" -o jsonpath='{.status.podIP}' 2>/dev/null)
102+
owner_phase=$(kubectl get pod -n "$epns" "$eppod" -o jsonpath='{.status.phase}' 2>/dev/null)
103+
if [ "$owner_ip" = "$ip" ] && [ "$owner_phase" = "Running" ]; then
104+
log "REFUSE ip=$ip on node=$node: held by LIVE pod $epns/$eppod (genuine duplicate IP, not the leak) — investigate"
105+
continue
106+
fi
107+
fi
108+
109+
# Confirmed orphan: endpoint $epid holds $ip, backs no live pod, and pod
110+
# $ns/$pod is wedged requesting the same IP. Evict only this endpoint.
111+
log "HEAL node=$node agent=$agent ep=$epid ip=$ip wedged=$ns/$pod -> endpoint disconnect ipv4:$ip"
112+
out=$(kubectl exec -n "$CILIUM_NS" "$agent" -c cilium-agent -- \
113+
cilium-dbg endpoint disconnect "ipv4:$ip" 2>&1)
114+
log " result: ${out:-<none>}"
115+
done <<EOF
116+
$events
117+
EOF
118+
119+
sleep "$POLL_INTERVAL"
120+
done

‎hack/e2e-cilium-leak-healer.yaml‎

Lines changed: 99 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,99 @@
1+
# Interim in-cluster self-heal for the Cilium endpoint leak (cilium/cilium#38313
2+
# class). Applied during e2e install (hack/e2e-install-cozystack.bats) and left
3+
# running for the whole cluster lifetime, so it covers install AND every app
4+
# test. The heal logic lives in hack/e2e-cilium-endpoint-leak-healer.sh and is
5+
# shipped to the pod via a ConfigMap built from that file at apply time (single
6+
# source of truth). REMOVE this file, that script, and the bats wiring once a
7+
# fixed Cilium ships. CI band-aid, not a product fix.
8+
apiVersion: v1
9+
kind: ServiceAccount
10+
metadata:
11+
name: cilium-leak-healer
12+
namespace: kube-system
13+
labels:
14+
app.kubernetes.io/name: cilium-leak-healer
15+
---
16+
apiVersion: rbac.authorization.k8s.io/v1
17+
kind: ClusterRole
18+
metadata:
19+
name: cilium-leak-healer
20+
labels:
21+
app.kubernetes.io/name: cilium-leak-healer
22+
rules:
23+
# find FailedCreatePodSandBox "IP already in use" rejections, cluster-wide
24+
- apiGroups: [""]
25+
resources: ["events"]
26+
verbs: ["get", "list", "watch"]
27+
# inspect the wedged pod and the endpoint's claimed owner
28+
- apiGroups: [""]
29+
resources: ["pods"]
30+
verbs: ["get", "list"]
31+
# run `cilium-dbg endpoint ...` inside the owning node's cilium-agent
32+
- apiGroups: [""]
33+
resources: ["pods/exec"]
34+
verbs: ["create"]
35+
---
36+
apiVersion: rbac.authorization.k8s.io/v1
37+
kind: ClusterRoleBinding
38+
metadata:
39+
name: cilium-leak-healer
40+
labels:
41+
app.kubernetes.io/name: cilium-leak-healer
42+
roleRef:
43+
apiGroup: rbac.authorization.k8s.io
44+
kind: ClusterRole
45+
name: cilium-leak-healer
46+
subjects:
47+
- kind: ServiceAccount
48+
name: cilium-leak-healer
49+
namespace: kube-system
50+
---
51+
apiVersion: batch/v1
52+
kind: Job
53+
metadata:
54+
name: cilium-leak-healer
55+
namespace: kube-system
56+
labels:
57+
app.kubernetes.io/name: cilium-leak-healer
58+
spec:
59+
# The pod runs an endless heal loop and is torn down with the ephemeral e2e
60+
# cluster, so it never completes on its own. backoffLimit only matters if the
61+
# watchdog itself ever crashes — keep restarting it.
62+
backoffLimit: 100
63+
template:
64+
metadata:
65+
labels:
66+
app.kubernetes.io/name: cilium-leak-healer
67+
spec:
68+
serviceAccountName: cilium-leak-healer
69+
restartPolicy: OnFailure
70+
# Immune to the very leak it heals: host networking needs no Cilium
71+
# endpoint, so this pod can never be wedged by the leak and can reach the
72+
# API server before pod networking is fully up. Broad tolerations let it
73+
# schedule before nodes are Ready and stay put through later taints.
74+
hostNetwork: true
75+
dnsPolicy: ClusterFirstWithHostNet
76+
tolerations:
77+
- operator: Exists
78+
containers:
79+
- name: healer
80+
# Reuse the in-tree, digest-pinned kubectl+jq image already shipped by
81+
# the installer (packages/core/installer/templates/cozy-system-labels.yaml).
82+
image: docker.io/alpine/k8s:1.33.4@sha256:b0523f0a244ddc4c8e055aa335c040d3d78b3ead5528f4544395f7f9f69c7b68
83+
command: ["/bin/sh", "/opt/healer/heal.sh"]
84+
env:
85+
- name: CILIUM_NS
86+
value: cozy-cilium
87+
volumeMounts:
88+
- name: script
89+
mountPath: /opt/healer
90+
resources:
91+
requests:
92+
cpu: 10m
93+
memory: 32Mi
94+
limits:
95+
memory: 128Mi
96+
volumes:
97+
- name: script
98+
configMap:
99+
name: cilium-leak-healer

‎hack/e2e-install-cozystack.bats‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,41 @@
11
#!/usr/bin/env bats
22

3+
setup_file() {
4+
# Interim mitigation for the Cilium in-memory endpoint leak
5+
# (cilium/cilium#38313 class): a deleted pod's endpoint is orphaned in the
6+
# agent's registry, IPAM re-hands its IP to a new pod, and the agent then
7+
# rejects the new sandbox with "IP <X> is already in use" until an agent
8+
# restart — wedging pods across install AND the app suite and failing
9+
# unrelated PRs. The watchdog surgically evicts only the orphaned endpoint.
10+
#
11+
# It runs as an in-cluster Job (not a bats-scoped background process, which
12+
# died at this file's teardown_file — before any app test ran — and left the
13+
# app-phase leak unhealed). Bound to the cluster, it covers every later test
14+
# step too. The heal logic stays in hack/e2e-cilium-endpoint-leak-healer.sh
15+
# (single source of truth) and is shipped to the pod via a ConfigMap built
16+
# from it here. Remove together with that script and
17+
# hack/e2e-cilium-leak-healer.yaml once a fixed Cilium ships.
18+
kubectl create configmap cilium-leak-healer -n kube-system \
19+
--from-file=heal.sh=hack/e2e-cilium-endpoint-leak-healer.sh \
20+
--dry-run=client -o yaml | kubectl apply -f - || true
21+
kubectl apply -f hack/e2e-cilium-leak-healer.yaml || true
22+
}
23+
24+
teardown_file() {
25+
# Surface any heal/refuse activity from the install phase into the test trace;
26+
# the run is otherwise green and the heal would be invisible. App-phase
27+
# activity lands in the same pod log (kubectl logs job/cilium-leak-healer) and
28+
# the end-of-job cozyreport. Do NOT delete the Job here — later test steps
29+
# still need the watchdog. REFUSE lines flag a genuine duplicate-IP (a real
30+
# bug the watchdog deliberately did not touch).
31+
if kubectl logs -n kube-system job/cilium-leak-healer --tail=-1 2>/dev/null \
32+
| grep -qE "HEAL |REFUSE "; then
33+
echo "# cilium endpoint-leak watchdog activity (install phase):" >&3
34+
kubectl logs -n kube-system job/cilium-leak-healer --tail=-1 2>/dev/null \
35+
| grep -E "HEAL |REFUSE " >&3 || true
36+
fi
37+
}
38+
339
@test "Required installer chart exists" {
440
if [ ! -f packages/core/installer/Chart.yaml ]; then
541
echo "Missing: packages/core/installer/Chart.yaml" >&2

0 commit comments

Comments
 (0)