Skip to content

Commit 436a19d

Browse files
committed
test(e2e): make kuberture split-horizon probes survive Cilium ingress-IP race
The split-horizon-routing-proof step ran two external-dns probes as bare Pods and polled them by fixed name. When the Cilium host-scope IPAM + Gateway API restore race (cilium/cilium#46799) hands a probe the per-node reserved:ingress IP, the sandbox is rejected with "IP already in use" and the e2e leak-healer deletes the wedged pod, expecting a controller to reschedule it. A bare Pod has no controller, so the delete was terminal: the probe vanished and the step failed polling a NotFound pod. Run the probes as Jobs so the pod is re-created on a fresh IP after the healer's delete, assert each Job reaches Complete instead of polling a fixed Pod name, and read the logs by job-name label. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent 8bffa34 commit 436a19d

2 files changed

Lines changed: 98 additions & 63 deletions

File tree

‎hack/e2e-chainsaw/kuberture/chainsaw-test.yaml‎

Lines changed: 40 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
# annotation-shape checks and the split-horizon log assertions stay scripts.
55
#
66
# Everything is applied by chainsaw (the cluster-scoped Package, the cluster
7-
# RBAC, the probe Pods), so cleanup is automatic — no inlined teardown.
7+
# RBAC, the probe Jobs), so cleanup is automatic — no inlined teardown.
88
apiVersion: chainsaw.kyverno.io/v1alpha1
99
kind: Test
1010
metadata:
@@ -22,6 +22,12 @@ spec:
2222
namespace: cozy-kuberture
2323
selector: app.kubernetes.io/name=kuberture
2424
tail: 100
25+
# The split-horizon probe pods don't carry the kuberture addon label, so the
26+
# selector above misses them; capture their stdout on failure too.
27+
- podLogs:
28+
namespace: cozy-kuberture
29+
selector: app.kubernetes.io/name=kuberture-e2e-edns-probe
30+
tail: 100
2531
steps:
2632
- name: create-package
2733
try:
@@ -129,34 +135,44 @@ spec:
129135
try:
130136
- apply:
131137
file: probes.yaml
132-
- script:
138+
# Assert each single-shot probe Job reaches Complete rather than polling a
139+
# fixed Pod name. The Job re-creates its pod on a fresh IP if the Cilium
140+
# reserved:ingress race wedges one and the leak-healer deletes it, so the
141+
# proof survives that infra flake instead of failing on a vanished Pod.
142+
- assert:
143+
timeout: 4m
144+
resource:
145+
apiVersion: batch/v1
146+
kind: Job
147+
metadata:
148+
name: kuberture-e2e-edns-public
149+
namespace: cozy-kuberture
150+
status:
151+
(conditions[?type == 'Complete']):
152+
- status: "True"
153+
- assert:
133154
timeout: 4m
155+
resource:
156+
apiVersion: batch/v1
157+
kind: Job
158+
metadata:
159+
name: kuberture-e2e-edns-internal
160+
namespace: cozy-kuberture
161+
status:
162+
(conditions[?type == 'Complete']):
163+
- status: "True"
164+
- script:
165+
timeout: 1m
134166
content: |
135167
set -eu
136-
for probe in public internal; do
137-
deadline=$(( $(date +%s) + 180 ))
138-
until kubectl -n cozy-kuberture get pod "kuberture-e2e-edns-${probe}" \
139-
-o jsonpath='{.status.phase}' 2>/dev/null | grep -Eq '^(Succeeded|Failed)$'; do
140-
[ "$(date +%s)" -ge "$deadline" ] && break
141-
sleep 3
142-
done
143-
# Guard the capture: if the loop timed out before the Pod existed,
144-
# an unguarded read fails under `set -e` and skips the dump below.
145-
if ! phase=$(kubectl -n cozy-kuberture get pod "kuberture-e2e-edns-${probe}" -o jsonpath='{.status.phase}' 2>/dev/null); then
146-
phase="Missing"
147-
fi
148-
if [ "$phase" != "Succeeded" ]; then
149-
echo "==== probe ${probe} failed: phase=${phase} ===="
150-
kubectl -n cozy-kuberture describe pod "kuberture-e2e-edns-${probe}" || true
151-
kubectl -n cozy-kuberture logs "pod/kuberture-e2e-edns-${probe}" || true
152-
exit 1
153-
fi
154-
done
155-
156168
pub_host=kuberture-e2e-public.cozystack.test
157169
int_host=kuberture-e2e-internal.cozystack.test
158-
pub_logs=$(kubectl -n cozy-kuberture logs pod/kuberture-e2e-edns-public)
159-
int_logs=$(kubectl -n cozy-kuberture logs pod/kuberture-e2e-edns-internal)
170+
171+
# Read each Job's succeeded pod by its job-name label. The healer may
172+
# have deleted earlier wedged pods, but the Complete condition asserted
173+
# above guarantees one succeeded pod remains to carry the logs.
174+
pub_logs=$(kubectl -n cozy-kuberture logs --tail=-1 --max-log-requests=8 -l batch.kubernetes.io/job-name=kuberture-e2e-edns-public)
175+
int_logs=$(kubectl -n cozy-kuberture logs --tail=-1 --max-log-requests=8 -l batch.kubernetes.io/job-name=kuberture-e2e-edns-internal)
160176
161177
# The default-prefix probe sees only the public hostname; the internal-
162178
# prefix probe sees only the internal hostname.
Lines changed: 58 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,14 @@
1-
# external-dns probe SA/RBAC + two single-shot probe Pods, each with a distinct
1+
# external-dns probe SA/RBAC + two single-shot probe Jobs, each with a distinct
22
# --annotation-prefix (the upstream Split Horizon DNS pattern). external-dns
33
# 0.20 initialises its informer cache cluster-wide before applying --namespace,
44
# so the probe needs a ClusterRole, not a namespaced Role.
5+
#
6+
# The probes are Jobs, not bare Pods, on purpose. When the Cilium host-scope
7+
# IPAM + Gateway API restore-window race hands a probe the per-node
8+
# reserved:ingress IP (cilium/cilium#46799), the sandbox is rejected with
9+
# "IP already in use" and the e2e leak-healer deletes the wedged pod, expecting
10+
# a controller to reschedule it onto a free IP. A bare Pod has no controller, so
11+
# that delete was terminal and the probe never returned; the Job re-creates it.
512
apiVersion: v1
613
kind: ServiceAccount
714
metadata:
@@ -33,50 +40,62 @@ roleRef:
3340
kind: ClusterRole
3441
name: kuberture-e2e-edns-probe
3542
---
36-
apiVersion: v1
37-
kind: Pod
43+
apiVersion: batch/v1
44+
kind: Job
3845
metadata:
3946
name: kuberture-e2e-edns-public
4047
namespace: cozy-kuberture
4148
spec:
42-
serviceAccountName: kuberture-e2e-edns-probe
43-
restartPolicy: Never
44-
containers:
45-
- name: edns
46-
image: registry.k8s.io/external-dns/external-dns:v0.20.0
47-
args:
48-
- --source=service
49-
- --namespace=cozy-kuberture
50-
- --provider=inmemory
51-
- --inmemory-zone=cozystack.test
52-
- --registry=noop
53-
- --once
54-
- --interval=10s
55-
- --annotation-prefix=external-dns.alpha.kubernetes.io/
56-
- --domain-filter=cozystack.test
57-
- --log-level=info
58-
- --log-format=text
49+
backoffLimit: 6
50+
template:
51+
metadata:
52+
labels:
53+
app.kubernetes.io/name: kuberture-e2e-edns-probe
54+
spec:
55+
serviceAccountName: kuberture-e2e-edns-probe
56+
restartPolicy: Never
57+
containers:
58+
- name: edns
59+
image: registry.k8s.io/external-dns/external-dns:v0.20.0
60+
args:
61+
- --source=service
62+
- --namespace=cozy-kuberture
63+
- --provider=inmemory
64+
- --inmemory-zone=cozystack.test
65+
- --registry=noop
66+
- --once
67+
- --interval=10s
68+
- --annotation-prefix=external-dns.alpha.kubernetes.io/
69+
- --domain-filter=cozystack.test
70+
- --log-level=info
71+
- --log-format=text
5972
---
60-
apiVersion: v1
61-
kind: Pod
73+
apiVersion: batch/v1
74+
kind: Job
6275
metadata:
6376
name: kuberture-e2e-edns-internal
6477
namespace: cozy-kuberture
6578
spec:
66-
serviceAccountName: kuberture-e2e-edns-probe
67-
restartPolicy: Never
68-
containers:
69-
- name: edns
70-
image: registry.k8s.io/external-dns/external-dns:v0.20.0
71-
args:
72-
- --source=service
73-
- --namespace=cozy-kuberture
74-
- --provider=inmemory
75-
- --inmemory-zone=cozystack.test
76-
- --registry=noop
77-
- --once
78-
- --interval=10s
79-
- --annotation-prefix=kuberture-e2e-internal-dns.io/
80-
- --domain-filter=cozystack.test
81-
- --log-level=info
82-
- --log-format=text
79+
backoffLimit: 6
80+
template:
81+
metadata:
82+
labels:
83+
app.kubernetes.io/name: kuberture-e2e-edns-probe
84+
spec:
85+
serviceAccountName: kuberture-e2e-edns-probe
86+
restartPolicy: Never
87+
containers:
88+
- name: edns
89+
image: registry.k8s.io/external-dns/external-dns:v0.20.0
90+
args:
91+
- --source=service
92+
- --namespace=cozy-kuberture
93+
- --provider=inmemory
94+
- --inmemory-zone=cozystack.test
95+
- --registry=noop
96+
- --once
97+
- --interval=10s
98+
- --annotation-prefix=kuberture-e2e-internal-dns.io/
99+
- --domain-filter=cozystack.test
100+
- --log-level=info
101+
- --log-format=text

0 commit comments

Comments
 (0)