Skip to content

Commit 9b695f2

Browse files
lexfreilexfrei
committed
test(e2e): capture host-netns address attribution and a healthy-pod reference
The collector characterises the host->local-pod route for an affected pod through ovn0 alone, so when a pod IP turns out to be a local address of the host netns instead, nothing records which interface actually holds it. The host netns capture now also takes `ip -o addr show` across every interface and `ip route show table local`, so the owner can be named on the next occurrence rather than only ruled out on ovn0. The collector also inspects NotReady pods only, so a run has no same-window baseline for what a normal route and conntrack state look like on the same node. The per-pod capture becomes a reusable function and runs a second time against one Ready, Running, non-hostNetwork pod on that node, into the same file, so the comparison is in front of whoever opens it. hostNetwork is excluded because such a pod's podIP IS the node's own address and would show the same local-route fingerprint as the fault for an entirely ordinary reason. That baseline depends on the node and nothing else -- every pod the run is already investigating is excluded by name, not just the one whose turn it is, since a pod that recovered between the two reads reports Ready here and would otherwise be compared against its own fault -- so it is captured once per node and folded into each affected pod's file from a cache. Several affected pods on one node is what a wedged host->local-pod datapath looks like rather than an edge case, and capturing the same healthy pod once per affected pod would spend the caller's envelope re-asking a question already answered -- the cost the node-global dedup and the lookup memo beside it exist to avoid. Both reads this adds hold the discipline the rest of the file already does. The node-scoped pod list is bounded and reports its own status, and the artifact separates a node with no eligible pod from a list that never answered: one line for both would make an unchecked baseline read as an established absence. The new row filter carries the field count its jsonpath emits, so a list cut mid-record cannot hand over a fragment as the pod every reading of the affected one is compared against. Assisted-By: Claude <[email protected]> Signed-off-by: Aleksei Sviridkin <[email protected]>
1 parent b9846fe commit 9b695f2

2 files changed

Lines changed: 555 additions & 83 deletions

File tree

‎hack/capture-dataplane.bats‎

Lines changed: 299 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,8 @@
1515
# succeeded, unknown when none ran or none could.
1616
# - pod_filter_affected -- retain scheduled NotReady pods even without IPs,
1717
# and only from whole rows.
18+
# - pod_first_ready -- pick the in-run healthy-pod reference for a node,
19+
# also only from whole rows.
1820
#
1921
# The tests come in two shapes, and neither needs a cluster.
2022
#
@@ -76,6 +78,89 @@ E2E_CAPTURE_DATAPLANE_LIB=1
7678
[ -z "$(printf '%s\n' "$rows" | pod_filter_affected)" ]
7779
}
7880

81+
@test "pod_first_ready picks the first Ready, Running pod and skips the rest" {
82+
rows="$(printf '%s\n' \
83+
'tenant|not-ready|192.0.2.90|False|Running|' \
84+
'tenant|pending|192.0.2.91|True|Pending|' \
85+
'tenant|healthy-1|192.0.2.92|True|Running|' \
86+
'tenant|healthy-2|192.0.2.93|True|Running|')"
87+
88+
out="$(printf '%s\n' "$rows" | pod_first_ready)"
89+
90+
[ "$out" = "tenant|healthy-1|192.0.2.92" ]
91+
}
92+
93+
@test "pod_first_ready emits nothing when no pod on the node is Ready and Running" {
94+
rows="$(printf '%s\n' \
95+
'tenant|not-ready|192.0.2.94|False|Running|' \
96+
'tenant|pending|192.0.2.95|True|Pending|')"
97+
98+
[ -z "$(printf '%s\n' "$rows" | pod_first_ready)" ]
99+
}
100+
101+
@test "pod_first_ready emits nothing on empty input" {
102+
[ -z "$(printf '' | pod_first_ready)" ]
103+
}
104+
105+
@test "pod_first_ready skips a Ready pod that has no podIP yet" {
106+
# The filter stops at the row it takes, so an addressless pod does not just
107+
# produce a thin baseline -- it consumes the only pick and the pod behind it
108+
# is never considered. Every IP-specific capture the comparison exists for
109+
# would then be skipped for want of an address.
110+
rows="$(printf '%s\n' \
111+
'tenant|no-ip-yet||True|Running|' \
112+
'tenant|healthy|192.0.2.99|True|Running|')"
113+
114+
out="$(printf '%s\n' "$rows" | pod_first_ready)"
115+
116+
[ "$out" = "tenant|healthy|192.0.2.99" ]
117+
}
118+
119+
@test "pod_first_ready drops a pod row cut off mid-record" {
120+
# Fourth record filter over a bounded list, same cut as the other three, and
121+
# the fragment has to come first because this one stops at the row it
122+
# accepts. What it would otherwise select is the baseline every reading of
123+
# the affected pod gets compared against -- a pod that does not exist, on an
124+
# address that was never whole.
125+
# The cut has to land where the fragment still satisfies every value test,
126+
# or the row is rejected for an unrelated reason and the count is never
127+
# consulted: here the stream ends right after `Running`, one separator short
128+
# of the hostNetwork column, so an unguarded filter reads a whole record and
129+
# takes it.
130+
rows="$(printf '%s\n' \
131+
'tenant|cut-off|192.0.2.9|True|Running' \
132+
'tenant|healthy|192.0.2.98|True|Running|')"
133+
134+
out="$(printf '%s\n' "$rows" | pod_first_ready)"
135+
136+
[ "$out" = "tenant|healthy|192.0.2.98" ]
137+
}
138+
139+
@test "pod_first_ready skips a hostNetwork pod even when it is Ready, Running, and first" {
140+
# cilium-agent / kube-ovn-cni / ovs-ovn run hostNetwork on every node, and a
141+
# hostNetwork pod's podIP IS the node's own address -- picking one as the
142+
# baseline would resolve to a local route for an entirely ordinary reason,
143+
# which is the same fingerprint the address-attribution capture exists to
144+
# recognise, so the baseline would defeat itself.
145+
rows="$(printf '%s\n' \
146+
'cozy-cilium|cilium-agent-abcde|192.0.2.10|True|Running|true' \
147+
'tenant|workload|192.0.2.96|True|Running|')"
148+
149+
out="$(printf '%s\n' "$rows" | pod_first_ready)"
150+
151+
[ "$out" = "tenant|workload|192.0.2.96" ]
152+
}
153+
154+
@test "pod_first_ready treats a blank hostNetwork column as NOT hostNetwork" {
155+
# The API omits spec.hostNetwork (omitempty) when it is false/unset, so
156+
# kubectl's jsonpath emits an empty string, not the literal "false".
157+
rows="$(printf '%s\n' 'tenant|workload|192.0.2.97|True|Running|')"
158+
159+
out="$(printf '%s\n' "$rows" | pod_first_ready)"
160+
161+
[ "$out" = "tenant|workload|192.0.2.97" ]
162+
}
163+
79164
@test "runtime checks LoadBalancers when there are no affected pods" {
80165
tmp=$(mktemp -d)
81166
trap 'rm -rf "$tmp"' EXIT
@@ -765,6 +850,220 @@ STUB
765850
rm -rf "$d"
766851
}
767852

853+
@test "a healthy pod on the node is captured beside the affected one" {
854+
# The baseline is the whole point of the reference leg, and it is only worth
855+
# anything if it lands in the SAME file as the affected pod: a reader
856+
# comparing a suspicious route against a normal one should not have to go
857+
# find the normal one. Nothing else in this suite reaches that leg.
858+
d=$(mktemp -d)
859+
mkdir -p "$d/bin"
860+
cat >"$d/bin/kubectl" <<'STUB'
861+
#!/bin/sh
862+
# The node-scoped list feeding the baseline. Checked before the bare `pods`
863+
# arm below, which would otherwise swallow it and answer with the wedged pod.
864+
case "$*" in
865+
*'get pods -A --field-selector'*)
866+
echo 'tenant-test|wedged|10.0.0.1|False|Running|'
867+
echo 'cozy-cilium|cilium-xyz|10.0.0.5|True|Running|true'
868+
echo 'tenant|healthy|10.0.0.9|True|Running|'
869+
exit 0 ;;
870+
esac
871+
for a in "$@"; do
872+
case $a in
873+
pods) echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running'; exit 0 ;;
874+
esac
875+
done
876+
exit 0
877+
STUB
878+
chmod +x "$d/bin/kubectl"
879+
PATH="$d/bin:$PATH" timeout 90 "$SCRIPT" "$d/out" >"$d/log" 2>&1 || true
880+
[ -f "$d/out/pod-tenant-test-wedged.txt" ]
881+
# Both pods, in one file, each stamped with the role it plays.
882+
grep -q 'POD tenant-test/wedged' "$d/out/pod-tenant-test-wedged.txt"
883+
grep -q 'REFERENCE (Ready)' "$d/out/pod-tenant-test-wedged.txt"
884+
grep -q 'POD tenant/healthy' "$d/out/pod-tenant-test-wedged.txt"
885+
# The hostNetwork pod sorts first in the stub and must not have been taken:
886+
# its podIP is the node's own address, which is the fingerprint the baseline
887+
# exists to rule out.
888+
if grep -q 'POD cozy-cilium/cilium-xyz' "$d/out/pod-tenant-test-wedged.txt"; then
889+
echo "a hostNetwork pod was taken as the baseline:"
890+
cat "$d/out/pod-tenant-test-wedged.txt"
891+
exit 1
892+
fi
893+
rm -rf "$d"
894+
}
895+
896+
@test "two affected pods on one node share a single baseline capture" {
897+
# The baseline depends on the node and nothing else, and several affected
898+
# pods on one node is what a wedged host->local-pod datapath actually looks
899+
# like -- so capturing it per pod pays twice for one answer, against a
900+
# backstop the pod leg already strains. Every other stub in this suite
901+
# returns exactly one affected pod, so nothing else here reaches the repeat.
902+
d=$(mktemp -d)
903+
mkdir -p "$d/bin"
904+
cat >"$d/bin/kubectl" <<'STUB'
905+
#!/bin/sh
906+
case "$*" in
907+
*'get pods -A --field-selector'*)
908+
echo ref-list >>"$STUB_CALLS"
909+
echo 'tenant|healthy|10.0.0.9|True|Running|'
910+
exit 0 ;;
911+
esac
912+
for a in "$@"; do
913+
case $a in
914+
pods)
915+
echo 'tenant-test|wedged-a|10.0.0.1|node-a|False|Running'
916+
echo 'tenant-test|wedged-b|10.0.0.2|node-a|False|Running'
917+
exit 0 ;;
918+
esac
919+
done
920+
exit 0
921+
STUB
922+
chmod +x "$d/bin/kubectl"
923+
STUB_CALLS="$d/calls" PATH="$d/bin:$PATH" timeout 120 "$SCRIPT" "$d/out" >"$d/log" 2>&1 || true
924+
# Both pods were walked, and both carry the baseline: the saving must not
925+
# come from one of them silently losing it.
926+
grep -q 'REFERENCE (Ready)' "$d/out/pod-tenant-test-wedged-a.txt"
927+
grep -q 'REFERENCE (Ready)' "$d/out/pod-tenant-test-wedged-b.txt"
928+
grep -q 'POD tenant/healthy' "$d/out/pod-tenant-test-wedged-b.txt"
929+
# And the node was asked once.
930+
n=$(grep -c ref-list "$d/calls")
931+
if [ "$n" -ne 1 ]; then
932+
echo "the node-scoped pod list was issued $n times for one node:"
933+
cat "$d/calls"
934+
exit 1
935+
fi
936+
rm -rf "$d"
937+
}
938+
939+
@test "a refused node pod list is re-asked for the next pod rather than cached" {
940+
# The baseline cache must not make one transient permanent for the node. This
941+
# is the policy pod_memo_put already states for the lookup memo and its own
942+
# test pins: an instant refusal costs nothing to ask again, while a cutoff
943+
# costs another full bound and is worth keeping. The stub refuses once and
944+
# answers after that, so a cached refusal shows up as a second list never
945+
# being issued -- and as the second pod inheriting an absence nobody rechecked.
946+
d=$(mktemp -d)
947+
mkdir -p "$d/bin"
948+
cat >"$d/bin/kubectl" <<'STUB'
949+
#!/bin/sh
950+
case "$*" in
951+
*'get pods -A --field-selector'*)
952+
n=$(cat "$CNT" 2>/dev/null || echo 0); n=$((n+1)); echo "$n" >"$CNT"
953+
if [ "$n" -eq 1 ]; then
954+
echo 'Error from server (Forbidden): pods is forbidden' >&2
955+
exit 1
956+
fi
957+
echo 'tenant|healthy|10.0.0.9|True|Running|'
958+
exit 0 ;;
959+
esac
960+
for a in "$@"; do
961+
case $a in
962+
pods)
963+
echo 'tenant-test|wedged-a|10.0.0.1|node-a|False|Running'
964+
echo 'tenant-test|wedged-b|10.0.0.2|node-a|False|Running'
965+
exit 0 ;;
966+
esac
967+
done
968+
exit 0
969+
STUB
970+
chmod +x "$d/bin/kubectl"
971+
CNT="$d/cnt" PATH="$d/bin:$PATH" timeout 120 "$SCRIPT" "$d/out" >"$d/log" 2>&1 || true
972+
n=$(cat "$d/cnt" 2>/dev/null || echo 0)
973+
if [ "$n" -lt 2 ]; then
974+
echo "a refused list was cached instead of re-asked (node lists issued: $n)"
975+
exit 1
976+
fi
977+
# The first pod records the refusal, and the second gets the answer the retry
978+
# produced rather than inheriting the first pod's dead end.
979+
grep -q 'the pod list did not answer' "$d/out/pod-tenant-test-wedged-a.txt"
980+
grep -q 'POD tenant/healthy' "$d/out/pod-tenant-test-wedged-b.txt"
981+
rm -rf "$d"
982+
}
983+
984+
@test "an affected pod is never offered as a healthy baseline" {
985+
# Excluding only the pod whose turn it is protects that one pod. The baseline
986+
# is shared across the node, so it has to exclude every pod this run is
987+
# already investigating: one that recovered between the cluster-wide list and
988+
# the node-scoped read still reports Ready/Running here, and offered as the
989+
# comparison it would be measured against its own fault -- into its own file
990+
# in the worst case. The two then agree by construction, which is exactly how
991+
# an artifact reads as "nothing anomalous here".
992+
d=$(mktemp -d)
993+
mkdir -p "$d/bin"
994+
cat >"$d/bin/kubectl" <<'STUB'
995+
#!/bin/sh
996+
case "$*" in
997+
*'get pods -A --field-selector'*)
998+
# wedged-b recovered between the two reads and now looks eligible.
999+
echo 'tenant-test|wedged-a|10.0.0.1|False|Running|'
1000+
echo 'tenant-test|wedged-b|10.0.0.2|True|Running|'
1001+
exit 0 ;;
1002+
esac
1003+
for a in "$@"; do
1004+
case $a in
1005+
pods)
1006+
echo 'tenant-test|wedged-a|10.0.0.1|node-a|False|Running'
1007+
echo 'tenant-test|wedged-b|10.0.0.2|node-a|False|Running'
1008+
exit 0 ;;
1009+
esac
1010+
done
1011+
exit 0
1012+
STUB
1013+
chmod +x "$d/bin/kubectl"
1014+
PATH="$d/bin:$PATH" timeout 120 "$SCRIPT" "$d/out" >"$d/log" 2>&1 || true
1015+
# Positive control: both pods were walked, so the absence below is a decision
1016+
# and not an empty run.
1017+
[ -f "$d/out/pod-tenant-test-wedged-a.txt" ]
1018+
[ -f "$d/out/pod-tenant-test-wedged-b.txt" ]
1019+
for f in "$d"/out/pod-tenant-test-wedged-a.txt "$d"/out/pod-tenant-test-wedged-b.txt; do
1020+
if grep 'REFERENCE' "$f" | grep -q 'tenant-test/wedged'; then
1021+
echo "a pod under investigation was used as the healthy baseline in $(basename "$f"):"
1022+
grep '^# POD' "$f"
1023+
exit 1
1024+
fi
1025+
done
1026+
# With every affected pod excluded this node has no eligible baseline, and
1027+
# that is what the artifact must say.
1028+
grep -q 'none found -- no baseline available' "$d/out/pod-tenant-test-wedged-b.txt"
1029+
rm -rf "$d"
1030+
}
1031+
1032+
@test "an unanswered node pod list is not recorded as a node without a baseline" {
1033+
# Same rule the rest of this collector follows, at the newest read: a list
1034+
# that never answered is not a node that has no healthy pod, and the artifact
1035+
# has to say which of the two happened rather than printing the same line for
1036+
# both.
1037+
d=$(mktemp -d)
1038+
mkdir -p "$d/bin"
1039+
cat >"$d/bin/kubectl" <<'STUB'
1040+
#!/bin/sh
1041+
case "$*" in
1042+
*'get pods -A --field-selector'*)
1043+
echo 'Error from server (Forbidden): pods is forbidden' >&2
1044+
exit 1 ;;
1045+
esac
1046+
for a in "$@"; do
1047+
case $a in
1048+
pods) echo 'tenant-test|wedged|10.0.0.1|node-a|False|Running'; exit 0 ;;
1049+
esac
1050+
done
1051+
exit 0
1052+
STUB
1053+
chmod +x "$d/bin/kubectl"
1054+
PATH="$d/bin:$PATH" timeout 90 "$SCRIPT" "$d/out" >"$d/log" 2>&1 || true
1055+
[ -f "$d/out/pod-tenant-test-wedged.txt" ]
1056+
if grep -q 'none found' "$d/out/pod-tenant-test-wedged.txt"; then
1057+
echo "a read that never answered was recorded as a node with no healthy pod:"
1058+
cat "$d/out/pod-tenant-test-wedged.txt"
1059+
exit 1
1060+
fi
1061+
grep -q 'the pod list did not answer' "$d/out/pod-tenant-test-wedged.txt"
1062+
# And the reason ships beside the capture, not only in the job log.
1063+
grep -q 'listing the pods on node node-a' "$d/out/capture-notes.txt"
1064+
rm -rf "$d"
1065+
}
1066+
7681067
@test "a memo hit reports the same answer as the miss that filled it" {
7691068
# The status has to survive the memo. Every caller reads pod_on_node through a
7701069
# command substitution, so a status kept in a variable dies with that subshell

0 commit comments

Comments
 (0)