Skip to content

Commit 1f6bc1b

Browse files
IvanHuntersIvanHunters
committed
test(e2e): share per-node baseline capture with e2e-capture-dataplane.sh
Address the review on the per-node network diagnostics. Instead of a divergent second copy of the CNI capture logic in cozyreport.sh, add a --baseline mode to hack/e2e-capture-dataplane.sh that cozyreport.sh invokes, so the namespaces, DaemonSet selectors, container names and the conntrack CLI->/proc fallback have a single source of truth shared with the failure-triggered capture. - replace the hubble capture (hubble.enabled is false on cozystack, so the socket is never created and it can never produce output) with cilium-dbg monitor --type drop, which needs no Hubble; - carry the /proc/net/nf_conntrack fallback for images without the conntrack CLI; - keep records of failed captures: prune only empty files and this script's own DIAG-SKIP absence markers, never a real kubectl exec error, so a broken capture stays distinguishable from an absent tool; - bound the whole capture with an aggregate timeout -k at the call site so a wedged API server cannot cost the entire cozyreport artefact; - unit-test the pruning predicate in hack/capture-dataplane.bats. Signed-off-by: IvanHunters <[email protected]>
1 parent 8fb1e11 commit 1f6bc1b

3 files changed

Lines changed: 208 additions & 135 deletions

File tree

‎hack/capture-dataplane.bats‎

Lines changed: 70 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,22 @@
11
#!/usr/bin/env bats
22
# -----------------------------------------------------------------------------
33
# Unit tests for the pure decision/parsing helpers in
4-
# hack/e2e-capture-dataplane.sh -- specifically the LoadBalancer-datapath
5-
# section, which fires for a Service type=LoadBalancer whose external IP is
6-
# unreachable while its backend stays Ready (the NotReady-pod path never sees
7-
# this case). Only the pure logic is unit-testable here:
4+
# hack/e2e-capture-dataplane.sh. Only the pure logic is unit-testable here:
85
#
6+
# LoadBalancer-datapath section (fires for a Service type=LoadBalancer whose
7+
# external IP is unreachable while its backend stays Ready -- the NotReady-pod
8+
# path never sees this case):
99
# - lb_filter_services -- keep only LoadBalancer rows with an ingress IP;
1010
# - lb_first_ready_endpoint -- pick the first addressed, non-NotReady endpoint;
1111
# - lb_announcer_node -- the speaker node of the most recent announce;
1212
# - lb_capture_decision -- capture only when every probe failed.
1313
#
14+
# Baseline per-node capture (--baseline mode, invoked by cozyreport.sh):
15+
# - _diag_prunable -- drop only empty files and this script's own
16+
# "DIAG-SKIP:" absence markers, never a real
17+
# kubectl exec failure (a broken capture must stay
18+
# distinguishable from a node that lacked the tool).
19+
#
1420
# The kubectl exec / tcpdump capture itself is not unit-testable (it needs a
1521
# live cluster); these tests pin the derivations that decide WHICH node to
1622
# capture on and WHETHER to capture at all, fed mock kubectl/log output.
@@ -225,3 +231,63 @@ E2E_CAPTURE_DATAPLANE_LIB=1
225231
[ "$captured" -eq 2 ]
226232
[ "$dropped" -eq 1 ]
227233
}
234+
235+
# --------------------------------------------------------------------------- #
236+
# _diag_prunable -- the baseline-capture pruning predicate. It must drop empty #
237+
# files and this script's own "DIAG-SKIP:" absence markers, but KEEP a genuine #
238+
# kubectl exec failure so a broken capture stays distinguishable from an absent #
239+
# tool (PR review blocker B2). Mock inputs use real kubectl failure text. #
240+
# --------------------------------------------------------------------------- #
241+
242+
@test "diag prunable drops an empty capture file" {
243+
f="$(mktemp)"
244+
: > "$f"
245+
if _diag_prunable "$f"; then rm -f "$f"; else rm -f "$f"; echo "FAIL: empty file must be prunable"; false; fi
246+
}
247+
248+
@test "diag prunable drops a lone DIAG SKIP absence marker" {
249+
f="$(mktemp)"
250+
printf 'DIAG-SKIP: iptables-save not present in cni-server\n' > "$f"
251+
if _diag_prunable "$f"; then rm -f "$f"; else rm -f "$f"; echo "FAIL: DIAG-SKIP stub must be prunable"; false; fi
252+
}
253+
254+
@test "diag prunable keeps a kubectl Error from server exec failure" {
255+
f="$(mktemp)"
256+
printf 'Error from server (BadRequest): pod does not have a host assigned\n' > "$f"
257+
if _diag_prunable "$f"; then rm -f "$f"; echo "FAIL: a capture failure must be kept, not pruned"; false; fi
258+
rm -f "$f"
259+
}
260+
261+
@test "diag prunable keeps a container not found OCI runtime exec failure" {
262+
f="$(mktemp)"
263+
printf 'error: unable to upgrade connection: container not found (cilium-agent)\n' > "$f"
264+
if _diag_prunable "$f"; then rm -f "$f"; echo "FAIL: an exec failure must be kept, not pruned"; false; fi
265+
rm -f "$f"
266+
}
267+
268+
@test "diag prunable keeps a real multi line capture" {
269+
f="$(mktemp)"
270+
printf 'Chain INPUT policy ACCEPT\ntarget prot source destination\nACCEPT all anywhere anywhere\n' > "$f"
271+
if _diag_prunable "$f"; then rm -f "$f"; echo "FAIL: a real capture must be kept"; false; fi
272+
rm -f "$f"
273+
}
274+
275+
@test "diag prunable keeps a legit one line capture that contains the word error" {
276+
f="$(mktemp)"
277+
printf 'flow verdict FORWARDED reason error none\n' > "$f"
278+
if _diag_prunable "$f"; then rm -f "$f"; echo "FAIL: a real one-line capture must be kept even if it contains the word error"; false; fi
279+
rm -f "$f"
280+
}
281+
282+
@test "diag prunable keeps a multi line file even when the first line is a DIAG SKIP" {
283+
f="$(mktemp)"
284+
printf 'DIAG-SKIP: header\nreal capture line follows\nand another\n' > "$f"
285+
if _diag_prunable "$f"; then rm -f "$f"; echo "FAIL: a multi-line file must be kept even if line 1 is DIAG-SKIP"; false; fi
286+
rm -f "$f"
287+
}
288+
289+
@test "diag prunable drops a DIAG SKIP marker that lacks a trailing newline" {
290+
f="$(mktemp)"
291+
printf 'DIAG-SKIP: iptables-save not present in cni-server' > "$f"
292+
if _diag_prunable "$f"; then rm -f "$f"; else rm -f "$f"; echo "FAIL: a newline-less DIAG-SKIP marker must still be prunable"; false; fi
293+
}

‎hack/cozyreport.sh‎

Lines changed: 16 additions & 124 deletions
Original file line numberDiff line numberDiff line change
@@ -350,132 +350,24 @@ fi
350350

351351
# -- per-node network diagnostics
352352
#
353-
# kubelet -> pod probe failures (readiness/liveness `dial tcp <podIP>:<port>:
354-
# connect: connection refused` on a pod whose stdout log confirms the server
355-
# is bound) are recurrent baseline flakes in the multi-CNI sandbox (kube-ovn
356-
# + cilium + multus). Cozyreport at end-of-job carries CiliumEndpoint
357-
# `state: ready`, the pod-side listener log, and the kube-ovn CNI ADD trace
358-
# for the pod IP -- all of which say the path should work -- but omits the
359-
# host-side networking state that would prove or disprove the drop. Without
360-
# it, the pod either (a) is silently on stale conntrack from a previous
361-
# incarnation with the same IP, (b) is blocked by a hostFirewall or Cilium
362-
# policy drop that only Hubble sees, or (c) hits a kube-ovn OVS flow race
363-
# between the kube-ovn-cni ADD and the kernel-side iptables NAT chain.
364-
# All three leave signature in per-node artefacts a cozyreport that only
365-
# captures pod-scoped state cannot recover.
353+
# Baseline host-side + CNI data-plane state (iptables, kernel conntrack, OVS
354+
# flows, cilium endpoints + drop events) for every node, captured on every
355+
# report. cozyreport otherwise carries only pod-scoped state (CiliumEndpoint,
356+
# listener log, CNI ADD trace), which cannot prove or disprove a host-side
357+
# kubelet->local-pod "connection refused" transient (stale conntrack on a reused
358+
# podIP, a policy drop, or a kube-ovn OVS flow race). Delegated to
359+
# hack/e2e-capture-dataplane.sh --baseline so the CNI topology knowledge
360+
# (namespaces, DaemonSet selectors, container names, the conntrack CLI->/proc
361+
# fallback) has a single source of truth shared with the failure-triggered
362+
# capture cozytest.sh runs, rather than a divergent second copy here.
366363
#
367-
# Every source below is a DaemonSet pod that already runs on the node with
368-
# hostNetwork and the required binary preinstalled: `iptables-save` +
369-
# `conntrack` live in `kube-ovn-cni` (`cni-server` container), OVS flows
370-
# live in `ovs-ovn` (`openvswitch` container), and `hubble observe` +
371-
# `cilium-dbg endpoint list` live in `cilium` (`cilium-agent`). Optional
372-
# binaries are `command -v` probed inline so an image without the tool
373-
# leaves a single "not present" line instead of an OCI runtime error
374-
# masquerading as a diagnostic capture. Every exec is wrapped in
375-
# `timeout 30 kubectl exec ...` (matching the upper bound the sibling
376-
# `hack/e2e-capture-dataplane.sh` uses for the heaviest captures) so a
377-
# wedged agent (bpf-map iteration stall, hubble ring-buffer replay,
378-
# ovn-controller lock) cannot hang the surrounding cozyreport run.
379-
# Files that end up empty or contain only a known error-signature line
380-
# are dropped so the artefact tree carries only real captures.
381-
#
382-
# `timeout(1)` from coreutils is a hard dependency of this block. Skip
383-
# the whole section on runners without it rather than filling the tree
384-
# with `sh: 1: timeout: not found` stubs.
385-
if ! command -v timeout >/dev/null 2>&1; then
386-
echo " timeout(1) not present -- skipping per-node network diagnostics"
387-
else
364+
# Diagnostic-only: never mutates the cluster, best-effort (|| true). The whole
365+
# capture is wrapped in an aggregate `timeout -k` so a wedged API server or agent
366+
# cannot cost the entire cozyreport artefact -- matching the wrapper cozytest.sh
367+
# puts around the same script (5 bounded execs x N nodes has no aggregate bound
368+
# otherwise).
388369
echo "Collecting per-node network diagnostics..."
389-
DIR=$REPORT_DIR/net-diag
390-
mkdir -p "$DIR"
391-
392-
# _keep_if_useful <file>
393-
# Drop the capture file if it is empty or its content is a known "no
394-
# useful data" signature (missing binary, container exec error). This
395-
# stops error-text stubs from pretending to be real captures. A file
396-
# with real data (dozens/hundreds of lines) is left untouched; a
397-
# single-line "not present" / "OCI runtime" / "command terminated"
398-
# stub gets removed so the artefact tree carries only real captures.
399-
_keep_if_useful() {
400-
_f="$1"
401-
if [ ! -s "$_f" ]; then
402-
rm -f "$_f"
403-
return
404-
fi
405-
if [ "$(wc -l < "$_f" 2>/dev/null)" -le 1 ] && \
406-
grep --extended-regexp --quiet "not (present|enabled|found)|socket missing|OCI runtime|executable file not found|command terminated|error:|Error from server" "$_f" 2>/dev/null; then
407-
rm -f "$_f"
408-
fi
409-
}
410-
411-
for pod in $(kubectl -n cozy-kubeovn get pods -l app=kube-ovn-cni -o name 2>/dev/null); do
412-
node=$(kubectl -n cozy-kubeovn get "$pod" -o jsonpath='{.spec.nodeName}' 2>/dev/null)
413-
[ -z "$node" ] && node=$(basename "$pod")
414-
# `iptables-save` dumps every table (filter/nat/mangle/raw) as a single
415-
# text blob; the whole thing lands in one file per node so an operator
416-
# can grep across chains without unpacking per-table splits.
417-
timeout 30 kubectl -n cozy-kubeovn exec "$pod" --container=cni-server -- \
418-
sh -c 'command -v iptables-save >/dev/null 2>&1 && iptables-save || echo "iptables-save not present in cni-server"' \
419-
> "$DIR/$node-iptables.txt" 2>&1 || true
420-
_keep_if_useful "$DIR/$node-iptables.txt"
421-
# `conntrack -L` lists the kernel conntrack table (5-tuple + state +
422-
# bytes). `head -n 5000` bounds output so a node with heavy tenant
423-
# traffic does not push megabytes of connections into the artefact.
424-
# NOTE: conntrack iterates the kernel hash table in bucket order, not
425-
# by connection age, so the truncation is not a guaranteed FIFO of
426-
# recent entries — an operator chasing a specific 5-tuple should grep
427-
# for the podIP/port rather than assume the tail is missing.
428-
timeout 30 kubectl -n cozy-kubeovn exec "$pod" --container=cni-server -- \
429-
sh -c 'command -v conntrack >/dev/null 2>&1 && conntrack -L 2>/dev/null | head -n 5000 || echo "conntrack not present in cni-server"' \
430-
> "$DIR/$node-conntrack.txt" 2>&1 || true
431-
_keep_if_useful "$DIR/$node-conntrack.txt"
432-
done
433-
434-
for pod in $(kubectl -n cozy-kubeovn get pods -l app=ovs -o name 2>/dev/null); do
435-
node=$(kubectl -n cozy-kubeovn get "$pod" -o jsonpath='{.spec.nodeName}' 2>/dev/null)
436-
[ -z "$node" ] && node=$(basename "$pod")
437-
# `ovs-ofctl dump-flows br-int` is the OVS side of the kube-ovn
438-
# datapath. Distinguishes a CNI ADD that finished from an ovn-controller
439-
# that has not programmed the pod's flow into OVS yet — the flow race
440-
# the block-comment above cites as failure mode (c).
441-
timeout 30 kubectl -n cozy-kubeovn exec "$pod" --container=openvswitch -- \
442-
sh -c 'command -v ovs-ofctl >/dev/null 2>&1 && ovs-ofctl dump-flows br-int || echo "ovs-ofctl not present in openvswitch"' \
443-
> "$DIR/$node-ovs-flows.txt" 2>&1 || true
444-
_keep_if_useful "$DIR/$node-ovs-flows.txt"
445-
done
446-
447-
for pod in $(kubectl -n cozy-cilium get pods -l app.kubernetes.io/name=cilium-agent -o name 2>/dev/null); do
448-
node=$(kubectl -n cozy-cilium get "$pod" -o jsonpath='{.spec.nodeName}' 2>/dev/null)
449-
[ -z "$node" ] && node=$(basename "$pod")
450-
# `hubble observe --last N` prints the recent flow log from the local
451-
# cilium-agent ring buffer, including verdicts (FORWARDED / DROPPED /
452-
# ERROR) with the reason field. Probe for both the binary AND the
453-
# hubble unix socket -- the CLI is present in every recent cilium
454-
# image, but hubble observability itself is optional (feature flag
455-
# `hubble.enabled` in the cilium chart), so the socket may not exist
456-
# even when the CLI does. Sized so a probe-drop cascade over a
457-
# 5-minute window still fits, and a healthy node's log stays small.
458-
timeout 30 kubectl -n cozy-cilium exec "$pod" --container=cilium-agent -- \
459-
sh -c 'command -v hubble >/dev/null 2>&1 && [ -S /var/run/cilium/hubble.sock ] && hubble observe --last 2000 --output=compact 2>&1 || echo "hubble not enabled or socket missing in cilium-agent"' \
460-
> "$DIR/$node-hubble-flows.txt" 2>&1 || true
461-
_keep_if_useful "$DIR/$node-hubble-flows.txt"
462-
# `cilium-dbg endpoint list` gives the per-endpoint policy state
463-
# (identity, security policies, state, controllers), which resolves
464-
# the question "was the drop a policy decision or a datapath race"
465-
# that hubble alone doesn't disambiguate.
466-
timeout 30 kubectl -n cozy-cilium exec "$pod" --container=cilium-agent -- \
467-
sh -c 'command -v cilium-dbg >/dev/null 2>&1 && cilium-dbg endpoint list || echo "cilium-dbg not present in cilium-agent"' \
468-
> "$DIR/$node-cilium-endpoints.txt" 2>&1 || true
469-
_keep_if_useful "$DIR/$node-cilium-endpoints.txt"
470-
done
471-
472-
# Drop the parent directory if no artefact landed (e.g. neither kube-ovn
473-
# nor cilium was present on the cluster the report is run against, or
474-
# every capture was pruned by `_keep_if_useful`).
475-
if [ -z "$(ls -A "$DIR" 2>/dev/null)" ]; then
476-
rmdir "$DIR" 2>/dev/null || true
477-
fi
478-
fi
370+
timeout -k 30 300 "$SCRIPT_DIR/e2e-capture-dataplane.sh" --baseline "$REPORT_DIR/net-diag" || true
479371

480372
# -- finalization
481373

0 commit comments

Comments
 (0)