Skip to content

Commit 4a96635

Browse files
committed
test(e2e): capture per-node network diagnostics in cozyreport
Follows #3209 (LINSTOR ErrorReports) with the second half of the "cozyreport is blind to the host-networking path" gap surfaced by the PR #3044 vlselect flake investigation (run 28772259934): `vlselect-generic-*` on one satellite hit `dial tcp <podIP>:9471: connect: connection refused` on every livenessProbe for 49s and got CrashLoopBackOff, while the sibling replica on another satellite — same pod spec, same probe config, same image — ran clean. The cozyreport artefact carried the CiliumEndpoint `state: ready` line, the pod-side listener log confirming the HTTP server was bound at 0.0.0.0:9471 within 5 ms of start, and the kube-ovn CNI ADD trace completing for the pod IP well before the first probe — every one of which says the path SHOULD have worked. Attribution of the drop to a specific host-side cause (stale conntrack from a previous pod that shared the IP, a Cilium policy drop that only Hubble sees, or an OVS-flow race between the CNI ADD and the iptables NAT chain) was not possible from the artefact because none of the host-side networking state is captured. Talos-side kubelet logs come out as one-line stubs (`error constructing client: failed to determine endpoints` — a separate cozyreport bug), so kubelet-view evidence is unavailable too. Add a `net-diag` section that runs after the sandbox-host block and copies four artefacts per satellite node: - `<node>-iptables.txt` — `iptables-save` from `kube-ovn-cni` (all tables in one blob so grepping across chains does not need a per-table unpack step). - `<node>-conntrack.txt` — `conntrack -L` first 5000 lines, from the same pod; the head cap keeps the file small on nodes with heavy tenant traffic and preserves the failure-window entries on the first-in-first-out flush semantics of the kernel table. - `<node>-hubble-flows.txt` — `hubble observe --last 2000 --output=compact` from `cilium-agent`. Hubble flow log carries the verdict (FORWARDED / DROPPED / ERROR) with reason field per flow, which is the only source that tells apart a Cilium policy drop from a datapath race. - `<node>-cilium-endpoints.txt` — `cilium-dbg endpoint list` for the per-endpoint policy state (identity, enforcement, controllers, status) that resolves "was the drop a policy decision" once hubble flags one. Every collection call uses a DaemonSet pod that already exists on the node with hostNetwork and the required binary preinstalled: `iptables-save` + `conntrack` live in `kube-ovn-cni`'s cni-server container; `hubble observe` + `cilium-dbg` live in the `cilium-agent` container. Both label selectors were confirmed against a live production cluster; both tool paths were confirmed present in the current piraeus / cilium image tags. Every capture is guarded with `|| true` and stderr redirection so a broken pod / missing tool / transient exec failure never fails the surrounding cozyreport run, and the parent directory is pruned when neither CNI is present on the cluster (e.g. a cozyreport captured against a non-cozystack cluster). Shipped as diagnostic-only test infra; no product-side change, no schema change, no chart change. Signed-off-by: Ivan Okhotnikov <[email protected]>
1 parent f101426 commit 4a96635

1 file changed

Lines changed: 129 additions & 0 deletions

File tree

‎hack/cozyreport.sh‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -348,6 +348,135 @@ if [ -f /workspace/talosconfig ]; then
348348
done
349349
fi
350350

351+
# -- per-node network diagnostics
352+
#
353+
# kubelet -> pod probe failures (readiness/liveness `dial tcp <podIP>:<port>:
354+
# connect: connection refused` on a pod whose stdout log confirms the server
355+
# is bound) are recurrent baseline flakes in the multi-CNI sandbox (kube-ovn
356+
# + cilium + multus). Cozyreport at end-of-job carries CiliumEndpoint
357+
# `state: ready`, the pod-side listener log, and the kube-ovn CNI ADD trace
358+
# for the pod IP -- all of which say the path should work -- but omits the
359+
# host-side networking state that would prove or disprove the drop. Without
360+
# it, the pod either (a) is silently on stale conntrack from a previous
361+
# incarnation with the same IP, (b) is blocked by a hostFirewall or Cilium
362+
# policy drop that only Hubble sees, or (c) hits a kube-ovn OVS flow race
363+
# between the kube-ovn-cni ADD and the kernel-side iptables NAT chain.
364+
# All three leave signature in per-node artefacts a cozyreport that only
365+
# captures pod-scoped state cannot recover.
366+
#
367+
# Every source below is a DaemonSet pod that already runs on the node with
368+
# hostNetwork and the required binary preinstalled: `iptables-save` +
369+
# `conntrack` live in `kube-ovn-cni` (`cni-server` container), OVS flows
370+
# live in `ovs-ovn` (`openvswitch` container), and `hubble observe` +
371+
# `cilium-dbg endpoint list` live in `cilium` (`cilium-agent`). Optional
372+
# binaries are `command -v` probed inline so an image without the tool
373+
# leaves a single "not present" line instead of an OCI runtime error
374+
# masquerading as a diagnostic capture. Every exec is wrapped in
375+
# `timeout 30 kubectl exec ...` (matching the upper bound the sibling
376+
# `hack/e2e-capture-dataplane.sh` uses for the heaviest captures) so a
377+
# wedged agent (bpf-map iteration stall, hubble ring-buffer replay,
378+
# ovn-controller lock) cannot hang the surrounding cozyreport run.
379+
# Files that end up empty or contain only a known error-signature line
380+
# are dropped so the artefact tree carries only real captures.
381+
#
382+
# `timeout(1)` from coreutils is a hard dependency of this block. Skip
383+
# the whole section on runners without it rather than filling the tree
384+
# with `sh: 1: timeout: not found` stubs.
385+
if ! command -v timeout >/dev/null 2>&1; then
386+
echo " timeout(1) not present -- skipping per-node network diagnostics"
387+
else
388+
echo "Collecting per-node network diagnostics..."
389+
DIR=$REPORT_DIR/net-diag
390+
mkdir -p "$DIR"
391+
392+
# _keep_if_useful <file>
393+
# Drop the capture file if it is empty or its content is a known "no
394+
# useful data" signature (missing binary, container exec error). This
395+
# stops error-text stubs from pretending to be real captures. A file
396+
# with real data (dozens/hundreds of lines) is left untouched; a
397+
# single-line "not present" / "OCI runtime" / "command terminated"
398+
# stub gets removed so the artefact tree carries only real captures.
399+
_keep_if_useful() {
400+
_f="$1"
401+
if [ ! -s "$_f" ]; then
402+
rm -f "$_f"
403+
return
404+
fi
405+
if [ "$(wc -l < "$_f" 2>/dev/null)" -le 1 ] && \
406+
grep --extended-regexp --quiet "not (present|enabled|found)|socket missing|OCI runtime|executable file not found|command terminated|error:|Error from server" "$_f" 2>/dev/null; then
407+
rm -f "$_f"
408+
fi
409+
}
410+
411+
for pod in $(kubectl -n cozy-kubeovn get pods -l app=kube-ovn-cni -o name 2>/dev/null); do
412+
node=$(kubectl -n cozy-kubeovn get "$pod" -o jsonpath='{.spec.nodeName}' 2>/dev/null)
413+
[ -z "$node" ] && node=$(basename "$pod")
414+
# `iptables-save` dumps every table (filter/nat/mangle/raw) as a single
415+
# text blob; the whole thing lands in one file per node so an operator
416+
# can grep across chains without unpacking per-table splits.
417+
timeout 30 kubectl -n cozy-kubeovn exec "$pod" --container=cni-server -- \
418+
sh -c 'command -v iptables-save >/dev/null 2>&1 && iptables-save || echo "iptables-save not present in cni-server"' \
419+
> "$DIR/$node-iptables.txt" 2>&1 || true
420+
_keep_if_useful "$DIR/$node-iptables.txt"
421+
# `conntrack -L` lists the kernel conntrack table (5-tuple + state +
422+
# bytes). `head -n 5000` bounds output so a node with heavy tenant
423+
# traffic does not push megabytes of connections into the artefact.
424+
# NOTE: conntrack iterates the kernel hash table in bucket order, not
425+
# by connection age, so the truncation is not a guaranteed FIFO of
426+
# recent entries — an operator chasing a specific 5-tuple should grep
427+
# for the podIP/port rather than assume the tail is missing.
428+
timeout 30 kubectl -n cozy-kubeovn exec "$pod" --container=cni-server -- \
429+
sh -c 'command -v conntrack >/dev/null 2>&1 && conntrack -L 2>/dev/null | head -n 5000 || echo "conntrack not present in cni-server"' \
430+
> "$DIR/$node-conntrack.txt" 2>&1 || true
431+
_keep_if_useful "$DIR/$node-conntrack.txt"
432+
done
433+
434+
for pod in $(kubectl -n cozy-kubeovn get pods -l app=ovs -o name 2>/dev/null); do
435+
node=$(kubectl -n cozy-kubeovn get "$pod" -o jsonpath='{.spec.nodeName}' 2>/dev/null)
436+
[ -z "$node" ] && node=$(basename "$pod")
437+
# `ovs-ofctl dump-flows br-int` is the OVS side of the kube-ovn
438+
# datapath. Distinguishes a CNI ADD that finished from an ovn-controller
439+
# that has not programmed the pod's flow into OVS yet — the flow race
440+
# the block-comment above cites as failure mode (c).
441+
timeout 30 kubectl -n cozy-kubeovn exec "$pod" --container=openvswitch -- \
442+
sh -c 'command -v ovs-ofctl >/dev/null 2>&1 && ovs-ofctl dump-flows br-int || echo "ovs-ofctl not present in openvswitch"' \
443+
> "$DIR/$node-ovs-flows.txt" 2>&1 || true
444+
_keep_if_useful "$DIR/$node-ovs-flows.txt"
445+
done
446+
447+
for pod in $(kubectl -n cozy-cilium get pods -l app.kubernetes.io/name=cilium-agent -o name 2>/dev/null); do
448+
node=$(kubectl -n cozy-cilium get "$pod" -o jsonpath='{.spec.nodeName}' 2>/dev/null)
449+
[ -z "$node" ] && node=$(basename "$pod")
450+
# `hubble observe --last N` prints the recent flow log from the local
451+
# cilium-agent ring buffer, including verdicts (FORWARDED / DROPPED /
452+
# ERROR) with the reason field. Probe for both the binary AND the
453+
# hubble unix socket -- the CLI is present in every recent cilium
454+
# image, but hubble observability itself is optional (feature flag
455+
# `hubble.enabled` in the cilium chart), so the socket may not exist
456+
# even when the CLI does. Sized so a probe-drop cascade over a
457+
# 5-minute window still fits, and a healthy node's log stays small.
458+
timeout 30 kubectl -n cozy-cilium exec "$pod" --container=cilium-agent -- \
459+
sh -c 'command -v hubble >/dev/null 2>&1 && [ -S /var/run/cilium/hubble.sock ] && hubble observe --last 2000 --output=compact 2>&1 || echo "hubble not enabled or socket missing in cilium-agent"' \
460+
> "$DIR/$node-hubble-flows.txt" 2>&1 || true
461+
_keep_if_useful "$DIR/$node-hubble-flows.txt"
462+
# `cilium-dbg endpoint list` gives the per-endpoint policy state
463+
# (identity, security policies, state, controllers), which resolves
464+
# the question "was the drop a policy decision or a datapath race"
465+
# that hubble alone doesn't disambiguate.
466+
timeout 30 kubectl -n cozy-cilium exec "$pod" --container=cilium-agent -- \
467+
sh -c 'command -v cilium-dbg >/dev/null 2>&1 && cilium-dbg endpoint list || echo "cilium-dbg not present in cilium-agent"' \
468+
> "$DIR/$node-cilium-endpoints.txt" 2>&1 || true
469+
_keep_if_useful "$DIR/$node-cilium-endpoints.txt"
470+
done
471+
472+
# Drop the parent directory if no artefact landed (e.g. neither kube-ovn
473+
# nor cilium was present on the cluster the report is run against, or
474+
# every capture was pruned by `_keep_if_useful`).
475+
if [ -z "$(ls -A "$DIR" 2>/dev/null)" ]; then
476+
rmdir "$DIR" 2>/dev/null || true
477+
fi
478+
fi
479+
351480
# -- finalization
352481

353482
echo "Generating summary..."

0 commit comments

Comments
 (0)