Skip to content

Commit 18e459a

Browse files
fix(e2e): improve failure diagnostics (#3421)
## What this PR does Improves E2E failure diagnostics tracked in #3368: - preserves scheduled, non-terminal NotReady pods before a pod IP is assigned - always captures load-balancer state while gating IP-dependent diagnostics - collects Talos dmesg, kubelet, and containerd logs per host node using explicit endpoint and node arguments - renders COSI Bucket catch output as YAML - makes pre-commit generation locking work from linked worktrees - adds regression coverage for dataplane and Talos collection Tenant-worker Talos logs remain outside this PR because the runner does not currently have a tenant talosconfig. Part of #3368. ## Testing - `hack/cozytest.sh hack/capture-dataplane.bats` - `hack/cozytest.sh hack/cozyreport-talos.bats` - `sh -n hack/e2e-capture-dataplane.sh` - `sh -n hack/cozyreport.sh` - `chainsaw test --no-cluster --config hack/e2e-chainsaw/.chainsaw.yaml --include-test-regex '^$' hack/e2e-chainsaw/bucket hack/e2e-chainsaw/harbor` - `pre-commit run --all-files` ### Screenshots Not applicable. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(e2e): improve diagnostic capture for dataplane, Talos, and COSI failures ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Improved dataplane diagnostics for scheduled pods that are NotReady, including cases without an assigned IP. - LoadBalancer checks now proceed even when no affected pods are found. - Standardized Talos node diagnostics collection for consistent kernel, kubelet, and containerd logs. - Improved reliability of generation hooks by using a git-common lockfile path for concurrent runs. - **Tests** - Added/expanded Bats coverage for dataplane pod filtering and Talos diagnostics argument correctness. - Improved end-to-end bucket test retrieval/formatting for clearer failure output. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
2 parents f077420 + 6a41d65 commit 18e459a

7 files changed

Lines changed: 165 additions & 43 deletions

File tree

‎.pre-commit-config.yaml‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ repos:
44
- id: run-make-generate-root
55
name: Run 'make generate' at repo root
66
entry: |
7-
flock -x .git/pre-commit.lock sh -c '
7+
sh -c 'exec flock -x "$(git rev-parse --git-common-dir)/pre-commit.lock" "$@"' -- sh -c '
88
echo "Running make generate at repo root"
99
make generate || exit $?
1010
git diff --color=always | cat
@@ -15,7 +15,7 @@ repos:
1515
- id: run-make-generate
1616
name: Run 'make generate' in all app directories
1717
entry: |
18-
flock -x .git/pre-commit.lock sh -c '
18+
sh -c 'exec flock -x "$(git rev-parse --git-common-dir)/pre-commit.lock" "$@"' -- sh -c '
1919
for dir in ./packages/apps/*/ ./packages/extra/*/; do
2020
if [ -d "$dir" ]; then
2121
echo "Running make generate in $dir"

‎hack/capture-dataplane.bats‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,6 +10,7 @@
1010
# - lb_first_ready_endpoint -- pick the first addressed, non-NotReady endpoint;
1111
# - lb_announcer_node -- the speaker node of the most recent announce;
1212
# - lb_capture_decision -- capture only when every probe failed.
13+
# - pod_filter_affected -- retain scheduled NotReady pods even without IPs.
1314
#
1415
# The kubectl exec / tcpdump capture itself is not unit-testable (it needs a
1516
# live cluster); these tests pin the derivations that decide WHICH node to
@@ -41,6 +42,47 @@ E2E_CAPTURE_DATAPLANE_LIB=1
4142
# shellcheck source=/dev/null
4243
. "$SCRIPT"
4344

45+
@test "pod_filter_affected keeps a scheduled NotReady pod without a podIP" {
46+
rows="$(printf '%s\n' \
47+
'tenant|no-ip||node-a|False|Pending' \
48+
'tenant|with-ip|192.0.2.80|node-b|False|Running')"
49+
50+
out="$(printf '%s\n' "$rows" | pod_filter_affected)"
51+
52+
[ "$(printf '%s\n' "$out" | awk 'END { print NR }')" -eq 2 ]
53+
printf '%s\n' "$out" | awk '$0 == "tenant|no-ip||node-a|False|Pending" { found = 1 } END { exit !found }'
54+
}
55+
56+
@test "pod_filter_affected drops Ready unscheduled and terminal pods" {
57+
rows="$(printf '%s\n' \
58+
'tenant|ready|192.0.2.81|node-a|True|Running' \
59+
'tenant|unscheduled|||False|Pending' \
60+
'tenant|succeeded|192.0.2.82|node-b|False|Succeeded' \
61+
'tenant|failed|192.0.2.83|node-b|False|Failed')"
62+
63+
[ -z "$(printf '%s\n' "$rows" | pod_filter_affected)" ]
64+
}
65+
66+
@test "runtime checks LoadBalancers when there are no affected pods" {
67+
tmp=$(mktemp -d)
68+
trap 'rm -rf "$tmp"' EXIT
69+
calls="$tmp/kubectl.calls"
70+
mkdir -p "$tmp/bin"
71+
cat > "$tmp/bin/kubectl" <<'EOF'
72+
#!/bin/sh
73+
printf '%s\n' "$*" >> "$MOCK_KUBECTL_CALLS"
74+
exit 0
75+
EOF
76+
chmod +x "$tmp/bin/kubectl"
77+
78+
MOCK_KUBECTL_CALLS="$calls" PATH="$tmp/bin:$PATH" \
79+
"$SCRIPT" "$tmp/out" > "$tmp/stdout" 2>&1
80+
81+
awk '$0 ~ /^get svc -A / { found = 1 } END { exit !found }' "$calls"
82+
awk 'index($0, "checking LoadBalancers independently") { found = 1 } END { exit !found }' "$tmp/stdout"
83+
awk 'index($0, "no Service type=LoadBalancer") { found = 1 } END { exit !found }' "$tmp/stdout"
84+
}
85+
4486
@test "lb_filter_services keeps only LoadBalancer rows that have an ingress IP" {
4587
rows="$(printf '%s\n' \
4688
'tenant|app|LoadBalancer|192.0.2.50|80|31000|Cluster' \

‎hack/cozyreport-talos.bats‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
#!/usr/bin/env bats
2+
# Regression coverage for host-node Talos diagnostics in hack/cozyreport.sh.
3+
# The full report needs a cluster; this test sources only the focused helper and
4+
# replaces talosctl with a shell function that records the exact argument vector.
5+
6+
HACK_DIR="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")" && pwd)"
7+
SCRIPT="$HACK_DIR/cozyreport.sh"
8+
9+
COZYREPORT_LIB=1
10+
# shellcheck source=/dev/null
11+
. "$SCRIPT"
12+
13+
# cozytest.sh's parser ends an @test block at the first bare `}`, so keep the
14+
# talosctl mock at top level rather than nesting a function inside the test.
15+
talosctl() {
16+
printf '%s\n' "$*" >> "$calls"
17+
}
18+
19+
@test "Talos host diagnostics pass an endpoint and use valid dmesg flags" {
20+
tmp=$(mktemp -d)
21+
trap 'rm -rf "$tmp"' EXIT
22+
calls="$tmp/talosctl.calls"
23+
24+
cozyreport_collect_talos_node /workspace/talosconfig 192.0.2.11 "$tmp"
25+
26+
[ "$(sed -n '1p' "$calls")" = "--talosconfig /workspace/talosconfig -e 192.0.2.11 -n 192.0.2.11 dmesg" ]
27+
[ "$(sed -n '2p' "$calls")" = "--talosconfig /workspace/talosconfig -e 192.0.2.11 -n 192.0.2.11 logs kubelet --tail=500" ]
28+
[ "$(sed -n '3p' "$calls")" = "--talosconfig /workspace/talosconfig -e 192.0.2.11 -n 192.0.2.11 logs containerd --tail=500" ]
29+
case "$(sed -n '1p' "$calls")" in *--tail*)
30+
echo "dmesg must not receive the boolean --tail flag" >&2
31+
false
32+
esac
33+
}

‎hack/cozyreport.sh‎

Lines changed: 28 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,31 @@
11
#!/bin/sh
2+
3+
# cozyreport_collect_talos_node <talosconfig> <node> <output-dir>
4+
#
5+
# The e2e talosconfig intentionally carries no endpoints, so both -e and -n are
6+
# required. Each node's InternalIP is itself a Talos API endpoint. `dmesg
7+
# --tail` is a boolean follow-mode flag (unlike `logs --tail`, which is an
8+
# integer), so request the complete kernel ring buffer and retain bounded tails
9+
# only for kubelet/containerd service logs.
10+
cozyreport_collect_talos_node() {
11+
_crt_config=$1
12+
_crt_node=$2
13+
_crt_dir=$3
14+
15+
talosctl --talosconfig "$_crt_config" -e "$_crt_node" -n "$_crt_node" \
16+
dmesg > "$_crt_dir/talos-$_crt_node-dmesg.txt" 2>&1 || true
17+
talosctl --talosconfig "$_crt_config" -e "$_crt_node" -n "$_crt_node" \
18+
logs kubelet --tail=500 > "$_crt_dir/talos-$_crt_node-kubelet.log" 2>&1 || true
19+
talosctl --talosconfig "$_crt_config" -e "$_crt_node" -n "$_crt_node" \
20+
logs containerd --tail=500 > "$_crt_dir/talos-$_crt_node-containerd.log" 2>&1 || true
21+
}
22+
23+
# Let the focused BATS test source the command builder without running the full
24+
# cluster report. Production callers never set this variable.
25+
if [ -n "${COZYREPORT_LIB:-}" ]; then
26+
return 0 2>/dev/null
27+
fi
28+
229
SCRIPT_DIR=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
330
REPORT_DATE=$(date +%Y-%m-%d_%H-%M-%S)
431
REPORT_NAME=${1:-cozyreport-$REPORT_DATE}
@@ -342,9 +369,7 @@ if [ -f /workspace/talosconfig ]; then
342369
NODES=$(kubectl get nodes -o jsonpath='{range .items[*]}{.status.addresses[?(@.type=="InternalIP")].address}{"\n"}{end}' 2>/dev/null)
343370
for node in ${NODES:-192.168.123.11 192.168.123.12 192.168.123.13}; do
344371
[ -z "$node" ] && continue
345-
talosctl --talosconfig /workspace/talosconfig -n "$node" dmesg --tail=200 > "$DIR/talos-$node-dmesg.txt" 2>&1 || true
346-
talosctl --talosconfig /workspace/talosconfig -n "$node" logs kubelet --tail=500 > "$DIR/talos-$node-kubelet.log" 2>&1 || true
347-
talosctl --talosconfig /workspace/talosconfig -n "$node" logs containerd --tail=500 > "$DIR/talos-$node-containerd.log" 2>&1 || true
372+
cozyreport_collect_talos_node /workspace/talosconfig "$node" "$DIR"
348373
done
349374
fi
350375

‎hack/e2e-capture-dataplane.sh‎

Lines changed: 58 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,8 @@
1818
# root-caused after the fact. This collects exactly that state from the pod's
1919
# node so the next recurrence is dispositive.
2020
#
21-
# What it captures, per affected pod (NotReady, scheduled, has a podIP):
21+
# What it captures, per affected pod (NotReady and scheduled; podIP may still
22+
# be empty while CNI endpoint allocation is the thing that is stuck):
2223
# - cilium-agent on the node: cilium-dbg endpoint list; bpf ct entries for
2324
# the podIP; a short bounded `cilium-dbg monitor --type drop`; hubble
2425
# dropped-verdict observations (if the hubble CLI is present in the agent).
@@ -200,6 +201,16 @@ lb_budget_ok() {
200201
if [ "$1" -lt "$2" ]; then echo yes; else echo no; fi
201202
}
202203

204+
# pod_filter_affected: stdin =
205+
# `namespace|name|podIP|nodeName|ready|phase` rows. Keep scheduled, non-terminal
206+
# pods whose Ready condition is not True. podIP is deliberately NOT a gate: the
207+
# cilium endpointManager leak this collector diagnoses can strand a pod before
208+
# an IP is assigned, and the node-global Cilium/OVN state plus pod events remain
209+
# useful in that state. IP-specific commands are gated later at their call sites.
210+
pod_filter_affected() {
211+
awk -F'|' '$4 != "" && $5 != "True" && $6 != "Succeeded" && $6 != "Failed"'
212+
}
213+
203214
# Sourcing guard: hack/capture-dataplane.bats sets E2E_CAPTURE_DATAPLANE_LIB and
204215
# sources this file purely to reach the helpers above; return before touching $1
205216
# or running any capture so the unit test never needs a cluster. The script's
@@ -240,40 +251,20 @@ mkdir -p "$OUT" 2>/dev/null || exit 0
240251

241252
log() { echo "[capture-dataplane] $*"; }
242253

243-
# Affected = scheduled (has nodeName), has a podIP (so an endpoint exists to
244-
# inspect), Ready!=True, and not already terminal. That is the superset of the
245-
# "readiness/startup probe failing with connection refused" symptom -- a pod
246-
# that is Running with an IP but whose Ready condition is False is exactly the
247-
# transient's signature. Succeeded/Failed pods (completed Jobs, hook pods) are
248-
# Ready!=True too but are not the symptom, so they are excluded to keep the
249-
# MAX_PODS budget on Running/Pending pods that are actually wedged.
254+
# Affected = scheduled (has nodeName), Ready!=True, and not already terminal.
255+
# A podIP is intentionally optional: a CNI endpoint leak can strand the pod
256+
# before allocation. Running pods with an IP cover the host->pod probe failure;
257+
# Pending pods without one cover the endpoint-allocation failure. Succeeded/
258+
# Failed pods (completed Jobs, hook pods) are Ready!=True too but are not either
259+
# symptom, so they are excluded to keep the MAX_PODS budget on actual wedges.
250260
# jsonpath keeps this dependency-free (no jq / go-template reassignment).
251261
affected=$(kubectl get pods -A \
252262
-o jsonpath='{range .items[*]}{.metadata.namespace}{"|"}{.metadata.name}{"|"}{.status.podIP}{"|"}{.spec.nodeName}{"|"}{.status.conditions[?(@.type=="Ready")].status}{"|"}{.status.phase}{"\n"}{end}' \
253-
2>/dev/null | awk -F'|' '$3!="" && $4!="" && $5!="True" && $6!="Succeeded" && $6!="Failed"')
254-
255-
if [ -z "$affected" ]; then
256-
log "no NotReady pods with a podIP+node -- nothing to capture"
257-
exit 0
258-
fi
263+
2>/dev/null | pod_filter_affected)
259264

260-
ncount=$(printf '%s\n' "$affected" | wc -l | tr -d ' ')
261-
log "capturing host->pod data-plane for up to $MAX_PODS of $ncount affected pod(s) -> $OUT"
262-
263-
# OVN southbound logical-flow dump is cluster-global, so capture it once rather
264-
# than per pod. ovn-central is a Deployment (not per-node); any replica answers.
265-
# --no-leader-only lets a read land on a raft follower instead of erroring.
266-
central=$(kubectl get pod -n "$KUBEOVN_NS" -l app=ovn-central \
267-
-o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
268-
if [ -n "$central" ]; then
269-
{
270-
echo "=== ovn-sbctl lflow-list (cluster-global, pod=$central) ==="
271-
timeout 30 kubectl exec -n "$KUBEOVN_NS" "$central" -c ovn-central -- \
272-
ovn-sbctl --no-leader-only lflow-list 2>&1 || true
273-
} > "$OUT/ovn-lflows.txt" 2>&1 || true
274-
else
275-
log "no ovn-central pod in $KUBEOVN_NS -- skipping OVN logical-flow dump"
276-
fi
265+
# Set even when the pod path is empty; the per-pod capture references it, while
266+
# the independent LoadBalancer path below does not.
267+
central=""
277268

278269
# pod_on_node <ns> <label> <node> -> first matching pod name (empty if none).
279270
pod_on_node() {
@@ -383,10 +374,32 @@ capture_node() {
383374
} >> "$nf" 2>&1 || true
384375
}
385376

386-
i=0
387-
printf '%s\n' "$affected" | {
377+
if [ -z "$affected" ]; then
378+
log "no scheduled NotReady pods -- skipping host->pod capture; checking LoadBalancers independently"
379+
else
380+
ncount=$(printf '%s\n' "$affected" | wc -l | tr -d ' ')
381+
log "capturing host->pod data-plane for up to $MAX_PODS of $ncount affected pod(s) -> $OUT"
382+
383+
# OVN southbound logical-flow dump is cluster-global, so capture it once
384+
# rather than per pod. ovn-central is a Deployment (not per-node); any replica
385+
# answers. --no-leader-only lets a read land on a raft follower instead of
386+
# erroring.
387+
central=$(kubectl get pod -n "$KUBEOVN_NS" -l app=ovn-central \
388+
-o jsonpath='{.items[0].metadata.name}' 2>/dev/null)
389+
if [ -n "$central" ]; then
390+
{
391+
echo "=== ovn-sbctl lflow-list (cluster-global, pod=$central) ==="
392+
timeout 30 kubectl exec -n "$KUBEOVN_NS" "$central" -c ovn-central -- \
393+
ovn-sbctl --no-leader-only lflow-list 2>&1 || true
394+
} > "$OUT/ovn-lflows.txt" 2>&1 || true
395+
else
396+
log "no ovn-central pod in $KUBEOVN_NS -- skipping OVN logical-flow dump"
397+
fi
398+
399+
i=0
400+
printf '%s\n' "$affected" | {
388401
while IFS='|' read -r ns pod podip node _ready _phase; do
389-
[ -n "$ns" ] && [ -n "$pod" ] && [ -n "$podip" ] && [ -n "$node" ] || continue
402+
[ -n "$ns" ] && [ -n "$pod" ] && [ -n "$node" ] || continue
390403
i=$((i + 1))
391404
if [ "$i" -gt "$MAX_PODS" ]; then
392405
log "reached MAX_PODS=$MAX_PODS cap; $((ncount - MAX_PODS)) more affected pod(s) NOT captured"
@@ -403,7 +416,7 @@ printf '%s\n' "$affected" | {
403416
ovs=$(pod_on_node "$KUBEOVN_NS" app=ovs "$node")
404417
{
405418
echo "################################################################"
406-
echo "# POD $ns/$pod podIP=$podip node=$node (Ready=$_ready)"
419+
echo "# POD $ns/$pod podIP=${podip:-<none>} node=$node (Ready=$_ready)"
407420
echo "# node-global captures are in node-$node.txt"
408421
echo "################################################################"
409422

@@ -414,15 +427,21 @@ printf '%s\n' "$affected" | {
414427
kubectl get events -n "$ns" --field-selector "involvedObject.name=$pod" \
415428
-o jsonpath='{range .items[*]}{.lastTimestamp}{" "}{.reason}{": "}{.message}{"\n"}{end}' 2>&1 || true
416429

417-
if [ -n "$agent" ]; then
430+
if [ -z "$podip" ]; then
431+
echo
432+
echo "=== pod has no podIP; IP-specific route/conntrack capture skipped ==="
433+
echo "Node-global Cilium/OVN state and pod allocation events remain captured."
434+
fi
435+
436+
if [ -n "$podip" ] && [ -n "$agent" ]; then
418437
echo
419438
echo "=== cilium-dbg bpf ct list global | grep $podip (node=$node agent=$agent) ==="
420439
timeout 25 kubectl exec -n "$CILIUM_NS" "$agent" -c cilium-agent -- \
421440
sh -c "cilium-dbg bpf ct list global 2>/dev/null | grep -F '$podip' || echo 'no CT entries for $podip'" 2>&1 || true
422441
fi
423442

424443
cni=$(pod_on_node "$KUBEOVN_NS" app=kube-ovn-cni "$node")
425-
if [ -n "$cni" ]; then
444+
if [ -n "$podip" ] && [ -n "$cni" ]; then
426445
echo
427446
echo "=== host netns: ip route get $podip (via kube-ovn cni-server) ==="
428447
timeout 15 kubectl exec -n "$KUBEOVN_NS" "$cni" -c cni-server -- \
@@ -471,7 +490,8 @@ printf '%s\n' "$affected" | {
471490
} > "$pf" 2>&1 || true
472491
done
473492
log "host->pod data-plane capture complete"
474-
}
493+
}
494+
fi
475495

476496
# ============================================================================ #
477497
# LoadBalancer-datapath capture (see the header block for the full rationale). #

‎hack/e2e-chainsaw/bucket/chainsaw-test.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,7 @@ spec:
2323
- get:
2424
apiVersion: objectstorage.k8s.io/v1alpha1
2525
kind: Bucket
26+
format: yaml
2627
- script:
2728
content: kubectl -n cozy-objectstorage-controller get pods
2829
steps:

‎hack/e2e-chainsaw/harbor/chainsaw-test.yaml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -31,6 +31,7 @@ spec:
3131
- get:
3232
apiVersion: objectstorage.k8s.io/v1alpha1
3333
kind: Bucket
34+
format: yaml
3435
- script:
3536
content: kubectl -n cozy-objectstorage-controller get pods
3637
- podLogs:

0 commit comments

Comments
 (0)