Repository navigation
e2e: a failed service-list read makes the LB-datapath capture claim the cluster has no LoadBalancers #3649
Description
Summary
In hack/e2e-capture-dataplane.sh, capture_lb_datapath derives its service list from kubectl get svc -A with stderr discarded and no exit-status check (the _raw=$(...) at the top of the function). When that read fails, for any reason from a refused connection to the bound firing, _raw is empty and the function logs "no Service type=LoadBalancer with an ingress IP -- skipping LB-datapath capture". That is a statement about the cluster the script never observed: a failed read and a cluster with no LoadBalancers produce the identical line, and the reader has no way to tell which one happened.
Why it matters
The LB-datapath section exists for exactly the runs where the apiserver is struggling, which is also when the list read is most likely to fail. On those runs the capture reports a clean-sounding "nothing to probe" and skips the whole section, so the diagnostic is at its weakest precisely on its target scenario. This is the same defect class as the reads that were recently given cut-off notes in the sibling collectors: an empty result is not evidence of an empty cluster.
Fix shape
Check the read's exit status before interpreting emptiness: on failure, log the real outcome (exit code plus kubectl's own message, in the discipline the previous-logs collector already uses) and still skip the section, so the skip names its true cause. Reproduction is one stub away: a kubectl that prints "connection refused" and exits 1 currently yields the "no Service" line.
Activity
lexfrei commentedon Aug 8, 2026
Scope note: the same shape covers all three reads in the LB section, not just the service list.
_epsand_eptport(the endpointslice reads at lines 635 and 644 on main) also discard stderr and are interpreted by emptiness alone, so a failed read downgrades an LB probe to "no endpoints" silently. Whoever fixes this issue should treat the three as one class with one discipline.lexfrei commentedon Aug 8, 2026
Second scope note, a landmine rather than a live defect: the
_eptportread usesjsonpath='{.items[0].ports[0].port}', and client-go'sevalArrayhard-errors on an empty list (array index out of bounds), with noallowMissingKeysescape, unlikeevalField. Today the swallowed status makes that invisible; the moment this read gets the failure-reporting discipline this issue asks for, an ordinary empty list starts reporting as a failed read. Whoever fixes this should switch to{.items[*]...}(empty list yields empty output and exit 0) in the same change, so a real failure and a legitimately empty answer stay distinguishable.lexfrei commentedon Aug 8, 2026
Scope update. The in-flight branch for #3642 now closes the log-level half of this class for all three "absence" lines: each one ("no scheduled NotReady pods", "no ovn-central pod", "no Service type=LoadBalancer with an ingress IP") is gated on its source read's exit status and has an "did not answer / unknown" counterpart, verified by enumeration rather than by a round not finding a fourth.
What remains for this issue is the generalized ARTIFACT-side half: absence claims written into the artifact files from reads whose failure is only visible in the job log. Concretely,
node-<node>.txtstill records "(no cilium-agent pod found on node X)" when the pod lookup failed rather than answered empty, and the endpointslice reads write<none>for values they could not read. A reader working from the artifact alone gets the assertion and never sees the refutation. Fixing this means threading the read status into what gets written to the files, which is an artifact-format change and deliberately out of scope for the bounding branch.lexfrei commentedon Aug 8, 2026
Correction to my previous scope note: the
node-<node>.txthalf turned out NOT to be pre-existing, and it is now fixed in the bounding branch rather than left here. Measured on both sides with the same input: on the merge base the unbounded per-node lookup hangs until the outer backstop kills the whole script, so no artifact is written at all; with per-call bounds the script survives and writes "(no cilium-agent pod found on node X)" about a pod nobody got an answer on. The bound converted "no artifact" into "an artifact with a false claim", which makes it an interaction of that change, not inherited debt. The lesson for attribution: byte-identical source lines are not the test, what actually lands in the artifact under the same input is.What remains for this issue after that branch: the endpointslice reads (
_eps/_eptport), which still write<none>for values they could not read, with no status and no note.fix(e2e): stop the data-plane capture reporting what it never observed (
lexfrei commentedon Sep 23, 2026
Fixed on main. #3659 checks the exit status of the service list read and logs the unknown case. #3672 does the same for both endpointslice reads, which now write
<unknown>instead of<none>on failure.