Repository navigation
Commit 4a96635
committed
test(e2e): capture per-node network diagnostics in cozyreport
Follows #3209 (LINSTOR ErrorReports) with the second half of the
"cozyreport is blind to the host-networking path" gap surfaced by
the PR #3044 vlselect flake investigation (run 28772259934):
`vlselect-generic-*` on one satellite hit `dial tcp <podIP>:9471:
connect: connection refused` on every livenessProbe for 49s and got
CrashLoopBackOff, while the sibling replica on another satellite —
same pod spec, same probe config, same image — ran clean. The
cozyreport artefact carried the CiliumEndpoint `state: ready` line,
the pod-side listener log confirming the HTTP server was bound at
0.0.0.0:9471 within 5 ms of start, and the kube-ovn CNI ADD trace
completing for the pod IP well before the first probe — every one
of which says the path SHOULD have worked. Attribution of the drop
to a specific host-side cause (stale conntrack from a previous pod
that shared the IP, a Cilium policy drop that only Hubble sees, or
an OVS-flow race between the CNI ADD and the iptables NAT chain)
was not possible from the artefact because none of the host-side
networking state is captured. Talos-side kubelet logs come out as
one-line stubs (`error constructing client: failed to determine
endpoints` — a separate cozyreport bug), so kubelet-view evidence
is unavailable too.
Add a `net-diag` section that runs after the sandbox-host block and
copies four artefacts per satellite node:
- `<node>-iptables.txt` — `iptables-save` from `kube-ovn-cni` (all
tables in one blob so grepping across chains does not need a
per-table unpack step).
- `<node>-conntrack.txt` — `conntrack -L` first 5000 lines, from the
same pod; the head cap keeps the file small on nodes with heavy
tenant traffic and preserves the failure-window entries on the
first-in-first-out flush semantics of the kernel table.
- `<node>-hubble-flows.txt` — `hubble observe --last 2000 --output=compact`
from `cilium-agent`. Hubble flow log carries the verdict
(FORWARDED / DROPPED / ERROR) with reason field per flow, which is
the only source that tells apart a Cilium policy drop from a
datapath race.
- `<node>-cilium-endpoints.txt` — `cilium-dbg endpoint list` for the
per-endpoint policy state (identity, enforcement, controllers,
status) that resolves "was the drop a policy decision" once hubble
flags one.
Every collection call uses a DaemonSet pod that already exists on the
node with hostNetwork and the required binary preinstalled:
`iptables-save` + `conntrack` live in `kube-ovn-cni`'s cni-server
container; `hubble observe` + `cilium-dbg` live in the `cilium-agent`
container. Both label selectors were confirmed against a live
production cluster; both tool paths were confirmed present in the
current piraeus / cilium image tags. Every capture is guarded with
`|| true` and stderr redirection so a broken pod / missing tool /
transient exec failure never fails the surrounding cozyreport run,
and the parent directory is pruned when neither CNI is present on
the cluster (e.g. a cozyreport captured against a non-cozystack
cluster).
Shipped as diagnostic-only test infra; no product-side change, no
schema change, no chart change.
Signed-off-by: Ivan Okhotnikov <[email protected]>1 parent f101426 commit 4a96635
1 file changed
Lines changed: 129 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
348 | 348 | | |
349 | 349 | | |
350 | 350 | | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
| 416 | + | |
| 417 | + | |
| 418 | + | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
| 438 | + | |
| 439 | + | |
| 440 | + | |
| 441 | + | |
| 442 | + | |
| 443 | + | |
| 444 | + | |
| 445 | + | |
| 446 | + | |
| 447 | + | |
| 448 | + | |
| 449 | + | |
| 450 | + | |
| 451 | + | |
| 452 | + | |
| 453 | + | |
| 454 | + | |
| 455 | + | |
| 456 | + | |
| 457 | + | |
| 458 | + | |
| 459 | + | |
| 460 | + | |
| 461 | + | |
| 462 | + | |
| 463 | + | |
| 464 | + | |
| 465 | + | |
| 466 | + | |
| 467 | + | |
| 468 | + | |
| 469 | + | |
| 470 | + | |
| 471 | + | |
| 472 | + | |
| 473 | + | |
| 474 | + | |
| 475 | + | |
| 476 | + | |
| 477 | + | |
| 478 | + | |
| 479 | + | |
351 | 480 | | |
352 | 481 | | |
353 | 482 | | |
| |||
0 commit comments