Skip to content

fix: report a busy HEALTH_PORT instead of missing IPv6 - #661

Open
fjvbn2003 wants to merge 1 commit into
kiwigrid:masterfrom
fjvbn2003:fix-health-port-in-use
Open

fjvbn2003 wants to merge 1 commit into
kiwigrid:masterfrom
fjvbn2003:fix-health-port-in-use

Conversation

@fjvbn2003

Copy link
Copy Markdown

When HEALTH_PORT is already taken, _create_health_http_server() reports it as missing IPv6. That happens, for example, with two k8s-sidecar containers in one pod (Grafana's dashboards and datasources sidecars) that both use the default 8080. The dual-stack bind fails with EADDRINUSE, the except OSError branch logs IPv6 not available, falling back to IPv4 for the health server, and the IPv4 bind then fails in the same way. The health thread then dies with an uncaught traceback (OSError: [Errno 98] Address in use), while the sidecar keeps running without a health endpoint.

We saw this in the datasources sidecar of a Grafana pod (k8s-sidecar 2.10.1, Grafana chart 12.11.0, neither sidecar had HEALTH_PORT set). The pod does have IPv6, so the warning sent us looking in the wrong place. Newer Grafana charts give each sidecar its own HEALTH_PORT, but any pod with two sidecars on the default port still hits this.

This PR:

  • re-raises EADDRINUSE from the IPv6 attempt instead of falling back, because the port is busy and IPv4 would fail the same way;
  • adds the actual error to the fallback warning, for example IPv6 not available ([Errno 97] Address family not supported by protocol), ...;
  • in start_health_server(), catches the OSError and logs a single error instead of a traceback: Cannot start the health server on port 8080: [Errno 98] Address in use. Set HEALTH_PORT to a free port; each k8s-sidecar container in the same pod needs its own.

Behaviour is otherwise unchanged: the sidecar keeps running without a health endpoint, as before. If you would rather have it exit in this case (so the conflict shows up as a failing container instead of a log line), I can change that.

Tests: added test/unit/test_health_server.py (standard library only, like test_liveness.py). I ran python -m unittest discover -s test/unit -t test/unit inside the kiwigrid/k8s-sidecar:2.10.1 image (Python 3.15.0b2) with this branch's src/. All 16 tests pass. With main's src/, the new port-in-use test fails on the IPv6 not available warning. I also started start_health_server() with a busy HEALTH_PORT. On main it prints the misleading warning and two tracebacks; with this change it prints the one error line above.

This change was drafted with AI assistance and reviewed by me.

🤖 Generated with Claude Code

When HEALTH_PORT is already taken, for example by another k8s-sidecar
container in the same pod using the default 8080, the dual-stack bind
fails with EADDRINUSE. The health server treated every OSError as
"IPv6 not available", logged that warning, retried on IPv4, failed the
same way and the health thread died with a traceback.

Only fall back to IPv4 when the error is not EADDRINUSE, include the
error in the fallback warning, and log one error naming the port and
HEALTH_PORT when the server cannot start.

Signed-off-by: YoungjuKim <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant