get_display_power still SIGKILLs the celery worker despite the soft-limit guard #3264

Summary

get_display_power still blows its 60s celery hard limit on Pi boards, SIGKILLing the worker roughly every 5 minutes on affected devices. The two existing mitigations are already deployed on builds that are still throwing, so this is not an un-upgraded-device problem.

Sentry: ANTHIAS-A (259 events, "Hard time limit (60s) exceeded for anthias_server.celery_tasks.get_display_power"), ANTHIAS-9 (240, TimeLimitExceeded), ANTHIAS-B (379, ForkPoolWorker exited with signal 9 (SIGKILL)), ANTHIAS-31 (9, WorkerLostError, fatal). One causal chain, ~890 events. All four left unresolved deliberately in the 2026-08-07 triage sweep.

Why this is not already fixed

Both existing mitigations are present in builds that continue to throw:

2026.7.4+7fe850c (2026-07-28) is among the releases these issues are seen on, and #3063's own code comment cites this exact issue group as what it was meant to fix. So the mitigations are insufficient, not merely undeployed.

Affected: device_type pi3, pi3-64, pi4-64; kernel_machine aarch64 and armv7l; both balena=true and false. Not x86 (no CEC adapter, so the cec_available() gate short-circuits there).

Hypothesis (needs reproduction before fixing)

diagnostics.get_display_power() bounds the CEC query with:

result = subprocess.run(
    [sys.executable, '-c', _CEC_QUERY_SCRIPT],
    capture_output=True, timeout=10, check=False,
)

On timeout, subprocess.run kills the direct child — but the follow-up pipe reap has no timeout of its own. If a surviving libcec thread still holds the inherited stdout/stderr, that read can block indefinitely. A 10s subprocess timeout that nonetheless overruns a 60s hard limit fits the symptom.

The SoftTimeLimitExceeded handler in celery_tasks.py cannot help if the interpreter never regains control to raise it.

Impact

The display_power beat runs every 5 minutes (60 * 5). An affected board therefore loses its celery worker repeatedly, and the worker owns asset normalisation, YouTube downloads and the cleanup sweep — so a wedged CEC adapter silently degrades unrelated functionality.

Suggested direction

Make the child unkillable-proof rather than relying on celery's limits:

• pass stdout/stderr to a temp file or use start_new_session=True so the parent never blocks on a pipe a grandchild can hold open;
• on TimeoutExpired, escalate to SIGKILL on the whole process group and reap with a bounded wait;
• consider dropping the CEC poll to a longer interval, or caching a "this adapter is unresponsive" negative result with a TTL so a wedged adapter is not re-probed every 5 minutes.

Verification

Reproduce on a Pi with a CEC adapter present but no responsive TV (cec_available() true, libcec blocking). Confirm the task returns within the soft limit and the worker is never SIGKILLed. Then confirm the Sentry group stops appearing on builds containing the fix.