fix(viewer): self-heal a headless-boot Wayland display wedge - #3155
Merged
Conversation
On cage/Wayland boards (x86, arm64, Pi 5) the compositor enumerates DRM connectors once at startup. If it boots with no connected display (headless boot, display off, slow HDMI sync) it binds zero wl_outputs and never recovers when the display later appears: the viewer container has no udev to deliver the hotplug uevent to wlroots, so the screen stays black until the container is restarted. eglfs/linuxfb boards self-heal for free (Qt exits non-zero on no framebuffer and the restart policy recreates the container); cage tolerates zero outputs and runs headless forever. Add an output watchdog to the viewer's asset loop (_wayland_output_ watchdog): on a Wayland board, if cage has no wl_output while the kernel reports a bindable display (connector "connected" with a non-empty EDID mode list) for longer than a 60s grace, exit non-zero so Docker recreates the container and cage re-enumerates DRM. SystemExit propagates past __main__'s "except Exception" to a clean exit. Gating on EDID modes (not bare "connected") avoids restart-looping a genuinely headless unit or a marginally-seated cable that asserts hotplug-detect but never delivers EDID. The restart-policy relaunch alone is racy on Pi 5 (vc4): the fresh cage can grab the DRM device before the old cage's master release and the HDMI connector reset drain, so its modeset races a half-reset connector, EDID is not re-read, and cage comes up headless again. A full "docker restart" recovers reliably because its stop->start gap lets the controller settle. Reproduce that gap in start_viewer.sh's cage branch: before launching cage, when a display is connected, settle briefly then force a connector re-probe and wait for its mode list to repopulate, so cage always starts against a settled, EDID-populated connector. Validated on a Pi 5: the watchdog arm -> exit -> restart -> re-probe -> bind chain recovers the display end-to-end, and the settle + re-probe took recovery from ~50% to 12/12 across restart trials. Adds unit coverage for the watchdog gating and the bindable-display check. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
There was a problem hiding this comment.
Pull request overview
Adds a self-healing mechanism for Wayland/cage viewer deployments that can wedge permanently after a headless boot (cage starts with zero wl_outputs and never recovers when a display later appears), by forcing a controlled restart and making the subsequent cage startup more robust against connector/EDID races.
Changes:
- Add a Wayland-only output watchdog (
_wayland_output_watchdog) that exits non-zero after a grace window when the kernel reports a bindable display but cage still has zero outputs. - Add a cage-branch startup settle + connector re-probe step in
start_viewer.shto improve restart recovery reliability (notably on Pi 5/vc4). - Add unit tests covering watchdog behavior and the “bindable display” sysfs gating logic.
Reviewed changes
Copilot reviewed 2 out of 3 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
src/anthias_viewer/__init__.py |
Introduces sysfs-based “bindable display” detection and a Wayland output watchdog invoked from asset_loop. |
bin/start_viewer.sh |
Adds a pre-cage settle + connector re-probe/wait loop to reduce restart races where EDID/modes are temporarily empty. |
tests/test_viewer.py |
Adds fixtures and tests for the watchdog timer behavior and bindable-display detection. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- _kernel_has_bindable_display() now treats any connector status other than "disconnected" (or empty/unreadable) as present, still gated by a non-empty EDID modes list — mirroring bin/start_viewer.sh's eglfs_has_display(). Some HDMI bridges report "unknown" for a real display; keying off "== connected" alone would miss them and stop the watchdog from ever self-healing (Copilot review). The modes gate still excludes the Writeback connector (unknown status, always modeless). - Assert the wedge timer with pytest.approx instead of raw float equality (SonarCloud reliability bug S1244). Adds a unit test for the unknown-status-with-modes case. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…e review) Two fixes from a high-effort review of the headless-boot self-heal: - The watchdog treated an empty _wlr_output_names() as "cage has no output", but that helper also returns [] when wlr-randr is missing, exits nonzero, or hits its 5s timeout (e.g. under sustained QtWebEngine GPU load). A tooling failure on a healthy, displaying board could therefore accumulate the 60s grace and sys.exit(1), restarting a fine display. Add _cage_output_probe() returning has-output / no-output / unknown; the watchdog only arms on a definitive no-output and degrades to a no-op on unknown, matching how the rotation/power paths tolerate the same failure. - The start_viewer.sh vc4 settle/re-probe forced an "echo detect" EDID re-probe on every connected connector on every cage launch, not just wedge-restart — needlessly re-negotiating and potentially flashing a healthy display on an ordinary boot, and wasting the 5s modes-wait on a marginal connector. Gate the re-probe on connected-but-empty-modes (the fast-restart EDID-drop symptom); a healthy connector already has its modes and is left untouched. The settle sleep still runs so the timing fix is unchanged. Re-validated on a Pi 5: recovery reliability stays 8/8 with the conditional re-probe, and a healthy boot now logs no re-probe. Adds unit coverage for the probe classifier and the unknown-failure no-op. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
vpetersson
force-pushed
the
fix/pi5-cage-headless-boot-display-wedge
branch
from
July 8, 2026 14:05
47d2d5b to
55d187a
Compare
Reorder _wayland_output_watchdog so the cheap _kernel_has_bindable_ display() sysfs check runs before the _cage_output_probe() wlr-randr call. A genuinely headless board (no connected connector, or a connected-but-no-EDID cable) now returns early and never spawns wlr-randr on the asset_loop hot path, nor risks its up-to-5s block per tick — the subprocess only runs when a display is actually attached. Every arm/exit decision is unchanged; this only avoids the per-tick fork/exec when there is nothing to recover. Strengthens the headless-unit test to assert the probe is not called (Copilot review). Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Problem
On cage/Wayland boards (x86, arm64, Pi 5), if the viewer boots with no display connected (headless boot, display powered off, or a slow HDMI sync),
cageenumerates DRM connectors once, binds zerowl_outputs, and then never recovers when the display later appears — the screen stays black indefinitely. The viewer scheduler keeps cycling assets into an unbound surface; only a container restart brings the display back.Root cause: the viewer container has no
udev/udevd, so wlroots' libudev hotplug monitor never receives the "HDMI connected" uevent.cagetolerates zero outputs and runs headless forever. eglfs/linuxfb boards get this recovery for free (Qt exits non-zero when there's no framebuffer, andrestart: alwaysrecreates the container, which re-enumerates DRM) — cage does not.Found while investigating the webpage-transition flash report (issue 2954); this is a separate display-availability bug, not the flash itself.
Fix
1. Output watchdog (
src/anthias_viewer/__init__.py) — runs eachasset_looptick. On a Wayland board, if cage has nowl_outputwhile the kernel reports a bindable display for longer than a 60 s grace, itsys.exit(1)s so Docker recreates the container and cage re-enumerates DRM.SystemExitsubclassesBaseException, so it propagates cleanly past__main__'sexcept Exception.statusisconnectedand it has a non-empty EDID mode list. Gating on modes (not bareconnected) is load-bearing: a genuinely headless unit (nothing plugged in) or a marginally-seated cable (HPD asserted but EDID unread) must not restart-loop, since a restart can't conjure a mode the kernel never read.2. Reliable recovery (
bin/start_viewer.sh, cage branch) — the restart-policy relaunch alone is racy on Pi 5 (vc4): the fresh cage can grab the DRM device before the old cage's master release and the HDMI connector reset drain, so its modeset races a half-reset connector, EDID isn't re-read, and cage comes up headless again. (A fulldocker restartrecovers reliably because its stop→start gap lets the controller settle.) Before launching cage, when a display is connected, we now settle briefly then force a connector re-probe and wait for its mode list to repopulate — so cage always starts against a settled, EDID-populated connector.Validation (Pi 5 testbed)
/sys/class/drm/<c>/statusoff/detect) — no cable involved.sys.exit→ container restart →start_viewerre-probe → cage binds → renders at 3840×2160.restarts=0, no loop) and while a cable is connected-but-no-EDID.Testing
ruff check+ruff format --checkclean;bash -nclean.Scoped to the cage branch (x86/arm64/pi5) since the no-udev wedge is architectural to all cage boards; hardware-validated on Pi 5.
🤖 Generated with Claude Code