Skip to content

fix(viewer): self-heal a headless-boot Wayland display wedge - #3155

Merged
vpetersson merged 4 commits into
masterfrom
fix/pi5-cage-headless-boot-display-wedge
Jul 8, 2026
Merged

fix(viewer): self-heal a headless-boot Wayland display wedge#3155
vpetersson merged 4 commits into
masterfrom
fix/pi5-cage-headless-boot-display-wedge

Conversation

@vpetersson

Copy link
Copy Markdown
Contributor

Problem

On cage/Wayland boards (x86, arm64, Pi 5), if the viewer boots with no display connected (headless boot, display powered off, or a slow HDMI sync), cage enumerates DRM connectors once, binds zero wl_outputs, and then never recovers when the display later appears — the screen stays black indefinitely. The viewer scheduler keeps cycling assets into an unbound surface; only a container restart brings the display back.

Root cause: the viewer container has no udev/udevd, so wlroots' libudev hotplug monitor never receives the "HDMI connected" uevent. cage tolerates zero outputs and runs headless forever. eglfs/linuxfb boards get this recovery for free (Qt exits non-zero when there's no framebuffer, and restart: always recreates the container, which re-enumerates DRM) — cage does not.

Found while investigating the webpage-transition flash report (issue 2954); this is a separate display-availability bug, not the flash itself.

Fix

1. Output watchdog (src/anthias_viewer/__init__.py) — runs each asset_loop tick. On a Wayland board, if cage has no wl_output while the kernel reports a bindable display for longer than a 60 s grace, it sys.exit(1)s so Docker recreates the container and cage re-enumerates DRM. SystemExit subclasses BaseException, so it propagates cleanly past __main__'s except Exception.

  • Bindable = connector status is connected and it has a non-empty EDID mode list. Gating on modes (not bare connected) is load-bearing: a genuinely headless unit (nothing plugged in) or a marginally-seated cable (HPD asserted but EDID unread) must not restart-loop, since a restart can't conjure a mode the kernel never read.
  • The 60 s persistence window ignores transient wlr-randr hiccups and displays that are merely slow to sync.

2. Reliable recovery (bin/start_viewer.sh, cage branch) — the restart-policy relaunch alone is racy on Pi 5 (vc4): the fresh cage can grab the DRM device before the old cage's master release and the HDMI connector reset drain, so its modeset races a half-reset connector, EDID isn't re-read, and cage comes up headless again. (A full docker restart recovers reliably because its stop→start gap lets the controller settle.) Before launching cage, when a display is connected, we now settle briefly then force a connector re-probe and wait for its mode list to repopulate — so cage always starts against a settled, EDID-populated connector.

Validation (Pi 5 testbed)

  • Reproduced the wedge deterministically via the DRM connector-force knob (/sys/class/drm/<c>/status off/detect) — no cable involved.
  • Full self-heal driven by the real watchdog at the committed 60 s grace: arm → sys.exit → container restart → start_viewer re-probe → cage binds → renders at 3840×2160.
  • Recovery reliability: the settle + re-probe took the restart from ~50 % to 12/12 across trials.
  • Correctly a no-op while genuinely headless (restarts=0, no loop) and while a cable is connected-but-no-EDID.

Testing

  • 11 new unit tests for the watchdog gating and the bindable-display check; full viewer suite (123) green.
  • ruff check + ruff format --check clean; bash -n clean.

Scoped to the cage branch (x86/arm64/pi5) since the no-udev wedge is architectural to all cage boards; hardware-validated on Pi 5.

🤖 Generated with Claude Code

On cage/Wayland boards (x86, arm64, Pi 5) the compositor enumerates DRM
connectors once at startup. If it boots with no connected display
(headless boot, display off, slow HDMI sync) it binds zero wl_outputs
and never recovers when the display later appears: the viewer container
has no udev to deliver the hotplug uevent to wlroots, so the screen
stays black until the container is restarted. eglfs/linuxfb boards
self-heal for free (Qt exits non-zero on no framebuffer and the restart
policy recreates the container); cage tolerates zero outputs and runs
headless forever.

Add an output watchdog to the viewer's asset loop (_wayland_output_
watchdog): on a Wayland board, if cage has no wl_output while the kernel
reports a bindable display (connector "connected" with a non-empty EDID
mode list) for longer than a 60s grace, exit non-zero so Docker
recreates the container and cage re-enumerates DRM. SystemExit
propagates past __main__'s "except Exception" to a clean exit. Gating on
EDID modes (not bare "connected") avoids restart-looping a genuinely
headless unit or a marginally-seated cable that asserts hotplug-detect
but never delivers EDID.

The restart-policy relaunch alone is racy on Pi 5 (vc4): the fresh cage
can grab the DRM device before the old cage's master release and the
HDMI connector reset drain, so its modeset races a half-reset connector,
EDID is not re-read, and cage comes up headless again. A full
"docker restart" recovers reliably because its stop->start gap lets the
controller settle. Reproduce that gap in start_viewer.sh's cage branch:
before launching cage, when a display is connected, settle briefly then
force a connector re-probe and wait for its mode list to repopulate, so
cage always starts against a settled, EDID-populated connector.

Validated on a Pi 5: the watchdog arm -> exit -> restart -> re-probe ->
bind chain recovers the display end-to-end, and the settle + re-probe
took recovery from ~50% to 12/12 across restart trials. Adds unit
coverage for the watchdog gating and the bindable-display check.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@vpetersson
vpetersson requested a review from a team as a code owner July 8, 2026 10:46
@vpetersson
vpetersson requested a review from Copilot July 8, 2026 10:46

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a self-healing mechanism for Wayland/cage viewer deployments that can wedge permanently after a headless boot (cage starts with zero wl_outputs and never recovers when a display later appears), by forcing a controlled restart and making the subsequent cage startup more robust against connector/EDID races.

Changes:

  • Add a Wayland-only output watchdog (_wayland_output_watchdog) that exits non-zero after a grace window when the kernel reports a bindable display but cage still has zero outputs.
  • Add a cage-branch startup settle + connector re-probe step in start_viewer.sh to improve restart recovery reliability (notably on Pi 5/vc4).
  • Add unit tests covering watchdog behavior and the “bindable display” sysfs gating logic.

Reviewed changes

Copilot reviewed 2 out of 3 changed files in this pull request and generated 1 comment.

File Description
src/anthias_viewer/__init__.py Introduces sysfs-based “bindable display” detection and a Wayland output watchdog invoked from asset_loop.
bin/start_viewer.sh Adds a pre-cage settle + connector re-probe/wait loop to reduce restart races where EDID/modes are temporarily empty.
tests/test_viewer.py Adds fixtures and tests for the watchdog timer behavior and bindable-display detection.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/anthias_viewer/__init__.py
- _kernel_has_bindable_display() now treats any connector status other
  than "disconnected" (or empty/unreadable) as present, still gated by a
  non-empty EDID modes list — mirroring bin/start_viewer.sh's
  eglfs_has_display(). Some HDMI bridges report "unknown" for a real
  display; keying off "== connected" alone would miss them and stop the
  watchdog from ever self-healing (Copilot review). The modes gate still
  excludes the Writeback connector (unknown status, always modeless).
- Assert the wedge timer with pytest.approx instead of raw float equality
  (SonarCloud reliability bug S1244).

Adds a unit test for the unknown-status-with-modes case.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 3 changed files in this pull request and generated no new comments.

…e review)

Two fixes from a high-effort review of the headless-boot self-heal:

- The watchdog treated an empty _wlr_output_names() as "cage has no
  output", but that helper also returns [] when wlr-randr is missing,
  exits nonzero, or hits its 5s timeout (e.g. under sustained
  QtWebEngine GPU load). A tooling failure on a healthy, displaying
  board could therefore accumulate the 60s grace and sys.exit(1),
  restarting a fine display. Add _cage_output_probe() returning
  has-output / no-output / unknown; the watchdog only arms on a
  definitive no-output and degrades to a no-op on unknown, matching how
  the rotation/power paths tolerate the same failure.

- The start_viewer.sh vc4 settle/re-probe forced an "echo detect" EDID
  re-probe on every connected connector on every cage launch, not just
  wedge-restart — needlessly re-negotiating and potentially flashing a
  healthy display on an ordinary boot, and wasting the 5s modes-wait on
  a marginal connector. Gate the re-probe on connected-but-empty-modes
  (the fast-restart EDID-drop symptom); a healthy connector already has
  its modes and is left untouched. The settle sleep still runs so the
  timing fix is unchanged.

Re-validated on a Pi 5: recovery reliability stays 8/8 with the
conditional re-probe, and a healthy boot now logs no re-probe. Adds unit
coverage for the probe classifier and the unknown-failure no-op.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@vpetersson
vpetersson force-pushed the fix/pi5-cage-headless-boot-display-wedge branch from 47d2d5b to 55d187a Compare July 8, 2026 14:05

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 3 changed files in this pull request and generated 1 comment.

Comment thread src/anthias_viewer/__init__.py Outdated
Reorder _wayland_output_watchdog so the cheap _kernel_has_bindable_
display() sysfs check runs before the _cage_output_probe() wlr-randr
call. A genuinely headless board (no connected connector, or a
connected-but-no-EDID cable) now returns early and never spawns
wlr-randr on the asset_loop hot path, nor risks its up-to-5s block per
tick — the subprocess only runs when a display is actually attached.

Every arm/exit decision is unchanged; this only avoids the per-tick
fork/exec when there is nothing to recover. Strengthens the headless-unit
test to assert the probe is not called (Copilot review).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@sonarqubecloud

sonarqubecloud Bot commented Jul 8, 2026

Copy link
Copy Markdown

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 3 changed files in this pull request and generated no new comments.

@vpetersson
vpetersson merged commit eab68d8 into master Jul 8, 2026
10 checks passed
@vpetersson
vpetersson deleted the fix/pi5-cage-headless-boot-display-wedge branch July 8, 2026 15:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants