Skip to content

fix(viewer): recover Wayland headless-boot display wedge (bounded restarts) - #3240

Merged
vpetersson merged 2 commits into
Screenly:masterfrom
vpetersson-bot:fix/wayland-headless-recover-6758
Aug 4, 2026
Merged

fix(viewer): recover Wayland headless-boot display wedge (bounded restarts)#3240
vpetersson merged 2 commits into
Screenly:masterfrom
vpetersson-bot:fix/wayland-headless-recover-6758

Conversation

@vpetersson-bot

Copy link
Copy Markdown
Contributor

Fixes the forum reports of a Pi 5 / Bookworm display stuck on "No Signal" when the TV powers on after the Pi has booted (e.g. a TV on an external timer). Reproduced and fixed on the Pi 5 testbed.

Root cause (confirmed on hardware)

On a Wayland (cage) board, cage enumerates DRM once at startup. Boot with the display off and cage binds no output. When the display powers on later, the connector goes connected but stays enabled=disabled with an empty mode list — cage holds the output disabled, so the kernel never re-reads the EDID. The screen stays dark forever.

Anthias already has a self-heal watchdog for a headless-boot wedge, but it only armed when the connector had a populated mode list (_kernel_has_bindable_display). This wedge never has one, so the watchdog never fired. Watched it sit connected / enabled=disabled / modes=0 for 100s+ with no recovery.

Fix

The watchdog now also arms when a connector is strictly connected (a display is physically present) even with empty modes, via a new _kernel_has_connected_display. The container restart it already performs is what re-enumerates DRM and re-reads the EDID — so no privileged sysfs write is needed.

Why it works on Balena too: the recovery is entirely in-container and permission-free (it relies on sys.exit + the container restart policy, which the balena supervisor honours just like compose). balenaOS is immutable, so a host-side udev/systemd approach would not exist on Balena — this deliberately avoids that.

Crash-loop safety (the load-bearing part)

Recovery restarts are capped per device boot (WAYLAND_MAX_RECOVERY_RESTARTS = 3) via a counter persisted under the config dir, keyed on /proc/sys/kernel/random/boot_id:

  • A real display binds on the first restart → the counter clears.
  • A connector that can never bind (dead cable, hotplug-without-EDID) is retried a few times, then given up on — logged once, then quiet — so a failed display cannot churn the container (and warm the SoC) through an unattended 8-hour night.
  • A real reboot starts fresh (boot_id changes), so a stale give-up never blocks a good display.
  • If the counter cannot be written, the watchdog refuses to restart rather than risk an unbounded loop.
  • The Writeback connector (unknown + modeless) and a wlr-randr unknown probe are still left alone.

Validation

  • Unit tests: empty-modes wedge recovery, the restart cap (gives up, logs once), the unwritable-counter guard, boot-scoped counter reset, _kernel_has_connected_display (incl. Writeback exclusion). Full viewer suite: 172 passed; ruff + mypy clean.
  • On the Pi 5 testbed: unplugged the display, restarted the viewer so cage came up headless, replugged — the display recovered on its own ~70s later (counter recorded restart 1, then reset once it bound). No manual docker restart.

🤖 Generated with Claude Code

https://claude.ai/code/session_01HJ3ucEkn62cbgPoisAZ5LQ

On a Wayland (cage) board, booting with the display off leaves cage with
no bound output; when the display powers on later the connector goes
`connected` but stays `enabled=disabled` with an empty mode list — cage
holds the output disabled so the kernel never re-reads the EDID — and the
screen stays on "No Signal" indefinitely (forum reports of TVs on
external timers). The existing output watchdog never fired here because
it required a populated mode list (`_kernel_has_bindable_display`), which
the wedge never has. Confirmed on the Pi 5 testbed.

Recover it: the watchdog now also arms when a connector is strictly
`connected` (a display is physically present) even with empty modes, via
the new `_kernel_has_connected_display`. The container restart it already
performs is what re-enumerates DRM and re-reads the EDID, so no
privileged sysfs write is needed — which keeps this working identically
under docker-compose and Balena (no host udev/service, which balenaOS
can't take).

Crash-loop safety is the load-bearing part: recovery restarts are capped
per device boot (`WAYLAND_MAX_RECOVERY_RESTARTS`) via a counter persisted
under the config dir and keyed on `/proc/.../boot_id`. A real display
binds on the first restart and clears the count; a connector that can
never bind (dead cable, hotplug-without-EDID) is retried a few times then
given up on — logged once, then quiet — so a failed display can't churn
the container and warm the SoC through an unattended 8h. A real reboot
starts fresh; an unwritable counter refuses to restart rather than risk
an unbounded loop; the Writeback connector and wlr-randr "unknown" are
still left alone.

Validated on the Pi 5: unplug, restart viewer headless, replug -> the
display recovered on its own (counter recorded restart 1, then reset when
it bound), no manual intervention.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HJ3ucEkn62cbgPoisAZ5LQ
@codecov

codecov Bot commented Aug 4, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (master@2f4bb1a). Learn more about missing BASE report.

Additional details and impacted files
@@            Coverage Diff            @@
##             master    #3240   +/-   ##
=========================================
  Coverage          ?   90.85%           
=========================================
  Files             ?       76           
  Lines             ?     8361           
  Branches          ?      885           
=========================================
  Hits              ?     7596           
  Misses            ?      548           
  Partials          ?      217           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the Wayland/cage “headless-boot display wedge” watchdog in anthias_viewer so that it can recover when a display is physically connected but still reports an empty EDID mode list, while adding a per-device-boot restart cap to prevent crash-loop churn.

Changes:

  • Expand the watchdog’s “display present” detection to include strictly connected DRM connectors even when modes is empty.
  • Add a boot-scoped, persisted recovery-restart budget (cap) with “give up” logging to avoid unbounded restart loops.
  • Extend the viewer test suite to cover the new modeless-connected recovery case and the bounded-restart behavior.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
src/anthias_viewer/__init__.py Adds _kernel_has_connected_display and boot-scoped recovery-budget helpers; updates _wayland_output_watchdog to recover modeless-connected wedges with bounded restarts.
tests/test_viewer.py Adds unit tests and mocking helpers for the new watchdog recovery logic and its crash-loop safety budget.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/anthias_viewer/__init__.py Outdated
Comment thread src/anthias_viewer/__init__.py Outdated
Copilot review on Screenly#3240:
- _current_boot_id() returned '' on read failure, which would break the
  per-boot restart cap (the same '' persists across reboots, pinning a
  give-up). Fall back to /proc/stat btime, which also changes on reboot,
  so the cap still resets on a real reboot.
- Reword the unclear 'could not bound the attempt' comment.

Co-Authored-By: Claude Opus 4.8 <[email protected]>
Claude-Session: https://claude.ai/code/session_01HJ3ucEkn62cbgPoisAZ5LQ
@sonarqubecloud

sonarqubecloud Bot commented Aug 4, 2026

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants