Skip to content

fix(viewer): bound AnthiasViewer output instead of buffering it in RAM - #3147

Merged
vpetersson merged 7 commits into
masterfrom
fix/viewer-webview-stdout-accumulation
Jul 8, 2026
Merged

fix(viewer): bound AnthiasViewer output instead of buffering it in RAM#3147
vpetersson merged 7 commits into
masterfrom
fix/viewer-webview-stdout-accumulation

Conversation

@vpetersson

@vpetersson vpetersson commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Problem

Fixes #3138.

On a 2 GB Raspberry Pi 4 with large MP4s, moving to the next asset blanks the screen for minutes and the viewer can wedge until the next asset is forced in the UI.

Root cause: the viewer spawns AnthiasViewer via the sh library with _bg=True, _err_to_out=True and no output sink. sh therefore aggregates the process's entire merged stdout+stderr in the viewer process's RAM for the whole session, and nothing ever drains browser.process.stdout after the D-Bus handshake. When a file's audio makes ffmpeg's AAC decoder spam channel element 0.0 duplicate on every frame (245k lines / 203 MB in the reporter's attached debug log), that in-RAM buffer grows without bound. On a 2 GB board it exhausts memory and drives the box into swap — hence the multi-minute blank transitions and the "device unresponsive for ~5 min" reports. With debug_logging on, sh's own per-chunk DEBUG logging multiplies it into the 203 MB log.

Fix

Pass a bounded _BoundedWebviewOutput sink as _out. sh then streams each chunk to it instead of retaining the stream — and, importantly, stops aggregating .process.stdout — so we keep only the last 64 KiB, which is enough for the startup-handshake scan and any crash tail. The handshake scan and the launch-failure message now read the bounded sink; WEBVIEW_DEBUG reads it too.

Verification

On the real Pi 4 (arm64), inside the viewer container, exact shipped sh 2.3.0 — feeding a fixed 28.6 MB of decoder-style spam through the two spawn variants:

Scenario Subprocess output Viewer RSS delta sh aggregated stdout
current (no _out sink) 28.6 MB +192 MB 28.6 MB retained
this PR (_out bounded sink) 28.6 MB 0.0 MB 0.0 MB

Same result reproduced on x86 (23 MB → 670 MB RSS in 24 s of continuous spam under the old flags; flat 23 MB with the sink). The bounded probe left the board healthy (no wedge).

Unit tests: added a direct regression test that the sink discards old data (bounded) and preserves the handshake in its window; reworked the spawn/handshake tests to drive the new sink. Full non-integration suite: 1245 passed. ruff check + ruff format --check clean.

🤖 Generated with Claude Code

The viewer spawns AnthiasViewer via sh with `_bg=True, _err_to_out=True`
and no output sink, so sh aggregates the process's entire merged
stdout+stderr in the viewer's RAM for the whole session and nothing
drains it after the D-Bus handshake. When a file's audio makes ffmpeg's
AAC decoder spam `channel element 0.0 duplicate` on every frame, that
buffer grows without bound. On a 2 GB Pi4 it exhausts RAM and drives the
box into swap, so asset transitions blank the screen for minutes and the
device can wedge until the next asset is forced.

Pass a bounded `_BoundedWebviewOutput` as `_out`: sh then streams each
chunk to it instead of retaining the stream (and stops aggregating
`.process.stdout`), and we keep only the last 64 KiB — enough for the
startup-handshake scan and a crash tail. The handshake scan and the
launch-failure message now read the sink; WEBVIEW_DEBUG reads it too.

Verified on hardware with the shipped sh 2.3.0: the old spawn flags grew
a Python process from 23 MB to 670 MB RSS in 24 s of decoder-style spam,
while an `_out` sink held RSS flat at 23 MB.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@vpetersson
vpetersson requested a review from a team as a code owner July 8, 2026 07:31
@vpetersson
vpetersson requested a review from Copilot July 8, 2026 07:31

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR addresses viewer memory exhaustion on low-memory devices by preventing sh from buffering AnthiasViewer’s merged stdout/stderr indefinitely in RAM (issue #3138). It introduces a bounded output sink that retains only the most recent output needed for handshake detection and crash diagnostics.

Changes:

  • Add _BoundedWebviewOutput and pass it as sh’s _out sink to avoid unbounded stdout aggregation.
  • Switch handshake detection and launch-failure diagnostics from candidate.process.stdout to the bounded sink.
  • Update and extend unit tests to drive/capture the _out sink and add bounded-buffer regression coverage.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
src/anthias_viewer/__init__.py Adds bounded output sink and routes handshake/debug output through it instead of sh’s accumulated stdout buffer.
tests/test_viewer.py Refactors spawn/handshake tests to capture the _out sink and adds tests for bounded-buffer behavior.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread src/anthias_viewer/__init__.py Outdated
Comment thread src/anthias_viewer/__init__.py
Address review feedback on the issue #3138 fix.

- _BoundedWebviewOutput.__call__ now coerces bytes to str. sh hands the
  _out callback str under the default encoding (verified with the
  shipped sh 2.3.0 on a real Pi4), so this is defensive: an
  _encoding=None / raw-bytes configuration would otherwise turn
  `str + bytes` into a TypeError that silently breaks the handshake scan.
- Publish `_webview_output = output` at the start of _spawn_webview_once
  instead of only on handshake success, so the module global truly
  mirrors `browser` for the live attempt and a WEBVIEW_DEBUG read during
  launch/retry can't surface a previous process's stale tail.
- Add a regression test for the bytes coercion.

Validated on the 1 GB Pi4 (.143) viewer container with the shipped sh
2.3.0: the current no-_out flags grow the process RSS +76 MB for 12 MB
of decoder-style spam (sh retains all 12 MB), while the bounded sink
holds RSS flat (+0.5 MB) with 0 bytes aggregated and still detects the
handshake.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Comment thread src/anthias_viewer/__init__.py Outdated
…hunk

Address further review feedback on the issue #3138 fix.

_BoundedWebviewOutput.__call__ appended the whole incoming chunk before
trimming, so a single chunk larger than the window (a big unbuffered
write from the child) would force a `buf + chunk` allocation
proportional to the chunk — spiking past maxlen, the very thing the sink
exists to prevent. Slice an oversized chunk to its tail first (identical
tail-retention result); the retained buffer and the transient now both
stay within ~2x maxlen regardless of chunk size. Add a regression test.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Comment thread src/anthias_viewer/__init__.py Outdated
…iced string

Address further review feedback on the issue #3138 fix.

`self._buf += chunk` followed by slicing to `[-maxlen:]` re-copied ~maxlen
characters on every call once the window was full. A chatty decoder calls
the sink per audio frame for the life of the process, so that is steady
CPU burn on a weak Pi — against the spirit of a fix meant to be gentle on
the box.

Retain a deque of raw chunks instead: append and drop whole oldest chunks
(amortized O(1), no recopy of kept data), and join to a single string
only in text(), which is read rarely (the startup handshake poll, where
the buffer is still small, and WEBVIEW_DEBUG). Retained size stays within
[maxlen, maxlen + newest-dropped-chunk) < 2x maxlen. Add a test asserting
the write path keeps a bounded, small chunk count under 10k writes.

Re-validated on the 1 GB Pi4 (.143) viewer container through the shipped
sh 2.3.0: handshake still detected, RSS flat (+0.4 MB) under 400k lines of
decoder spam, retained tail exactly 64 KiB, sh aggregates 0 bytes.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Comment thread tests/test_viewer.py
…DISPLAY

Address review feedback on the issue #3138 fix.

_spawn_webview_once calls _wait_for_wayland_socket before spawning the
command. On a Wayland desktop (WAYLAND_DISPLAY set to a socket that isn't
present) that wait calls the mocked sleep, firing feed_handshake before
the _out sink is captured — a KeyError on holder['sink'], or a spinning
socket-wait loop. CI/Docker leaves WAYLAND_DISPLAY unset so it passed
there, but the test was host-dependent.

Clear WAYLAND_DISPLAY so _wayland_socket_path() returns None and the
pre-spawn wait is a guaranteed no-op (sleep is then exercised only by the
handshake poll loop), and guard feed_handshake so a stray pre-spawn sleep
can never KeyError on an uncaptured sink. Verified the test now passes
with WAYLAND_DISPLAY pointed at a nonexistent socket.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

tests/test_viewer.py:252

  • These tests can become environment-dependent: if WAYLAND_DISPLAY/XDG_RUNTIME_DIR are set but the Wayland socket path is absent (common in some CI/dev containers), _spawn_webview_once() will spend the entire startup budget in _wait_for_wayland_socket(). That changes the control flow (timeout path) and can make this test hang or fail (e.g., calling terminate() unexpectedly). Clear WAYLAND_DISPLAY here (as done in test_load_browser) to keep the test hermetic.
def test_spawn_webview_once_raises_on_early_exit(
    viewer_fixtures: _ViewerFixtures,
) -> None:

Comment thread tests/test_viewer.py
Comment thread tests/test_viewer.py
Address review feedback on the issue #3138 fix.

The hermeticity fix from the previous commit covered only
test_load_browser, but every test that drives _spawn_webview_once /
load_browser has the same exposure: with sleep mocked, a set-but-dangling
WAYLAND_DISPLAY makes the pre-spawn _wait_for_wayland_socket busy-loop for
the whole startup budget (test_spawn_webview_once_raises_on_early_exit,
_terminates_on_timeout, load_browser retry tests). Clear WAYLAND_DISPLAY
once in the viewer_fixtures fixture so _wayland_socket_path() returns None
and the wait is a no-op for the whole module; drop the now-redundant
per-test delenv. The dedicated _wait_for_wayland_socket tests set their
own env and don't use this fixture, so they're unaffected.

Verified the full viewer suite passes with WAYLAND_DISPLAY pointed at a
nonexistent socket (previously several tests would spin for the budget).

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

…stdout-accumulation

# Conflicts:
#	src/anthias_viewer/__init__.py
@sonarqubecloud

sonarqubecloud Bot commented Jul 8, 2026

Copy link
Copy Markdown

@vpetersson
vpetersson merged commit c510d37 into master Jul 8, 2026
9 checks passed
@vpetersson
vpetersson deleted the fix/viewer-webview-stdout-accumulation branch July 8, 2026 13:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[BUG] Anthias Viewer - significant time including of blacking of screen when moving to next asset

2 participants