Summary
The viewer emits ~1250 log lines per hour while doing nothing, and in production those lines land in a small volatile journald ring shared with everything else. On the low-RAM boards that means only ~8 hours of history survives — so the log around a crash is usually gone before anyone looks at the device.
Found while trying to establish whether #3266 had ever occurred naturally on the Pi 2 testbed. It hadn't, but I could only prove that for the last ~8 hours of a 3-week uptime.
Measured
| Board |
RAM |
Journal retention |
Uptime |
| Pi 2 |
801 MB |
~8 h |
3 weeks |
| Pi 3-64 |
787 MB |
~9.5 h |
5 days |
| Pi 5 |
8 GB |
11 days |
4 weeks |
Retention tracks RAM because the journal is volatile (/run tmpfs, capped at ~10% of it): 19–24 journal files under /run/log/journal, zero under /var/log/journal. On the Pi 2 that is a ~15 MB ring on a 161 MB /run.
Viewer contribution, measured on an idle board with 0 enabled assets: 1253 lines in one hour — Playlist is empty. Sleeping for %s seconds plus Current url is ..., roughly two lines every 5 seconds (src/anthias_viewer/__init__.py:1468,1498,2192).
What is NOT the bug
To be explicit, because it is the tempting fix: the volatile journal is Raspberry Pi OS's own deliberate default, set in /usr/lib/systemd/journald.conf.d/40-rpi-volatile-storage.conf, and it exists to protect the SD card from journal write wear. Anthias does not set it and arguably should not override it — SD wear is a real field failure mode on signage, and trading card life for logs is a poor deal. systemd-journal-flush ran successfully; nothing is broken.
The fixable part is that Anthias spends that scarce ring on messages which carry no information.
Why it matters
Anthias uses the journald docker logging driver in production (see bin/enable_ssl.sh:205 and bin/collect_debug.sh), so every container's output competes for the same ~15 MB. The idle chatter is therefore actively evicting crash diagnostics, and it does so worst on the low-RAM boards — which are exactly where the crash volume is concentrated (#3266's live events are all pi3-64; #3264's affected set is pi3/pi3-64/pi4-64).
It also degrades bin/collect_debug.sh: a bundle collected the morning after an overnight failure will not contain the failure.
Suggested direction
- Stop logging the idle case every tick.
Playlist is empty and Current url is at 5-second cadence should be debug, or log-once-on-transition (log when the playlist becomes empty and when it stops being empty, not every loop). The steady state carries no information; the transition does.
- Audit the other per-tick
logger.info calls in the viewer loop on the same principle.
- Optionally cap the docker journald driver's share per service (
log-opts) so one chatty container cannot evict the others.
A rough sense of the win: removing ~1250 lines/hour of the ring's traffic should multiply retention on an 800 MB board severalfold, at zero cost to SD wear and with no change to the OS default.
Summary
The viewer emits ~1250 log lines per hour while doing nothing, and in production those lines land in a small volatile journald ring shared with everything else. On the low-RAM boards that means only ~8 hours of history survives — so the log around a crash is usually gone before anyone looks at the device.
Found while trying to establish whether #3266 had ever occurred naturally on the Pi 2 testbed. It hadn't, but I could only prove that for the last ~8 hours of a 3-week uptime.
Measured
Retention tracks RAM because the journal is volatile (
/runtmpfs, capped at ~10% of it): 19–24 journal files under/run/log/journal, zero under/var/log/journal. On the Pi 2 that is a ~15 MB ring on a 161 MB/run.Viewer contribution, measured on an idle board with 0 enabled assets: 1253 lines in one hour —
Playlist is empty. Sleeping for %s secondsplusCurrent url is ..., roughly two lines every 5 seconds (src/anthias_viewer/__init__.py:1468,1498,2192).What is NOT the bug
To be explicit, because it is the tempting fix: the volatile journal is Raspberry Pi OS's own deliberate default, set in
/usr/lib/systemd/journald.conf.d/40-rpi-volatile-storage.conf, and it exists to protect the SD card from journal write wear. Anthias does not set it and arguably should not override it — SD wear is a real field failure mode on signage, and trading card life for logs is a poor deal.systemd-journal-flushran successfully; nothing is broken.The fixable part is that Anthias spends that scarce ring on messages which carry no information.
Why it matters
Anthias uses the
journalddocker logging driver in production (seebin/enable_ssl.sh:205andbin/collect_debug.sh), so every container's output competes for the same ~15 MB. The idle chatter is therefore actively evicting crash diagnostics, and it does so worst on the low-RAM boards — which are exactly where the crash volume is concentrated (#3266's live events are all pi3-64; #3264's affected set is pi3/pi3-64/pi4-64).It also degrades
bin/collect_debug.sh: a bundle collected the morning after an overnight failure will not contain the failure.Suggested direction
Playlist is emptyandCurrent url isat 5-second cadence should bedebug, or log-once-on-transition (log when the playlist becomes empty and when it stops being empty, not every loop). The steady state carries no information; the transition does.logger.infocalls in the viewer loop on the same principle.log-opts) so one chatty container cannot evict the others.A rough sense of the win: removing ~1250 lines/hour of the ring's traffic should multiply retention on an 800 MB board severalfold, at zero cost to SD wear and with no change to the OS default.