pi3-64 hardware video playback (30fps, zero drops) + audio - #3164
Conversation
Fixes #3084 (pi3-64 video renders black). On the Raspberry Pi 3 arm64 board (VideoCore IV, eglfs) QtMultimedia's VideoOutput cannot present a video node on the vc4 GL driver — the frame is black — and compositing a full-screen video widget through eglfs GL caps at ~9 fps regardless. This adds a pi3-64-only path that keeps decode + scanout on the hardware and composites the video on a vc4 DRM overlay plane, reaching a stable 30 fps with zero ongoing dropped frames, with audio. Video (VideoView::gstPlayOverlay): - filesrc ! qtdemux ! h264parse ! v4l2h264dec (bcm2835 HW decoder) ! queue ! kmssink on a vc4 overlay plane, using eglfs's own DRM master fd + connector via QPlatformNativeInterface; the plane is HW-composited with the eglfs UI plane. - The decoder is driven capture-io-mode=mmap so kmssink COPIES each frame into its own scanout buffer instead of pinning the decoder's DPB pool — that pin deadlocked every plain decoder-direct attempt. The decoder emits I420 at full rate; the alternative ISP convert to NV12 is a second M2M pass that caps at ~18 fps, so it is only a fallback (ANTHIAS_GST_ISP_CONVERT=1). - Falls back to a CPU-raster appsink path, then to QMediaPlayer, if the DRM resources or an overlay plane can't be resolved. Audio (VideoView::gstStartAudio) runs in a SEPARATE pipeline: - filesrc ! qtdemux audio ! decodebin ! audioconvert ! audioresample ! alsasink on the HDMI card. Independent clock/preroll/bus so no audio condition (missing track, slow device, decode gap) can stall the video — a shared pipeline froze the video at the first frame. - Adds gstreamer1.0-libav for the audio decoders (avdec_ac3/aac/mp3/…); decodebin reported a missing plug-in on AC3 with only faad (AAC). Also: only pi3-64 links GStreamer (AnthiasViewer.pro CONFIG, Dockerfile build + runtime deps); _build_webview_env enables the path via ANTHIAS_VIDEO_RASTER/OVERLAY; start_viewer chowns the GStreamer registry cache to avoid a per-launch rescan; main.cpp installs a fatal-signal backtrace handler. Validated on a real Pi 3 (1080p30 H.264/AC3, HDMI): 30 fps, dropped flat at the startup ramp only, clean across the loop boundary, HDMI PCM RUNNING, respawns=1. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
There was a problem hiding this comment.
Pull request overview
This PR adds a Raspberry Pi 3 arm64–specific hardware video playback path to AnthiasViewer to avoid QtMultimedia/VideoOutput black-screening on pi3-64 (VideoCore IV + eglfs), and wires the runtime/build environment to enable that path (including a separate GStreamer audio pipeline).
Changes:
- Add a
pi3-64-only raster/overlay presentation path inVideoViewusing in-process GStreamer (DRM overlay viakmssink, with appsink→raster fallback). - Plumb
ANTHIAS_VIDEO_RASTER/ANTHIAS_VIDEO_OVERLAYfrom the Python viewer env forpi3-64, with unit tests. - Update
pi3-64build/runtime dependencies (GStreamer + libdrm) and viewer startup cache permissions.
Reviewed changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/test_viewer.py | Adds unit tests ensuring pi3-64 enables raster mode and other boards don’t inherit stale flags. |
| src/anthias_webview/src/videoview.h | Declares raster-mode presenter and GStreamer (overlay/appsink + separate audio) hooks and state. |
| src/anthias_webview/src/videoview.cpp | Implements raster widget presenter and the GStreamer overlay/appsink pipelines + stats instrumentation. |
| src/anthias_webview/src/main.cpp | Adds a fatal-signal backtrace handler (currently unconditional). |
| src/anthias_webview/AnthiasViewer.pro | Adds anthias_gstreamer build config (GStreamer/libdrm/gui-private + -rdynamic) for pi3-64. |
| src/anthias_viewer/init.py | Sets/clears ANTHIAS_VIDEO_RASTER + ANTHIAS_VIDEO_OVERLAY based on DEVICE_TYPE. |
| docker/Dockerfile.viewer.j2 | Installs pi3-64-only build/runtime deps and passes CONFIG+=anthias_gstreamer to qmake6. |
| bin/lib/viewer/common.sh | Creates/chowns a writable GStreamer registry cache to avoid per-launch plugin rescans. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
The RPi archive ships RPi-tuned +rpt3 builds of gstreamer plugins-base, plugins-bad and alsa (the archive + keyring are already configured by _rpt1-ffmpeg-pin.j2 — pi3-64 is in its board list). plugins-bad carries the vc4 kmssink the overlay-plane path scans out through, so the RPi build is the right one for production parity with Pi OS. plugins-good (the V4L2 HW decode/convert elements) and libav (audio decoders) aren't in the RPi archive, so they stay on Debian. The +rpt3 plugins share upstream 1.26.2 with Debian's libgstreamer1.0-0 core (not in the RPi archive), so the ABI matches. Validated on a real Pi 3 (clean reboot): base/bad/alsa install as 1.26.2*+rpt3*, good/libav as Debian; 30fps zero ongoing drops, audio playing, respawns=1 — unchanged from the Debian-plugins build. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Added: RPi
|
- main.cpp: scope the fatal-signal backtrace handler to the pi3-64 build (#ifdef ANTHIAS_GSTREAMER) instead of installing it fleet-wide. It was added to debug the pi3-64 overlay crashes; other boards keep the default crash behaviour. (It already re-raised via SIG_DFL, so core dumps were never suppressed — but this keeps the PR to its board scope.) - gstPlay: normalise a file:// URI to a filesystem path before handing it to filesrc (Anthias passes bare local paths today; this is defensive). - Start audio once in gstPlay so BOTH the overlay AND the appsink-raster fallback get sound — previously the raster fallback played silent (audio was only started in gstPlayOverlay). gstStartAudio now escapes its own path; gstPlayOverlay drops the unused options param. - play(): log when gstPlay exhausts every GStreamer path (there is no QMediaPlayer fallback on this board) instead of returning silently. Validated on a real Pi 3: video 30fps zero ongoing drops + audio (3.5mm) still play; pi3-64 (with GStreamer) and pi4-64 (without) both compile. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Addressed Copilot review (commit ef01e89)
Re-validated on a real Pi 3: 30 fps zero ongoing drops + audio (3.5 mm); pi3-64 (with GStreamer) and pi4-64 (without) both compile. |
- videoview.cpp: clamp ANTHIAS_GST_QUEUE — a 0/negative/non-numeric value made queueBuffers 0, i.e. an unbounded queue (max-size-buffers/time/bytes all 0) → OOM on the 1 GB board; fall back to the default 4. - gstStartAudio: check gst_element_set_state()'s result; on GST_STATE_CHANGE_FAILURE log + tear the audio pipeline down instead of leaving a dead pipeline reporting "started". - containerFps: was written from the kmssink pad-probe (streaming thread) and read by sampleStats (GUI thread) as a plain double — a data race. Make the handoff atomic (QAtomicInteger fps×1000 via containerFps() / setContainerFps()). - main.cpp: the fatal-signal handler now uses only async-signal-safe calls (write() + backtrace_symbols_fd) instead of fprintf/fflush. - test_viewer.py: also assert ANTHIAS_VIDEO_OVERLAY is set on pi3-64 and cleared on other boards. Validated on a real Pi 3: 30fps zero ongoing drops, container-fps=30.00, audio (3.5mm) playing; pi3-64 (with GStreamer) and pi4-64 (without) both compile. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Addressed Copilot review round 2 (commit e5f631e)
Re-validated on a real Pi 3: 30 fps zero ongoing drops, |
- sampleStats: on the gstMode appsink-raster fallback, query the GStreamer pipeline position instead of QMediaPlayer::position() (the player isn't the one playing there, so position-ms was misleading). - gstPlayOverlay: guard a null QGuiApplication::primaryScreen() before using it for the DRM native-resource lookups; document why driFd<=0 is the correct "not available" test here (eglfs returns the fd through a void* API — a real DRM master fd is always >2, "no fd" comes back as 0). - main.cpp: soften the crash-handler comment — backtrace()/ backtrace_symbols_fd() are glibc extensions, not formally guaranteed async-signal-safe (best-effort last-gasp diagnostic). Compiles on pi3-64 (with GStreamer) and pi4-64 (without); Pi 3 still plays 30fps with audio. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- gstPlay: treat a missing appsink (gst_bin_get_by_name null) as a hard failure — otherwise the pipeline goes PLAYING to a silent black screen. - play(): only start the stats sampler when gstPlay() succeeds, so a failed start doesn't emit SAMPLE lines with position=-1/elapsed=-1. - gstStartAudio: escape the ALSA device string before interpolating it into the gst_parse_launch description (same as the filesrc path). - gstStop: drop the stashed frame (under the frame mutex) and reset the drain flag, so a late-queued onGstFrame() can't repaint a stale frame after teardown. pi3-64 compiles; changes are pi3-64-scoped (#ifdef ANTHIAS_GSTREAMER) so other boards are unaffected. Pi 3 still plays 30fps with audio. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
The gstMode STOP stats line hard-coded position-ms=0, which read like playback never advanced. Query the pipeline position (it's still up at that point; gstStop() runs after) and log it, or -1 if the query fails. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…und 6) - anthias_gst_bus_loop: on GST_MESSAGE_ERROR, tear the pipeline down (queued VideoView::stop()) and return FALSE to drop the watch, instead of logging and leaving a dead pipeline on a stale frame — the next asset re-shows and rebuilds. - gstPlay (appsink-raster path): install the same bus watch so pipeline errors are surfaced/recovered and EOS looping goes through one path (gstRestartLoop). Drops the now-redundant appsink EOS callback (anthias_gst_on_eos); appsink posts EOS to the bus like any GstBaseSink. Validated on a real Pi 3: overlay path still plays 30fps, loops cleanly across the 60s boundary (no respawn), audio playing. pi3-64-scoped (#ifdef ANTHIAS_GSTREAMER); other boards unaffected. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
Comment-only. The code evolved past several comments: - appsink note said sync=true; the pipeline uses sync=false (GUI-side coalescing paces the paint) — corrected. - gstStop said the appsink path has no bus watch; it now installs one (round 6) — corrected. - two comments claimed decoder-direct I420 "renders blue" on the vc4 plane, but decoder-direct I420 is the working default; reworded to describe the formats/mmap-copy accurately without the (thrashed-board artifact) "blue" assertion. No code change. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…nd 8) - STOP: on the overlay path report gstRawSamples as frames-rendered (the Qt frame counters stay 0 there, kmssink's tally is the real count). - appsink-raster path: latch the source framerate from the negotiated caps (latchSourceFps) so its SAMPLE computes expected/dropped instead of reporting -1. - Remove dead gstUri / gstRequeueUri (leftover playbin about-to-finish requeue path; the current pipelines don't use playbin). - Fix the gstRestartLoop header comment (loops via the pipeline bus watch, not the removed appsink EOS callback). Compiles on pi3-64 + pi4-64; overlay hot path unchanged from the already-validated build. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
gstRawSamples was reset only in gstPlayOverlay, so the appsink-raster path carried the previous asset's cumulative count into its stats. Reset it once in gstPlay() right after teardown (covers overlay + appsink); drop the now-redundant reset in gstPlayOverlay. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…ound 10) - onGstFrame(): release gstDrainPending at the END of the drain, not the start. pushGstFrame() uses testAndSetOrdered(0,1) so exactly one drain is ever in flight; clearing at the end keeps it that way (frames arriving mid-drain just update gstLatestFrame) — bounded GUI event queue, no unbounded QImage backlog. - gstPlay(): clear the raster surface right after teardown so the previous asset's last frame can't linger (stale-frame flash) while the new pipeline prerolls or if it fails to start. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- main.cpp: install the fatal-signal handler via sigaction() with SA_RESETHAND instead of signal(). The handler no longer calls signal() (not async-signal-safe) — SA_RESETHAND restores the default disposition so the re-raise takes the default path (core dump / exit). - onGstFrame: release gstDrainPending with storeRelease (was storeRelaxed) so it pairs with pushGstFrame's testAndSetOrdered acquire — the drain's writes are visible before the next drain can be queued. (Copilot also flagged toImage() in onRasterPainted; that is the dormant QMediaPlayer-raster path — pi3-64 uses gstMode, which feeds frames via onGstFrame/setImage and never runs that toImage-in-paint path.) Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|
Round 11: switched the crash handler to |
Comment-only. The two "CPU-raster presenter" comments predate the overlay default; reworded to: pi3-64 uses the non-VideoOutput presenter — the GStreamer vc4 overlay-plane path (preferred, ANTHIAS_VIDEO_OVERLAY) with the CPU-raster blit as fallback. (Copilot also suggested a QtTest that constructs VideoView with ANTHIAS_VIDEO_RASTER=1; that headless C++ harness is a reasonable follow-up — the env wiring is already covered by tests/test_viewer.py.) Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
|



What
Enables real hardware video playback on the Raspberry Pi 3 arm64 board (
pi3-64, VideoCore IV, eglfs) — 30 fps with zero ongoing dropped frames, with audio.Closes #3084
Why
On
pi3-64, QtMultimedia'sVideoOutputcannot present a video node on the vc4 GL driver, so video rendered black; and compositing a full-screen video widget through eglfs GL caps at ~9 fps regardless. Anthias has a lot of active Pi 3 users, so this needed a proper hardware path, not a workaround.How
Video (
VideoView::gstPlayOverlay) — in-process GStreamer, scanning out on a vc4 DRM overlay plane HW-composited with the eglfs UI:QPlatformNativeInterfaceand a libdrm free-overlay-plane finder.capture-io-mode=mmapis the key: kmssink then copies each frame into its own scanout buffer instead of pinning the decoder's DPB pool — that pin deadlocked every plain decoder-direct attempt. The bcm2835 decoder emits I420 at full rate; the ISP convert-to-NV12 alternative is a second M2M pass that caps at ~18 fps, so it's only a fallback (ANTHIAS_GST_ISP_CONVERT=1).Audio (
VideoView::gstStartAudio) — a separate GStreamer pipeline (qtdemux ! decodebin ! audioconvert ! audioresample ! alsasinkon the HDMI card):gstreamer1.0-libavfor the audio decoders (avdec_ac3/aac/mp3/…); decodebin reported a missing plug-in on AC3 with only faad (AAC) present.Plumbing: only
pi3-64links GStreamer (AnthiasViewer.pro, Dockerfile build + runtime deps);_build_webview_envenables the path;start_viewerchowns the GStreamer registry cache to avoid a per-launch rescan;main.cppinstalls a fatal-signal backtrace handler.Testing
Validated on a real Pi 3 (192.168.200.99) with a 1080p30 H.264/AC3 clip over HDMI, deployed as the full production image (no docker-cp, no env overrides):
droppedflat at the startup ramp only (zero ongoing drops)respawns=1(no crash, no deadlock)/dev/video10held (HW decode) on the vc4 overlay planePython unit tests for the env wiring pass (
tests/test_viewer.py);ruff check/formatclean.Follow-ups (not blocking)
+rpt3archive for production parity.audio-devicecard, default HDMI).🤖 Generated with Claude Code