feat(processing): normalise HEIC/HEIF/TIFF images and exotic-codec videos at upload time - #2832

…-codec videos

Two new Celery tasks run on every fresh upload, mirroring the
``download_youtube_asset`` async pattern:

* ``normalize_image_asset`` converts HEIC / HEIF / TIFF to lossless
  WebP via Pillow + pillow-heif, preserving alpha. Other image
  formats short-circuit out as a no-op.
* ``normalize_video_asset`` ffprobes the upload, passes through if
  it's already H.264/HEVC in an accepted container with a viewer-
  friendly audio codec, otherwise transcodes to H.264 + AAC MP4 with
  ``-threads 2 -preset medium -crf 23`` so two cores stay free for
  the on-device viewer.

Both tasks land their output via a staging-file rename, write
``Asset.metadata`` flags (``original_ext``, ``transcoded`` /
``converted``, ``error_message``), and clear ``is_processing`` on
success — or via a custom ``Task.on_failure`` on permanent failure
so a row never stays stuck on the "Processing" pill.

Schema:
* New ``Asset.metadata`` JSONField (default dict) plus migration.
  Exposed read+write on ``AssetSerializerV2`` (read-only on v1.x).

Wiring:
* ``CreateAssetSerializerMixin.prepare_asset`` flags ``is_processing``
  and stashes ``_pending_normalize`` (``image``/``video``/``None``);
  ``AssetListViewV2`` and ``AssetListViewV1_2`` dispatch the matching
  task after persistence.
* The HTMX ``assets_upload`` view now persists the source extension
  on disk so the task can identify the format, replaces the
  ``probe_video_duration`` hop with ``normalize_video_asset``
  (whose passthrough branch is the same probe + duration path),
  and dispatches ``normalize_image_asset`` for HEIC/HEIF/TIFF.
* Add-asset modal accepts the wider extension list.

Resource control:
* ``anthias-celery`` worker command wrapped with
  ``nice -n 19 ionice -c 3`` in compose templates so transcodes
  never starve the on-device viewer.
* ``ffmpeg`` invocation pins ``-threads 2`` for the same reason.

Dependencies:
* New: Pillow, pillow-heif (Python); libheif1 in
  ``base_apt_dependencies`` (~1 MB extracted).
* No changes to ffmpeg/ffprobe — already runtime deps.

Tests:
* ``tests/test_processing.py`` covers both tasks: HEIC/HEIF/TIFF
  conversion (incl. uppercase ext, RGBA handling), JPEG no-op,
  corrupt-input failure path, six-row passthrough decision table,
  exotic-codec → H.264 transcode (mpeg2, mjpeg), MP4-with-non-H264
  in-place transcode, ffmpeg timeout/failure/zero-byte cleanup,
  ffprobe missing-stream parsing, on_failure metadata write,
  prepare_asset routing for HEIC / video / remote URL / JPEG.
* PDF support is deferred to a follow-up — out of scope here per
  the issue's "image/video first" framing.

* assets_upload now falls back to UploadedFile.content_type and
  finally an extension-based classification so HEIC/HEIF/TIFF
  uploads still classify on hosts whose mimetypes DB doesn't ship
  ``image/heic`` mappings.
* _ffprobe_summary derives the container from ffprobe's
  ``format.format_name`` (a comma-joined synonym list — pick the
  first token in the passthrough set) instead of trusting the
  filename extension. A ``.bin`` file containing MP4 bytes now
  classifies correctly; a ``.mp4`` file containing avi-only bytes
  no longer slips into the passthrough branch.
* Zero-byte ffmpeg output now removes the staging file before
  raising, mirroring the timeout/error branches above. All three
  failure paths share a small _drop_staging() helper so cleanup
  stays consistent.

New tests:
* ffprobe summary prefers format_name over filename, with a
  deterministic fallback to the extension when format_name is
  absent.
* zero-byte transcode output cleans up its staging file (asserts
  no leftover ``staging`` files in assetdir).
* assets_upload classifies HEIC via Content-Type when guess_type
  returns None.

Extends the image-normalisation pipeline to cover the realistic set
of "operator drags an unusual image format into the upload modal"
cases, all handled by Pillow's built-in decoders without a new apt
or wheel dependency:

  ┌──────────┬────────────────────────────────────────────────────┐
  │ Format   │ Why we want it converted                           │
  ├──────────┼────────────────────────────────────────────────────┤
  │ BMP      │ Uncompressed; a 4K BMP is ~30 MB vs ~1 MB as WebP. │
  │ ICO      │ Multi-frame Windows icon; pick the largest, flatten│
  │ TGA      │ Screenshot tools / game asset exports; no browser  │
  │          │ support.                                           │
  │ JPEG2000 │ .jp2/.j2k/.jpx/.jpc/.jpf — scanner output; no      │
  │          │ browser support.                                   │
  │ AVIF     │ Modern phone exports. Chromium 85+ renders AVIF,   │
  │          │ but the legacy Pi 2/3 Qt5 WebEngine predates it,   │
  │          │ so converting on upload means one playback path    │
  │          │ across the fleet.                                  │
  └──────────┴────────────────────────────────────────────────────┘

JPEG / PNG / WebP / GIF / SVG remain untouched — already
viewer-friendly *and* well-compressed.

Implementation:
* Extend ``NORMALIZE_IMAGE_EXTS``; the rest of the pipeline already
  accepts any extension in this set (RGBA conversion happens inside
  ``_convert_image_to_webp`` regardless of source format).
* Replace the duplicate extension set in ``assets_upload`` with a
  call to ``processing.needs_image_normalisation`` so the source of
  truth lives in one place.
* Widen the upload modal's <input accept> attribute.

Tests:
* ``test_image_normalises_to_lossless_webp_across_formats`` is a
  parametrised matrix that round-trips each new format end-to-end:
  source synthesised via Pillow, runs through
  ``_run_image_normalisation``, asserts the WebP output decodes
  cleanly back to a 16x16 image. Catches both decoder-side
  regressions (Pillow drops a format) and writer-side regressions
  (RGBA convert mode breaks one source).
* ``test_needs_image_normalisation`` extended to cover every entry
  in the new set plus negative cases (.jpg/.png/.webp/.gif/.svg
  stay False). Total: 109 image-format assertions.

Six items from Copilot's fresh review pass:

* ``assets_upload`` last-resort image-extension allowlist now
  derives from ``processing.NORMALIZE_IMAGE_EXTS`` rather than
  duplicating the set. Adding a new normalisable format (or
  removing one) only touches one place.
* ``_run_image_normalisation`` cleans up the ``.webp.tmp`` staging
  file on every failure path — Pillow's ``UnidentifiedImageError``
  *and* a generic OSError mid-encode (disk pressure, libheif
  crash). Mirrors the video pipeline's _drop_staging contract.
* ffmpeg failure messages decode the bytes ``stderr`` to UTF-8
  text (with replacement on malformed bytes) and tail-trim long
  output, so ``metadata.error_message`` reads as a real
  diagnostic instead of ``b'Invalid data found'``.
* Removed the dead ``path.normpath(staging) == path.normpath(
  src_uri)`` branch in the video transcode path. With the staging
  suffix in place the two paths can never collide; expanded the
  surrounding comment to explain why.
* Updated ``normalize_video_asset``'s docstring to describe the
  per-board codec grid (libx264 on pi2/pi3, libx265 on pi4-64 /
  pi5 / x86) rather than the now-stale "transcode to H.264 MP4".
* Fixed "truecate" → "truncate" typo in ``_asset_row.html``
  comment.

New tests:
* ``test_image_partial_write_cleans_staging`` — half-writes the
  ``.webp.tmp`` then raises OSError; asserts the runner removes
  the partial file before propagating.
* ``test_format_subprocess_stderr_decodes_and_trims`` — covers
  the bytes-decode, malformed-byte-replacement, tail-trim, and
  empty-stderr cases for the new helper.
* ``test_video_ffmpeg_error_cleans_staging`` strengthened to
  assert the error message contains *no* ``b'...'`` Python repr
  prefix — it's now operator-readable text.

Three code fixes plus a PR-description sync:

* ``_run_image_normalisation`` no-op path (when src_ext isn't in
  NORMALIZE_IMAGE_EXTS) now also clears
  ``metadata.error_message``. Without this, a row re-uploaded as
  a JPEG/PNG after a previously-failed HEIC conversion would
  drop is_processing but keep showing the "Failed" pill — the
  operator's table would lie about the row's current state.
* ``_ffprobe_summary`` now catches ``sh.CommandNotFound`` in
  addition to ``TimeoutException`` / ``ErrorReturnCode``. A
  stripped-down image / dev box without ffprobe in PATH used to
  crash the task with an unhandled CommandNotFound; now it
  collapses to the same all-'unknown' summary so the runner
  falls through to the transcode branch (which itself fails
  clean if ffmpeg is also missing — same on_failure contract).
* Rewrote the ``_ffprobe_summary`` docstring: the actual
  behaviour is "unknown" for missing video stream, "none" only
  for genuinely missing audio stream. The previous "''" claim
  was wrong and would have misled callers / future maintainers.

Tests:
* ``test_image_no_op_path_clears_stale_error_message`` — JPEG
  re-uploaded over a row whose previous attempt failed; the
  no-op branch must wipe the stale error_message.
* ``test_ffprobe_summary_handles_missing_ffprobe_binary`` —
  CommandNotFound side-effect; asserts all-'unknown' summary
  rather than a propagating exception.

Four contract gaps Copilot flagged:

* ``normalize_image_asset`` / ``normalize_video_asset`` use
  ``autoretry_for=(OSError,)`` to recover from transient disk
  pressure. ``FileNotFoundError`` is-a ``OSError`` so the filter
  was catching it too — but a missing source file is permanent,
  and retrying just delays the on_failure that writes
  ``metadata.error_message``. Adding
  ``dont_autoretry_for=(FileNotFoundError,)`` to both decorators
  makes the missing-source raise propagate immediately, so the
  operator sees the "Failed" pill and the error message at the
  next browser refresh instead of waiting through up-to-3
  exponential-backoff retry cycles.

* ``_run_image_normalisation`` and ``_run_video_normalisation``
  both call ``os.replace(staging, final_uri)`` after a successful
  conversion / transcode. A rename failure (cross-device link,
  filesystem-full at the very last step, permissions) was
  outside the existing try/except, so the staging file would
  linger until cleanup()'s 1h sweep. Wrap both in a try/except
  that calls ``_drop_image_staging`` / ``_drop_staging`` on any
  OSError before propagating — the "no leftover staging
  artifacts on failure" contract now holds across every failure
  path.

Tests:
* ``test_image_rename_failure_cleans_staging`` and
  ``test_video_rename_failure_cleans_staging`` — patch
  ``os.replace`` to raise OSError; assert the staging file is
  gone before the exception reaches the runner's caller.
* ``test_normalize_tasks_exclude_filenotfounderror_from_autoretry``
  — celery-config-time check that both tasks expose
  ``FileNotFoundError`` in their dont_autoretry_for tuple, so a
  future change to the decorator can't silently regress the
  immediate-fail contract.

Three stale docstring callouts from Copilot's review of 7099b25:

* ``processing.py`` module docstring — listed only HEIC/HEIF/TIFF
  for the image task. Updated to enumerate the full set
  (HEIC/HEIF/TIFF/BMP/ICO/TGA/JPEG 2000 family/AVIF) and to call
  out the JPEG/PNG/WebP/GIF/SVG no-op short-circuit.
* ``needs_image_normalisation`` docstring — same drift; rewrote
  the leading sentence to match what the predicate actually
  checks (``_ext(...) in NORMALIZE_IMAGE_EXTS``).
* ``tests/test_processing.py`` module docstring — said the image
  task covers only HEIC/HEIF/TIFF and that video transcodes are
  libx264-only. Both stale: extended to enumerate every image
  format the suite exercises, and to describe the per-board
  ``DEVICE_TYPE`` codec grid (libx264 on pi2/pi3, libx265 on
  pi4-64/pi5/x86) that the parametrised video tests pin down.

No code changes; documentation only.

…ad on intermediate hops

Two more Copilot items:

* ``_format_subprocess_stderr`` had two trim branches: bytes (via
  byte-precise tail) and str (via character-count tail). The str
  branch could exceed _STDERR_TAIL_BYTES under multibyte text.
  Normalise to bytes once at the top (encoding str via UTF-8 with
  replacement) and run a single byte-precise trim — both paths
  now respect the byte budget identically.

* ``_notify`` gains a ``reload_viewer`` keyword. The YouTube
  task's intermediate notification (after writing title/duration
  but before chaining into normalize_video_asset, while
  is_processing is still True) now passes ``reload_viewer=False``.
  The browser-side dashboard nudge still fires so the operator
  sees the resolved title immediately; the on-device viewer
  doesn't reload its playlist for a row that's still mid-flight.
  The chained normalize step's _notify (which runs once
  is_processing clears and the file is final) handles the actual
  viewer reload — saves the viewer one redundant playlist refresh
  per YouTube upload.

Tests:
* ``test_notify_browser_only_skips_viewer_reload`` exercises the
  new flag.
* The YouTube-success test now mocks Redis and asserts
  ``publish.assert_not_called()`` to lock in the no-intermediate-
  reload contract.

* processing.py: _set_processing_error docstring listed
  "encrypted PDF" as a permanent-failure case from the issue's
  three-workstream framing. PDF is explicitly out of scope for
  this PR — replaced with concrete failure modes the current
  image/video tasks actually surface.
* docker-compose.yml.tmpl: "a single configure here" reads as
  a verb. Changed to "a single configuration here".

Real security gap Copilot caught: Pillow happily allocates pixel
buffers proportional to ``width × height`` regardless of how
small the source file is on disk. A few KB of crafted bytes
advertising a 1,000,000×1,000,000 image would force the celery
worker to attempt a multi-TB allocation — at best a hard OOM
that kills the worker and stalls the upload pipeline; at worst
a swap-storm that drags the on-device viewer with it.

Pillow ships ``MAX_IMAGE_PIXELS`` (default ~89 MP) which raises
``DecompressionBombError`` past 2× that threshold and warns
softly at the first level. That default is too lax for signage
content (where 4K @ 8 MP is already large) and pillow-heif's own
decoder can bypass the check on certain HEIF/AVIF inputs.

Two layers of protection:

1. ``_MAX_IMAGE_PIXELS = 50_000_000`` constant — bigger than any
   legitimate phone-camera output (modern flagships top out
   around 50 MP at the standard 4:3 aspect after JPEG/HEIC
   compression) but tiny compared to typical bomb fixtures.
2. ``_convert_image_to_webp`` reads ``image.size`` from the
   format header *before* any decode and raises ValueError if
   the dimensions exceed the cap. The on_failure path writes
   the message to ``metadata.error_message`` like any other
   permanent failure. Lowering Pillow's global
   ``Image.MAX_IMAGE_PIXELS`` to the same value protects any
   future call site that goes through ``Image.open`` outside
   this helper.

New test ``test_image_decompression_bomb_is_rejected`` mocks
``Image.open`` to return a stub whose ``.size`` exceeds the cap
(synthesising a real billion-pixel fixture would itself need
GBs of memory) and asserts the runner raises before any
``convert()`` / ``save()`` is reached.

Inline comment listed only HEIC/HEIF/TIFF/BMP, but the constant it
points at also covers ICO/TGA/JP2 family/AVIF. Rewrote to reference
the constant as source of truth and enumerate the current set so
the comment stops drifting on the next addition.

Two real bugs Copilot caught:

* ``_set_processing_error`` cleared ``is_processing`` but left
  ``is_enabled=True``, so a failed normalisation would still get
  queued for playback by the viewer's scheduler (which filters on
  is_enabled + date window only — it doesn't check
  ``metadata.error_message``). The on-screen result was a black
  rectangle for the row's duration. Flipping ``is_enabled=False``
  alongside ``is_processing=False`` keeps the bad row out of
  rotation; the operator can re-enable from the dashboard once
  the underlying issue is fixed. The ``error-pill`` template
  already replaces the active toggle so the operator sees the
  failure state before they re-enable.

* ``assets_upload`` deferred to ``mimetypes.guess_type`` first and
  only consulted ``UploadedFile.content_type`` when guess_type
  produced no image/video classification. If an operator renamed
  a HEIC to ``photo.jpg`` and uploaded it, guess_type returned
  ``image/jpeg`` (a passthrough type), the Content-Type fallback
  was skipped, the file landed as ``.jpg``, and the normalise
  pipeline never ran — a silent failure-to-render. Modern
  browsers sniff the actual bytes and tag the upload with
  ``image/heic`` regardless of filename, so the view now
  cross-checks: when guess_type and Content-Type share a
  top-level (image/* or video/*) but disagree on subtype, AND
  Content-Type's subtype maps to a NORMALIZE_IMAGE_EXTS
  extension, prefer Content-Type. Only upgrades — never downgrades —
  to avoid the inverse case (a JPEG mis-tagged as image/heic by
  the browser somehow) accidentally routing into the pipeline.

Tests:
* test_set_processing_error_writes_metadata extended to assert
  is_enabled flips to False alongside the error message write.
* New test_assets_upload_misnamed_heic_uses_browser_content_type
  uploads HEIC bytes named ``photo.jpg`` and asserts the file
  lands as .heic with the normalise task dispatched.

…builds

Real concern Copilot caught about the Pillow / pillow-heif
introduction: neither ships armv7l manylinux wheels (Pillow 11
explicitly dropped them in its release notes; pillow-heif only
publishes x86_64 / aarch64). uv's resolution on a pi2 / pi3
image build therefore falls back to sdist, and the existing
``builder_extra_apt`` only covers libcec / libdbus headers — the
``uv sync`` step would gcc-fail at the first JPEG / HEIF binding.

Extend ``get_uv_builder_context`` to take a ``board`` argument
and append the Pillow / pillow-heif build-time deps when
``service='server'`` and ``board in {'pi2', 'pi3'}``. 64-bit
boards (pi4-64 / pi5 / x86) and the test image still get binary
wheels and the apt list stays unchanged for them — adding the
deps unconditionally would waste ~70 MB of layer space on every
non-armv7 build.

Pillow's documented build deps:
  libjpeg62-turbo-dev / libfreetype-dev / liblcms2-dev /
  libopenjp2-7-dev / libtiff-dev / libwebp-dev / zlib1g-dev

pillow-heif: libheif-dev (the libheif1 runtime is already in
``base_apt_dependencies`` for both architectures).

Verified: ``--build-target pi3`` now generates a Dockerfile that
installs the new build deps; ``--build-target pi5`` does not.