Viewer is doing server-shaped work: move URL reachability to the server, delete dead IP/Balena startup code #2803

Context

The viewer process owns a few responsibilities that don't belong to it.
This issue covers two of them:

• It runs per-play URL reachability checks (including ffprobe for
  streams) — work that the server already does at upload time, and
  that's structurally a server concern (it touches the asset DB, fits
  Celery, and feeds the admin UI).
• It contains a 90-second Balena startup stall that throws away its
  result, plus an unused wait_for_node_ip helper that imports the
  host-IP-discovery dance for nothing. IP discovery is already wired
  end-to-end through the server: splash_page view → get_node_ip →
  host_agent.py. The viewer's participation is residual.

Both are independent issues with their own justifications: a
performance bug, and dead code. They're bundled here because they're
the same shape of problem (viewer doing things that aren't viewer
work) and they touch the same module.

Each part can ship as its own PR.

Part A — Move asset URL reachability to the server

Problem

viewer/__init__.py:222 calls lib/utils.url_fails(asset['uri']) on
every asset play. The same check already runs server-side at asset
upload (api/serializers/mixins.py:120,
api/serializers/v1_1.py:141), so the viewer's call is redundant
drift detection.

It's also expensive in the wrong place:

1. For HTTP assets it's a HEAD request (~10-100 ms) every rotation —
   typically many times per hour per asset.
2. For streaming assets (RTSP/RTMP) it shells out to ffprobe, which
   takes 1-5 s and blocks the asset loop. Users see a stall on
   every rotation of every stream asset.
3. It pulls ffprobe into the viewer's hot path even though the
   server is the natural owner of asset metadata.

URL reachability is a property of an asset, not a property of "what
the viewer is about to play right now." It belongs next to the asset
record, refreshed on a cadence, surfaced through the admin UI/API.

Proposal

• Migration: add Asset.is_reachable: BooleanField(default=True)
  and Asset.last_reachability_check: DateTimeField(null=True).
• Add Celery beat task anthias_app.tasks.revalidate_asset_urls
  that runs url_fails(asset.uri) per asset on a configurable
  interval (default: 15 min) and updates the fields.
• Add an on-demand revalidation path (Celery task or internal
  endpoint) so a single asset can be re-checked without waiting
  for the next periodic sweep.
• Update viewer/__init__.py:asset_loop to skip assets where
  is_reachable=False instead of calling url_fails itself, and
  to publish a "recheck this asset" message via the existing Redis
  anthias.viewer channel when display fails (preserves the
  "stream just went down" failure-mode coverage that the per-play
  check used to provide).
• Remove the per-play url_fails(asset['uri']) call and its
  import from viewer/__init__.py.
• Surface is_reachable in the asset serializer so the admin UI
  can show "broken asset" state.

Trade-offs

We trade per-play freshness for up-to-15-min staleness on broken
streams. The error-triggered recheck is the mitigation: when a stream
goes down, the viewer's first failed playback triggers a server
revalidate, the asset is marked unreachable, and the next rotation
skips it.

The 15-minute default is a starting point — it should be a setting.

Part B — Remove misplaced IP/Balena startup ceremony from the viewer

Problem

Two pieces of viewer startup code don't do what they look like
they're doing.

B1. The Balena pre-wait throws away its result
(viewer/__init__.py:326-333):

if settings['show_splash']:
    if is_balena_app():
        for attempt in Retrying(
            stop=stop_after_attempt(MAX_BALENA_IP_RETRIES),  # 90
            wait=wait_fixed(BALENA_IP_RETRY_DELAY),           # 1s
        ):
            with attempt:
                get_balena_device_info()   # result is discarded
    view_webpage(SPLASH_PAGE_URL)

The viewer hits the Balena supervisor up to 90 times (1 s apart) and
discards the response. The IPs that actually appear on the splash
come from the server-side splash_page view
(anthias_app/views.py:57-75), which calls get_node_ip() at render
time and has no awareness of whether the viewer's pre-wait succeeded.

Net effect: up to a 90-second startup stall on Balena devices, with
no guarantee that the splash actually shows valid IPs afterwards. If
the goal is "splash always shows real IPs," the fix belongs on the
server, not in the viewer.

B2. wait_for_node_ip is defined but never called. grep -r wait_for_node_ip returns only the definition at
viewer/__init__.py:288-294. It's dead code, and its get_node_ip
import drags the host-agent IP-discovery dance (Redis publish to
hostcmd, host_agent.py reads, sets ip_addresses in Redis) into
the viewer's import graph for nothing.

Both pieces also blur a clean responsibility boundary: IP discovery
is a server/host concern (the server already orchestrates it via
host_agent.py for non-Balena and via the Balena supervisor for
Balena), and the viewer should be a thin renderer that doesn't
participate.

Proposal

• Delete the Balena pre-wait Retrying block at
  viewer/__init__.py:326-333.
• Delete wait_for_node_ip (viewer/__init__.py:288-294).
• Drop now-unused imports from viewer/__init__.py:
  is_balena_app, get_balena_device_info, get_node_ip,
  Retrying, stop_after_attempt, wait_fixed,
  MAX_BALENA_IP_RETRIES, BALENA_IP_RETRY_DELAY.
• Remove MAX_BALENA_IP_RETRIES and BALENA_IP_RETRY_DELAY from
  viewer/constants.py.
• Confirm tenacity can be removed from the viewer's pyproject
  group if no other viewer code uses it (grep tenacity viewer/
  after the change).

Server-side robustness fix (only if verification shows it's needed)

If removing the Balena pre-wait causes the splash to show "Unknown"
on slow first-boot devices, fix it server-side so the responsibility
stays in the right place:

• Have host_agent.py cache the last-known-good IP set in Redis
  with a long TTL, so get_node_ip() can return cached IPs on a
  fresh boot before the live lookup completes.
• Or: splash_page view uses its own short-lived retry loop on
  get_node_ip failure, so the page itself waits, rather than
  having the viewer wait for it.

Verification

The risk is that "delete a 90-second wait" actually breaks a Balena
boot path no-one explicitly remembers. Before merging:

• Boot a fresh Balena device with the pre-wait removed; confirm the
  splash renders with real IPs (not "Unknown") within a reasonable
  window.
• Confirm splash on non-Balena devices is unchanged (this code path
  was already inside if is_balena_app()).

Out of scope

• Changes to viewer/scheduling.py, viewer/messaging.py, or
  mpv/VLC subprocess management — those are correctly viewer-side.
• Any rewrite of the viewer in a different language. This issue is
  about responsibility hygiene, not portability.

Suggested PR sequence

1. PR 1 (Part A): schema migration + Celery task + serializer
   surface for is_reachable.
2. PR 2 (Part A cont.): error-triggered revalidate channel,
   viewer asset_loop change, remove per-play url_fails.
3. PR 3 (Part B): delete Balena pre-wait + wait_for_node_ip +
   import cleanup. Independent of Part A; can ship in any order.
4. PR 4 (optional, Part B follow-up): host_agent IP caching or
   splash-side retry, only if PR 3 verification shows it's needed.