fix: e2e-test findings (host-agent venv, celery beat, asset GET 404) - #2881
Merged
Conversation
…203/EXEC) PR #2843 switched the installer venv to a mktemp tmpdir cleaned up on EXIT, but anthias-host-agent.service's ExecStart still hardcodes /home/${USER}/installer_venv/bin/python. Every fresh install since that refactor leaves the unit in a status=203/EXEC restart loop with no Python at the configured path, and /api/v2/info then blocks ~80s on get_node_ip() waiting for the host_agent_ready key that will never appear. Split the two venvs: * INSTALLER_VENV: still ephemeral mktemp, used by ansible-core during install/upgrade and torn down by the EXIT trap. * HOST_AGENT_VENV: new persistent venv at /home/${USER}/installer_venv (path kept stable so devices installed before the refactor don't need a unit rewrite), recreated from the host dep group on every install + upgrade so deps track pyproject.toml. provision_host_agent_venv runs after install_ansible() and before run_ansible_playbook() so the venv exists before ansible's state: started fires the unit. On upgrade the unit is already loaded with the previous venv's in-memory interpreter, so the state: started no-op never picks up the new deps — restart explicitly when the unit is already active. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
…te3 locking) celery -B with the default PersistentScheduler stores its schedule via shelve. On Python 3.13, shelve defaults to dbm.sqlite3, which raises dbm.sqlite3.error: locking protocol intermittently under contention — observed on x86 but not pi4-64 in this build matrix, which is consistent with a benign-looking race specific to the amd64 docker layer's filesystem ordering. When Beat stalls, reconcile_stuck_processing and the other periodic tasks set up by setup_periodic_tasks stop firing, so stuck-in-is_processing assets never get re-dispatched. setup_periodic_tasks defines every periodic task statically (no django-celery-beat / no dynamic schedule edits), so a non-persistent scheduler is sufficient. Switch to celery.beat.Scheduler in all three compose files (prod template + dev + test) and drop the --schedule /tmp/celerybeat-schedule flag that's now unused. The telemetry cooldown comment is updated to reference the new flag — the actual 24h cooldown is still gated by the Redis TTL, which is the persisted source of truth. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
….2/v2
AssetViewV{1,1_1,1_2,2}.get / put / patch / update and the shared
DeleteAssetViewMixin / AssetContentViewMixin / ViewerCurrentAssetViewV1
all called Asset.objects.get(asset_id=...) bare. The Asset.DoesNotExist
that fires for a deleted-or-typo'd id has no DRF exception handler
registered, so it bubbled up as a 500 with the database traceback —
caller sees a server error for what is structurally a missing
resource. AssetRecheckViewV2 already gets this right via
filter(...).exists() + explicit 404; standardise the rest by routing
the lookup through django.shortcuts.get_object_or_404 (DRF's exception
handler converts the resulting Http404 to a clean 404 Response).
The new test_unknown_asset_id_returns_404 parametrises across every
API version so a future view that reverts to Asset.objects.get bare
trips immediately.
Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
There was a problem hiding this comment.
Pull request overview
Addresses three e2e-discovered regressions affecting host installs, Celery beat reliability, and asset API error semantics.
Changes:
- Split installer/runtime Python environments by provisioning a persistent venv for
anthias-host-agentand restarting the unit on upgrade (bin/install.sh). - Switch Celery beat to the in-memory scheduler to avoid Python 3.13
shelve/dbmlocking issues (compose files) and update related telemetry docs. - Return proper 404s for unknown
asset_idacross all API versions by usingget_object_or_404, with a new parametrized regression test.
Reviewed changes
Copilot reviewed 10 out of 11 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
bin/install.sh |
Adds persistent host-agent venv provisioning and upgrade-time service restart. |
docker-compose.yml.tmpl |
Uses in-memory Celery beat scheduler instead of persisted schedule file. |
docker-compose.dev.yml |
Mirrors Celery scheduler change for dev. |
docker-compose.test.yml |
Mirrors Celery scheduler change for tests/CI. |
src/anthias_server/lib/telemetry.py |
Updates comment to reflect in-memory beat scheduler. |
src/anthias_server/api/views/v2.py |
Uses get_object_or_404 for asset detail/update lookups. |
src/anthias_server/api/views/v1.py |
Uses get_object_or_404 for asset detail/update and current-asset lookup. |
src/anthias_server/api/views/v1_1.py |
Uses get_object_or_404 for asset detail/update lookups. |
src/anthias_server/api/views/v1_2.py |
Uses get_object_or_404 for asset detail/update lookups. |
src/anthias_server/api/views/mixins.py |
Uses get_object_or_404 in delete + content mixins. |
src/anthias_server/api/tests/test_assets.py |
Adds regression test asserting 404 for unknown asset IDs across API versions. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
get_object_or_404 returns a single Asset, not a queryset; the variable name was already misleading under the previous bare Asset.objects.get(...) call. Address Copilot review. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
INSTALLER_VENV lands in /tmp (the mktemp -t default), while uv's cache lives at ~/.cache/uv on $HOME. On the typical Pi/Debian install /tmp is tmpfs and $HOME is the SD card, so uv's default hardlink mode fails for every wheel and falls back to a noisy "Failed to hardlink files; falling back to full copy" line. Set UV_LINK_MODE=copy on the install_ansible invocation so the fallback becomes the documented choice. provision_host_agent_venv is unaffected — both its venv and the uv cache live on $HOME, so hardlinks work there. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
Surfaced during e2e testing: after a compose recreate, anthias-server's up -d emitted "Found orphan containers ([anthias-anthias-viewer-run-…]) … you can run this command with the --remove-orphans flag to clean it up." These linger from earlier `docker compose run` invocations that created run-NNN sidecar containers — without --remove-orphans they just keep running and clutter `docker ps`. Apply to both the prod upgrade path (upgrade_containers.sh) and the dev bring-up (start_development_server.sh). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
|
vpetersson
added a commit
that referenced
this pull request
May 12, 2026
* Master's #2881 (e2e-test findings) added an `installer_venv` persistent venv provisioned by `bin/install.sh::provision_host_agent_venv` and kept the systemd unit ExecStart pointed at that stable path. Drops the duplicate ansible-side `.anthias-venv` task and reverts the unit-template path I'd introduced — master's approach is upstream and avoids rewriting the unit path on devices installed pre-refactor. * Master's #2880 replaced the gum UI with whiptail. The auto-merge took whiptail's wiring; my arm64-aware INTRO_MESSAGE / set_device_type / Pi-tag-skip branches survived alongside it. * Master's #2878 viewer locale changes auto-merged with my arm64 start_viewer.sh conditional. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three independent fixes found by running an end-to-end test of master against an x86 device and a Pi 4B side-by-side.
1. Persistent host-agent venv (
bin/install.sh)The installer venv was made ephemeral (mktemp tmpdir, cleaned on EXIT) without updating
anthias-host-agent.service's ExecStart, which still hardcodes/home/${USER}/installer_venv/bin/python. Result on every fresh install since: 203/EXEC restart loop, no IP cache,/api/v2/infoblocks ~80s inget_node_ip()waiting forhost_agent_readythat never lands. Now split into two venvs — ephemeral for ansible-core, persistent for the host-agent — and restart the unit explicitly on upgrade since the in-memory interpreter survives the venv removal.2. Celery beat in-memory scheduler
celery -Bwith the default PersistentScheduler stores its schedule via shelve. On Python 3.13, shelve defaults todbm.sqlite3, which intermittently raisesdbm.sqlite3.error: locking protocol— observed in this build matrix on x86 only. When Beat stalls,reconcile_stuck_processingand the other periodic tasks stop, sois_processing=truerows never get re-dispatched.setup_periodic_tasksdefines every periodic task statically, so a non-persistent scheduler is sufficient — switch tocelery.beat.Schedulerin all three compose files.3. 404 (not 500) for unknown asset_id
AssetView{V1,V1_1,V1_2,V2}.get/put/patch/update,DeleteAssetViewMixin,AssetContentViewMixin, andViewerCurrentAssetViewV1all calledAsset.objects.get(asset_id=...)bare. TheDoesNotExisthad no DRF handler registered → 500 + traceback for what is structurally a missing resource. Route the lookup throughdjango.shortcuts.get_object_or_404(DRF's exception handler converts theHttp404to a clean 404 Response). New parametrised test guards every API version.Test plan
ruff check+ruff format --checkclean on all changed files/home/mvip/installer_venv/on the x86 test rig with the same uv command the new function emits, restartedanthias-host-agent.service— service active,host_agent_ready=truein Redis,/api/v2/inforeturns in 1s (down from 79s)celery -A anthias_server.celery_tasks.celery beat --scheduler celery.beat.Schedulersmoke-tested insideanthias-anthias-celery-1— beat starts cleanly, schedule loads, no shelve/dbm files writtenSample Mpegasset deleted manually from x86 (root cause The 'view playlist' web interface doesn't obey 'end date/time' #2)