fix(celery): gate worker startup on applied database migrations - #3016
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds a Celery worker startup gate to prevent the worker from consuming tasks until Django database migrations have been fully applied, addressing a race during upgrades/first boot that caused OperationalError: no such table: assets (Sentry ANTHIAS-1).
Changes:
- Add a migration-readiness probe (
_migrations_ready) that mirrorsmanage.py migrate --checksemantics and treats probe errors as “not ready”. - Register a
worker_inithook (wait_for_migrations) that blocks worker startup until migrations are ready. - Add unit tests covering readiness detection and the blocking/unblocking behavior.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.
| File | Description |
|---|---|
src/anthias_server/celery_tasks.py |
Adds migration readiness polling and a worker_init startup gate to delay worker consumption until the schema is ready. |
tests/test_celery_tasks.py |
Adds tests for migration readiness probing and the worker init waiting loop. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
- The celery container boots in parallel with anthias-server, whose start script is still running dbbackup -> migrate (or the dbrestore fallback that drops and re-creates every table) - A task replayed off the Redis broker in that window died with OperationalError: no such table: assets — one burst per device on every upgrade/first boot (Sentry ANTHIAS-1) - Block in worker_init until the unapplied-migration plan is empty; tasks stay queued in the broker while waiting, so nothing is lost - Treat probe errors (database locked, missing django_migrations table) as not-ready instead of raising - Works for every deployment topology (compose, balena, dev, test) without touching the five compose files that spell the worker CMD Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…ning - Log the migration-readiness probe's underlying error at DEBUG so a persistent non-transient cause is identifiable from device logs - Repeat the waiting warning every 30s instead of every 5s poll - Patch time.sleep by dotted path for strict mypy Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
- Catch django.db.utils.DatabaseError in the readiness probe; a programming bug now fails fast instead of parking the worker in an infinite wait - Reword the docstring to be accurate about DEBUG-level visibility - Add a fail-fast regression test Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…t warning Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
…be logs Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
vpetersson
force-pushed
the
fix/celery-wait-for-migrations
branch
from
June 7, 2026 12:41
2ff314b to
24042bd
Compare
|
5 tasks
vpetersson
added a commit
that referenced
this pull request
Jun 9, 2026
- CalVer (YYYY.0M.MICRO); still June 2026, micro 2 -> 3 - Gives Sentry a real release boundary: every build since 2026.6.2 reported the same base version (only the +git-hash differed), so resolved-in-next-release never stuck and fixed issues kept reopening on the next event. A version bump lets the deployed fixes actually clear from the board. - Ships the crash/noise fixes merged since 2026.6.2: SQLite WAL + busy timeout (#3015), celery migration-gate (#3016) and asset-probe soft limits (#3017), transient-redis/CancelledError Sentry filtering + redis healthcheck (#3018/#3028), GitHub update-check log level (#3019), webview respawn on D-Bus death at setup and mid-play (#3020/#3031), resilient static-file scan (#3026), Wayland-socket wait (#3030), and Sentry release/board triage tags (#3021/#3025) Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issues Fixed
Sentry: ANTHIAS-1 (
OperationalError: no such table: assetsinnormalize_video_asset— 112 events, one burst per device on every upgrade/first boot).Description
The celery container boots in parallel with anthias-server, whose start script is still running its
dbbackup→migratepass (or thedbrestorefallback, which drops and re-creates every table). A task replayed off the Redis broker in that window dies withno such table: assets, and itson_failurecleanup then fails the same way.worker_initgate that blocks the worker until the unapplied-migration plan is empty (same check asmanage.py migrate --check)django_migrationstable on an empty DB) count as not-ready instead of raisingChecklist
🤖 Generated with Claude Code