Skip to content

reboot_anthias never survives a down balena supervisor: RetryError after 5×1s (Sentry ANTHIAS-3F) #3123

Description

@vpetersson

Sentry: ANTHIAS-3F — 60 events in <1 day from a pi4-64 balena device on 2026.7.0+87e8259.

reboot_anthias (src/anthias_server/celery_tasks.py:1135) calls reboot_via_balena_supervisor under Retrying(stop_after_attempt(5), wait_fixed(1)). On this device the supervisor API refuses connections ([Errno 111] Connection refused from get_balena_supervisor_api_response), so after 5 attempts in ~5s tenacity raises RetryError, which lands in Sentry unhandled.

Two problems:

  • 5 × 1s is far too tight for the common cause (supervisor restarting, e.g. right after an OTA). The reboot silently never happens and the operator gets no feedback.
  • A down supervisor is an expected transient on balena; an unhandled RetryError traceback in Sentry is the wrong reporting level.

Fix shape: longer/backoff retry window, catch the terminal failure to log a clear actionable error (and surface it), and consider falling back or re-queuing rather than dropping the reboot on the floor. Same pattern applies to shutdown_anthias.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions