Skip to content

fix(sentry): also drop transient redis TimeoutError events - #3028

Merged
vpetersson merged 1 commit into
masterfrom
fix/sentry-filter-redis-timeout
Jun 7, 2026
Merged

fix(sentry): also drop transient redis TimeoutError events#3028
vpetersson merged 1 commit into
masterfrom
fix/sentry-filter-redis-timeout

Conversation

@vpetersson

Copy link
Copy Markdown
Contributor

Issues Fixed

Sentry ANTHIAS-1BTimeoutError: Timeout connecting to server in the viewer's resolution reporter; a gap in #3018 found while auditing the post-deploy 2026.6.2+eb1baa6 events.

Description

#3018's before_send filter drops redis.exceptions.ConnectionError, but redis-py's TimeoutError is a sibling of ConnectionError under RedisError — not a subclass. So the same transient redis outage, when the socket hangs instead of refusing, slipped past the filter and reached Sentry. (The build-hash release tags from #3025 are what made this newly-distinguishable event easy to spot in the audit.)

  • Match on (ConnectionError, TimeoutError) from redis.exceptions
  • Add TimeoutError to stubs/redis-stubs/exceptions.pyi
  • Regression test that also asserts the sibling (not subclass) relationship, so the two can't be collapsed into one check later

Checklist

  • I have performed a self-review of my own code.
  • New and existing unit tests pass locally and on CI with my changes.
  • I have done an end-to-end test for Raspberry Pi devices.
  • I have tested my changes for x86 devices.
  • I added a documentation for the changes I have made (when necessary).

🤖 Generated with Claude Code

- The before_send filter caught redis.exceptions.ConnectionError but
  not redis.exceptions.TimeoutError — in redis-py the two are
  siblings under RedisError, not parent/child, so a redis outage that
  hangs the socket (rather than refusing) slipped through to Sentry
- Surfaced post-deploy as ANTHIAS-1B (Timeout connecting to server,
  viewer resolution reporter) once the build-hash release tags made
  it identifiable
- Match on both types; add the redis-stubs TimeoutError entry and a
  regression test that also pins the sibling (not subclass) relation

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>
@vpetersson
vpetersson requested a review from a team as a code owner June 7, 2026 19:03
@vpetersson vpetersson self-assigned this Jun 7, 2026
@vpetersson
vpetersson requested a review from Copilot June 7, 2026 19:03
@sonarqubecloud

sonarqubecloud Bot commented Jun 7, 2026

Copy link
Copy Markdown

@vpetersson
vpetersson merged commit fe942a5 into master Jun 7, 2026
10 checks passed

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR tightens the Sentry before_send noise filter to also drop transient Redis socket TimeoutError events (a sibling of ConnectionError in redis-py), preventing expected Redis blips from being reported to Sentry.

Changes:

  • Update _sentry_before_send to treat both redis.exceptions.ConnectionError and redis.exceptions.TimeoutError as transient Redis noise.
  • Add a regression test ensuring TimeoutError is filtered and asserting the sibling (non-subclass) relationship to ConnectionError.
  • Extend the local redis stub exceptions to include TimeoutError for type checking.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.

File Description
src/anthias_server/django_project/settings.py Expands transient Redis exception filtering to include TimeoutError via a shared tuple in the before_send hook.
tests/test_sentry.py Adds regression coverage for dropping Redis TimeoutError and locks in the intended redis-py exception hierarchy assumption.
stubs/redis-stubs/exceptions.pyi Adds TimeoutError to the redis exception stubs to match the newly-referenced exception type.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

vpetersson added a commit that referenced this pull request Jun 9, 2026
- CalVer (YYYY.0M.MICRO); still June 2026, micro 2 -> 3
- Gives Sentry a real release boundary: every build since 2026.6.2
  reported the same base version (only the +git-hash differed), so
  resolved-in-next-release never stuck and fixed issues kept
  reopening on the next event. A version bump lets the deployed
  fixes actually clear from the board.
- Ships the crash/noise fixes merged since 2026.6.2: SQLite WAL +
  busy timeout (#3015), celery migration-gate (#3016) and
  asset-probe soft limits (#3017), transient-redis/CancelledError
  Sentry filtering + redis healthcheck (#3018/#3028), GitHub
  update-check log level (#3019), webview respawn on D-Bus death at
  setup and mid-play (#3020/#3031), resilient static-file scan
  (#3026), Wayland-socket wait (#3030), and Sentry release/board
  triage tags (#3021/#3025)

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
@jeff-johnson

Copy link
Copy Markdown

When performing a full clean install of 2026.7.1 (after zero configuration), I get the following error repeating in the logs of anthias-anthias-server-1:

...
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/venv/lib/python3.13/site-packages/redis/asyncio/connection.py", line 783, in read_response
    response = await self._parser.read_response(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        disable_decoding=disable_decoding, push_request=push_request
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/venv/lib/python3.13/site-packages/redis/_parsers/resp3.py", line 185, in read_response
    response = await self._read_response(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
        disable_decoding=disable_decoding, push_request=push_request
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/venv/lib/python3.13/site-packages/redis/_parsers/resp3.py", line 197, in _read_response
    raw = await self._readline()
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.13/site-packages/redis/_parsers/base.py", line 578, in _readline
    data = await self._stream.readline()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/asyncio/streams.py", line 562, in readline
    line = await self.readuntil(sep)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/asyncio/streams.py", line 677, in readuntil
    await self._wait_for_data('readuntil')
  File "/usr/lib/python3.13/asyncio/streams.py", line 539, in _wait_for_data
    await self._waiter
asyncio.exceptions.CancelledError

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/venv/lib/python3.13/site-packages/redis/asyncio/connection.py", line 782, in read_response
    async with async_timeout(read_timeout):
               ~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/asyncio/timeouts.py", line 116, in __aexit__
    raise TimeoutError from exc_val
TimeoutError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/venv/lib/python3.13/site-packages/uvicorn/protocols/websockets/websockets_sansio_impl.py", line 350, in run_asgi
    result = await self.app(self.scope, self.receive, self.send)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.13/site-packages/uvicorn/middleware/proxy_headers.py", line 63, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.13/site-packages/channels/routing.py", line 48, in __call__
    return await application(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.13/site-packages/channels/security/websocket.py", line 37, in __call__
    return await self.application(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.13/site-packages/channels/routing.py", line 118, in __call__
    return await application(
           ^^^^^^^^^^^^^^^^^^
    ...<10 lines>...
    )
    ^
  File "/venv/lib/python3.13/site-packages/channels/consumer.py", line 95, in app
    return await consumer(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/venv/lib/python3.13/site-packages/channels/consumer.py", line 58, in __call__
    await await_many_dispatch(
        [receive, self.channel_receive], self.dispatch
    )
  File "/venv/lib/python3.13/site-packages/channels/utils.py", line 57, in await_many_dispatch
    await task
redis.exceptions.TimeoutError: Timeout reading from redis:6379
...

Is this the same condition this pull was to fix or is this something different?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants