fix(worker): add hub.reset() to synloop on connection error by antonio-mello-ai · Pull Request #10204 · celery/celery

antonio-mello-ai · 2026-03-15T14:15:48Z

Summary

Add hub.reset() cleanup to the synloop (used by gevent/eventlet pools) when a connection error occurs, matching the cleanup already present in asynloop.

Root cause

When a broker connection drops during asynloop, hub.reset() is called in the except Exception block (loops.py:107-108), cleaning up stale file descriptors and callbacks. The synloop lacked this cleanup entirely. With gevent/eventlet pools, stale state from the old connection persisted across reconnection attempts, preventing proper consumer re-registration.

This is especially problematic because gevent disables broker_connection_timeout (consumer.py:230-234), meaning a silent Redis death can leave the worker stuck in drain_events indefinitely. When it eventually reconnects, the stale hub state prevents the new consumers from registering properly.

Fix

Wrap the synloop main loop in try/except Exception and call hub.reset() on error, matching the asynloop pattern. The hub is guarded against None since it may not always be available in the sync path.

Companion PR

Kombu: fix(redis): do not remove on_poll_start from on_tick on disconnect kombu#2492 — fixes the race condition in Redis transport _on_disconnect that removes the on_poll_start callback from on_tick, preventing consumer re-registration after reconnection.

Together, these two PRs address the "catatonic worker" problem from both sides:

Kombu (this PR's companion): prevents the race condition that removes the poll callback
Celery (this PR): ensures stale state is cleaned up in the gevent/eventlet code path

Related issues

Fixes Worker stops consuming tasks after redis reconnection (gevent) #9191 — gevent-specific catatonic worker (open)
Related: Worker stops consuming tasks after redis reconnection on celery 5.2.3 #8030 — original report (closed but not fully fixed)
Related: Celery consumers losing connection behind redis haproxy #9631 — Redis + HAProxy variant (open)
Related: worker stops consuming tasks after rabbitmq reconnection on celery 5 #9095 — Discussion with reports through Nov 2025 on Celery 5.5.3

Test plan

test_hub_reset_on_connection_error — verifies hub.reset() is called on socket.error
test_hub_not_reset_on_graceful_shutdown — verifies no reset on normal exit
test_hub_reset_with_none_hub — verifies safe handling when hub is None
All 48 loop tests pass (asynloop + synloop + quick_drain)

The asynloop already calls hub.reset() when an exception occurs during the event loop, cleaning up stale file descriptors and callbacks from the old connection. The synloop (used by gevent/eventlet pools) lacked this cleanup, leaving stale state that could prevent consumer re-registration after broker reconnection. This adds the same hub.reset() pattern from asynloop to synloop, guarded by a None check since hub may not always be available in the sync path. Three new tests verify: reset on error, no reset on graceful shutdown, and safe handling when hub is None. Fixes #9191 Related: #8030, #9631 Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

auvipy · 2026-03-15T18:52:38Z

@ChickenBenny

Copilot

Pull request overview

This PR adds hub.reset() cleanup to the synloop (used by gevent/eventlet pools) when a connection error occurs, matching the existing cleanup pattern in asynloop. This addresses "catatonic worker" issues where stale hub state from a dropped broker connection prevented proper consumer re-registration after reconnection.

Changes:

Wrap the synloop main loop in try/except Exception to call hub.reset() on error, with a None guard since the hub may not always be available in the sync path
Add three focused unit tests covering: hub reset on connection error, no reset on graceful shutdown, and safe handling when hub is None

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated no comments.

File	Description
celery/worker/loops.py	Adds `hub.reset()` error handling to `synloop`, mirroring `asynloop`'s cleanup pattern
t/unit/worker/test_loops.py	Adds three tests for the new hub reset behavior in synloop

You can also share your feedback on Copilot code review. Take the survey.

codecov · 2026-03-15T18:57:18Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.24%. Comparing base (c38600d) to head (03ab494).
⚠️ Report is 71 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #10204   +/-   ##
=======================================
  Coverage   88.23%   88.24%           
=======================================
  Files         153      153           
  Lines       19573    19581    +8     
  Branches     2249     2250    +1     
=======================================
+ Hits        17270    17279    +9     
  Misses       2004     2004           
+ Partials      299      298    -1

Flag	Coverage Δ
unittests	`88.21% <100.00%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

…s in synloop Add test_hub_reset_error_logs_exception to assert the error message is actually logged, complementing the existing test that only verified hub.reset() was called. Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

themavik

Reviewed the changes — the fix is minimal and targeted. Good contribution.

themavik

Reviewed the changes — the approach looks correct and addresses the reported issue.

themavik

Calling hub.reset() from synloop on errors mirrors asynloop cleanup so gevent/eventlet pools do not keep dead callbacks across reconnect (#9191). nit: if hub.reset() itself fails you log that exception but still re-raise the original loop error, so ops have to grep logs for reset failures separately.

Celery 5.7.0 includes upstream fixes for the catatonic worker bug where workers reconnect after a Redis broker restart but fail to re-register their queue consumers (celery/celery#10204, celery/kombu#2492). This replaces the previous worker-health-check CLI approach with a version bump, as recommended by maintainers.

This was referenced Mar 15, 2026

fix(redis): do not remove on_poll_start from on_tick on disconnect celery/kombu#2492

Merged

feat(providers/celery): bump minimum celery to 5.7.0 for catatonic worker fix apache/airflow#63583

Closed

auvipy requested review from auvipy and Copilot March 15, 2026 18:52

Copilot started reviewing on behalf of auvipy March 15, 2026 18:52 View session

Copilot AI reviewed Mar 15, 2026

View reviewed changes

antonio-mello-ai and others added 2 commits March 17, 2026 08:45

test(worker): add coverage for hub.reset() error path in synloop

c2c4a68

Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

themavik reviewed Mar 19, 2026

View reviewed changes

themavik reviewed Mar 23, 2026

View reviewed changes

Merge branch 'main' into fix/synloop-hub-reset-on-reconnect

ade8ee4

auvipy added this to the 5.7.0 milestone Mar 26, 2026

auvipy approved these changes Mar 26, 2026

View reviewed changes

auvipy added Component: Redis Broker Worker labels Mar 26, 2026

Merge branch 'main' into fix/synloop-hub-reset-on-reconnect

03ab494

auvipy merged commit c167ccf into celery:main Mar 26, 2026
16 checks passed

auvipy added the PR Type: Bugfix label Mar 26, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(worker): add hub.reset() to synloop on connection error#10204

fix(worker): add hub.reset() to synloop on connection error#10204
auvipy merged 5 commits into
celery:mainfrom
antonio-mello-ai:fix/synloop-hub-reset-on-reconnect

antonio-mello-ai commented Mar 15, 2026

Uh oh!

auvipy commented Mar 15, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

codecov Bot commented Mar 15, 2026 •

edited

Loading

Uh oh!

themavik left a comment

Uh oh!

themavik left a comment

Uh oh!

themavik left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Uh oh!

Conversation

antonio-mello-ai commented Mar 15, 2026

Summary

Root cause

Fix

Companion PR

Related issues

Test plan

Uh oh!

auvipy commented Mar 15, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

codecov Bot commented Mar 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

themavik left a comment

Choose a reason for hiding this comment

Uh oh!

themavik left a comment

Choose a reason for hiding this comment

Uh oh!

themavik left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

codecov Bot commented Mar 15, 2026 •

edited

Loading