Fix mock connection interfaces to prevent `TypeError` during exception handling by ChickenBenny · Pull Request #10178 · celery/celery

ChickenBenny · 2026-03-06T04:23:25Z

Description

Fixes #10177. Relates to kombu PR #2473.

Several unit tests passed a bare Mock() as the broker connection object, leaving connection_errors and channel_errors as auto-generated Mock attributes rather than tuples. Kombu's ignore_errors helper (and similar exception-handling paths) performs tuple concatenation (connection_errors + channel_errors), which results in:

TypeError: unsupported operand type(s) for +: 'Mock' and 'Mock'

This was recently exposed by kombu PR #2473 and broke Celery CI.

Explicitly set connection_errors = () and channel_errors = () on mock connection objects in three test files:

t/unit/worker/test_bootsteps.py

test_start_stop_shutdown and test_start_no_consumers — both had bare Mock() connections missing connection_errors and channel_errors.

t/unit/worker/test_consumer.py

Five test_disable_prefetch_* test methods — all used consumer.connection = Mock() without the error tuple attributes.

t/unit/worker/test_worker.py

LoopConsumer() helper — shared by the majority of test_Consumer tests.
_get_on_message() helper — also shared across multiple tests.

Fixing the mocks directly is more robust than relying solely on lazy evaluation inside Kombu (proposed in kombu #2478). If any Celery test legitimately triggers an exception path that evaluates those attributes, the bare Mock() approach would crash with the same TypeError. Explicit empty tuples correctly reflect the Kombu Connection interface contract and make the intent clear to future readers.

codecov · 2026-03-06T04:27:24Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 88.12%. Comparing base (df57d9a) to head (30efd26).
⚠️ Report is 74 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main   #10178   +/-   ##
=======================================
  Coverage   88.12%   88.12%           
=======================================
  Files         153      153           
  Lines       19501    19501           
  Branches     2242     2242           
=======================================
+ Hits        17185    17186    +1     
  Misses       2017     2017           
+ Partials      299      298    -1

Flag	Coverage Δ
unittests	`88.10% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

ChickenBenny · 2026-03-06T11:29:30Z

@auvipy The smoke test failure seems to be an environment issue rather than a code bug.

It looks like the CI container is missing the system library for memcached (libmemcached-dev), which causes the pylibmc wheel build to fail with memcached.h: No such file or directory.

Here is the error log:

 Building wheels for collected packages: kombu, pylibmc, softlayer_messaging, pyArango
    Building wheel for kombu (pyproject.toml): started
    Building wheel for kombu (pyproject.toml): finished with status 'done'
    Created wheel for kombu: filename=kombu-5.6.2-py3-none-any.whl size=224936 sha256=2c737bf4fa784c1e2db88b8e935a0c1a5f1d5a282eeaf079580c9ba683f26d66
    Stored in directory: /tmp/pip-ephem-wheel-cache-h_b_xvc7/wheels/5d/4e/d8/eec12e40993b3dc07ff637e9f044822e431fc5861fb70a01a7
    Building wheel for pylibmc (pyproject.toml): started
    Building wheel for pylibmc (pyproject.toml): finished with status 'error'
    error: subprocess-exited-with-error
    
    × Building wheel for pylibmc (pyproject.toml) did not run successfully.
    │ exit code: 1
    ╰─> [8 lines of output]
        warning: build_py: byte-compiling is disabled, skipping.
        
        In file included from src/_pylibmcmodule.c:34:
        src/_pylibmcmodule.h:42:10: fatal error: libmemcached/memcached.h: No such file or directory
           42 | #include <libmemcached/memcached.h>
              |          ^~~~~~~~~~~~~~~~~~~~~~~~~~
        compilation terminated.
        error: command '/usr/bin/gcc' failed with exit code 1
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
    ERROR: Failed building wheel for pylibmc
    Building wheel for softlayer_messaging (pyproject.toml): started
    Building wheel for softlayer_messaging (pyproject.toml): finished with status 'done'
    Created wheel for softlayer_messaging: filename=softlayer_messaging-1.0.3-py3-none-any.whl size=7745 sha256=cc9131e6fc02c3313f5c826ff8d077d4b83e8c37f89b061a20de15a546bbc939
    Stored in directory: /home/runner/.cache/pip/wheels/dd/b6/40/3fadf00806d15991d09c6f606b844bd071aa23691b9f042caa
    Building wheel for pyArango (pyproject.toml): started
    Building wheel for pyArango (pyproject.toml): finished with status 'done'
    Created wheel for pyArango: filename=pyarango-2.1.1-py2.py3-none-any.whl size=53327 sha256=2d418e3e6a8587247233d24feb944025cf37b95a9335d53dd09b5c54546defe9
    Stored in directory: /home/runner/.cache/pip/wheels/c9/11/b7/21c814c4eb347988b571d899d691067ea7cd7d0193cdaaa79e
  Successfully built kombu softlayer_messaging pyArango
  Failed to build pylibmc
  error: failed-wheel-build-for-install
  
  × Failed to build installable wheels for some pyproject.toml based projects
  ╰─> pylibmc
  3.12-smoke: 19955 C exit 1 (19.42 seconds) /home/runner/_work/celery/celery> python -I -m pip install 'pytest-xdist>=3.5' -r

auvipy · 2026-03-07T11:23:48Z

ok thanks

Copilot

Pull request overview

This PR hardens Celery’s worker unit tests by ensuring mock broker connection objects implement Kombu’s expected Connection interface for connection_errors and channel_errors, preventing TypeError during exception-handling paths (as exposed by recent Kombu changes).

Changes:

Set connection_errors = () and channel_errors = () on shared/mock connection objects in worker-related unit tests.
Update helper-style test scaffolding (LoopConsumer, _get_on_message) so most consumer/worker tests inherit the corrected mock interface.
Apply the same fix across multiple disable_prefetch_* consumer tests that used bare Mock() connections.

Reviewed changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.

File	Description
`t/unit/worker/test_worker.py`	Ensures shared test consumer connection mocks define `connection_errors`/`channel_errors` to avoid Kombu `ignore_errors` tuple-concat failures.
`t/unit/worker/test_consumer.py`	Adds the missing error tuple attributes to mock connections used by `disable_prefetch_*` tests.
`t/unit/worker/test_bootsteps.py`	Updates bootstep ConsumerStep tests so the mock connection matches Kombu’s expected interface.

* Fix celery#6912 rpc backend reconnection error (celery#10179) * feat: add handle connection errors retry function * fix: wrap the drain event with handle connection errors * fix: catch the oserror for retry the connection * test: drain events connection and channel errors * test: drainer without greenlet * fix: sleep after catch the os error * fix: move the importorskip to module-level calls * Update celery/backends/rpc.py Co-authored-by: Copilot <[email protected]> * fix: start the comsumer with self._no_ack --------- Co-authored-by: Asif Saif Uddin {"Auvi":"অভি"} <[email protected]> * Fix NameError with TYPE_CHECKING annotations on Python 3.14+ (PEP 649) (celery#10165) * Fix NameError with TYPE_CHECKING annotations on Python 3.14+ (PEP 649) In Python 3.14, annotations are deferred by default (PEP 649). Two sites in Celery eagerly triggered annotation evaluation, raising NameError for types only imported under TYPE_CHECKING: 1. `fun.__annotations__` in `_task_from_fun` (celery/app/base.py): Use `inspect.get_annotations(fun, format=Format.STRING)` on 3.14+ to return annotations as strings without evaluation. 2. `inspect.getfullargspec(fun)` in `head_from_fun` (utils/functional.py): On 3.14+, getfullargspec internally evaluates annotations and raises TypeError for unresolvable names. Fall back to reading the function's __code__ object directly (which is annotation-free) when that happens. Fixes celery#10099 Assisted by AI * Remove unused annotationlib import in functional.py * Simplify _getfullargspec: use inspect.signature with Format.STRING Replace the __code__ introspection fallback with a cleaner approach: use inspect.signature(target, annotation_format=Format.STRING) which avoids annotation evaluation and handles all argument kinds directly. For bound methods, pass __func__ so that 'self' is included in args, preserving the behaviour of getfullargspec on older Python versions. * Skip PEP 649 regression tests on Python < 3.14 These tests exercise deferred annotation evaluation introduced by PEP 649, which is only the default on Python 3.14+. Skipping on earlier versions avoids NameError at exec time when annotations are evaluated eagerly. Assisted by AI * Update celery/app/base.py --------- Co-authored-by: Asif Saif Uddin {"Auvi":"অভি"} <[email protected]> * docs: Add elaboration on prefetch multiplier settings (worker_prefetch_multiplier) and worker_eta_task_limit (celery#10181) * docs: add description for prefetch multiplier related to worker_eta_task_limit * docs: add description for prefetch multiplier related to worker_eta_task_limit * Update docs/userguide/configuration.rst * Update docs/glossary.rst * Update docs/userguide/configuration.rst * Update docs/glossary.rst --------- Co-authored-by: Asif Saif Uddin {"Auvi":"অভি"} <[email protected]> * Fix O(K²) message bloat in a chain of chords (celery#10171) * fix multiple chord chaining * Apply suggestions from code review * fix test name --------- Co-authored-by: Asif Saif Uddin {"Auvi":"অভি"} <[email protected]> * test: mock the channel and connection errors in a more elegant way (celery#10178) Co-authored-by: Asif Saif Uddin {"Auvi":"অভি"} <[email protected]> * fix(trace): dispatch chain/callbacks on dedup fast-path for redelivered tasks (celery#10159) * fix(trace): dispatch chain/callbacks on dedup fast-path for redelivered tasks (celery#9835) When a redelivered task is deduplicated via backend state check, the dedup fast-path returned immediately without dispatching the chain or callbacks from the message. This caused chains to be permanently lost when a worker crashed after mark_as_done but before the broker ack. Also replace chain.pop() with non-mutating chain[-1] + chain[:-1] to prevent in-place corruption of task_request.chain on retry/redelivery. Closes celery#9835 * fix(trace): harden dedup dispatch and add successful_requests guard (celery#9835) - Move successful_requests.add() inside try block so failed dispatches allow retry on next redelivery - Handle group callbacks in dedup path matching normal success path (Issue celery#1936) - Pass priority to callback dispatch in dedup path - Re-raise MemoryError from dedup dispatch exception handler - Escalate dispatch failure log from warning to error - Remove leftover fault injection debug code from chain dispatch - Add tests for multi-element chain, successful_requests guard, combined chain+callbacks, and dispatch failure logging * fix(test): clean up successful_requests and align task_id in dedup tests - Add successful_requests.discard(task_id) to tests that trigger the backend-dedup path to avoid leaking global state - Pass task_id to trace() in test_chain_dispatch_does_not_mutate to match request['id'] * fix(trace): guard dedup dispatch with stored children to avoid duplicates Skip chain/callback dispatch when the stored result meta already contains children, indicating the previous worker successfully dispatched them before storing the result. * fix(test): patch maybe_signature in chain mutation test to avoid side effects * fix(trace): match normal path dispatch order in dedup branch (callbacks before chain) * fix(test): add test for children guard skipping dedup dispatch * Update celery/app/trace.py * fix(trace): raise Reject(requeue=True) on dedup dispatch failure instead of acking * fix(trace): track push_task/push_request separately to unwind partial pushes * refactor(trace): extract shared dispatch helper, remove dead children persistence Extract callback/chain dispatch logic into _dispatch_callbacks_and_chain() helper inside build_tracer, called from both the normal success path and the dedup fast-path. This eliminates duplicated dispatch code that could drift between the two paths. Remove the push_task/push_request context wrapper and children persistence block from the dedup path — store_result is silently blocked by the KV-backend SUCCESS guard so children never persist, and the _meta dict mutation was a local no-op. * test(trace): cover MemoryError propagation and Reject passthrough in dedup path Add tests for two uncovered lines flagged by Codecov: - MemoryError during dedup dispatch propagates directly (not wrapped in Reject) - Reject from dedup dispatch propagates through the module-level trace_task wrapper * test(integration): add integration tests for dedup chain/callback dispatch Add 4 integration tests exercising the full worker pipeline with Redis broker/backend to verify chain and callback dispatch on the dedup fast-path (issue celery#9835). New integration tasks: store_success_then_reject (simulates redelivery by storing SUCCESS then raising Reject) and reject_then_succeed (tests normal reject passthrough). Also address review findings: consolidate redundant _get_task_meta() calls, document the _children guard and partial-dispatch limitations, strengthen root_id assertion, and add unit tests for root_id fallback and empty-chain edge case. * refactor(test): move dedup integration tests to standalone file, fix CI - Move test_dedup_chain_dispatch to its own file following test_prefork_shutdown.py pattern (celery_session_app + start_worker) - Add test_dedup_chain_dispatch.py to CI integration test matrix - Extract flaky/timeout boilerplate from test_canvas.py to conftest.py - Fix skip condition to use backend.persistent instead of startswith('redis') - Wrap dedup_worker fixture teardown in try/finally - Fix callback test to assert actual callback result via AsyncResult - Move inline Reject import to module level in tasks.py - Add unit test for backend-read failure during dedup check * style: remove trailing blank line in test_canvas.py * Update t/integration/test_dedup_chain_dispatch.py --------- Co-authored-by: Asif Saif Uddin {"Auvi":"অভি"} <[email protected]> --------- Co-authored-by: ChickenBenny <[email protected]> Co-authored-by: Asif Saif Uddin {"Auvi":"অভি"} <[email protected]> Co-authored-by: Doug Richardson <[email protected]> Co-authored-by: William <[email protected]> Co-authored-by: Feliks Borzik <[email protected]> Co-authored-by: Rana Aurangzaib <[email protected]>

…10178) Co-authored-by: Asif Saif Uddin {"Auvi":"অভি"} <[email protected]>

test: mock the channel and connection errors in a more elegant way

9efa653

ChickenBenny mentioned this pull request Mar 6, 2026

Fix/2463 gevent concurrent error celery/kombu#2478

Merged

Merge branch 'main' into test/10177-improve-mock-connection-interfaces

68f7c02

ChickenBenny force-pushed the test/10177-improve-mock-connection-interfaces branch from 05e6f90 to 68f7c02 Compare March 6, 2026 09:09

auvipy requested review from auvipy and Copilot March 7, 2026 11:22

auvipy added this to the 5.7.0 milestone Mar 7, 2026

Copilot started reviewing on behalf of auvipy March 7, 2026 11:23 View session

Copilot AI reviewed Mar 7, 2026

View reviewed changes

Comment thread t/unit/worker/test_bootsteps.py

Merge branch 'main' into test/10177-improve-mock-connection-interfaces

30efd26

auvipy approved these changes Mar 8, 2026

View reviewed changes

auvipy merged commit ffc1aaa into celery:main Mar 8, 2026
338 checks passed

auvipy added the PR Type: Test Case label Mar 8, 2026

ChickenBenny mentioned this pull request Mar 10, 2026

BaseResultConsumer is missing a shared reconnect_on_error contract #10188

Closed

Nusnus pushed a commit that referenced this pull request Mar 26, 2026

test: mock the channel and connection errors in a more elegant way (#…

06565ab

…10178) Co-authored-by: Asif Saif Uddin {"Auvi":"অভি"} <[email protected]>

Nusnus pushed a commit that referenced this pull request Mar 26, 2026

test: mock the channel and connection errors in a more elegant way (#…

fa82af1

…10178) Co-authored-by: Asif Saif Uddin {"Auvi":"অভি"} <[email protected]>

Nusnus pushed a commit that referenced this pull request Mar 26, 2026

test: mock the channel and connection errors in a more elegant way (#…

37f16e0

…10178) Co-authored-by: Asif Saif Uddin {"Auvi":"অভি"} <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

Fix mock connection interfaces to prevent `TypeError` during exception handling#10178

Fix mock connection interfaces to prevent `TypeError` during exception handling#10178
auvipy merged 3 commits into
celery:mainfrom
ChickenBenny:test/10177-improve-mock-connection-interfaces

ChickenBenny commented Mar 6, 2026 •

edited

Loading

Uh oh!

codecov Bot commented Mar 6, 2026 •

edited

Loading

Uh oh!

ChickenBenny commented Mar 6, 2026 •

edited

Loading

Uh oh!

auvipy commented Mar 7, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

ChickenBenny commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Uh oh!

codecov Bot commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ChickenBenny commented Mar 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

auvipy commented Mar 7, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ChickenBenny commented Mar 6, 2026 •

edited

Loading

codecov Bot commented Mar 6, 2026 •

edited

Loading

ChickenBenny commented Mar 6, 2026 •

edited

Loading