Fix: prioritize request ignore_result over task definition#10184
Conversation
There was a problem hiding this comment.
Pull request overview
This PR fixes result-publishing behavior when ignore_result is overridden at call time (e.g., apply_async(..., ignore_result=False)) but the task/app defaults would otherwise ignore results, addressing #9654 by ensuring request-level ignore_result takes precedence during tracing and error handling.
Changes:
- Added
get_actual_ignore_result()to compute the effectiveignore_resultusing request-level overrides. - Updated tracing/error paths to use the effective
ignore_resultwhen deciding whether to publish/store results. - Added unit tests covering request-overrides-task and request-overrides-config scenarios.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
celery/app/trace.py |
Computes effective ignore_result per-execution and uses it for publish_result and error storage decisions. |
celery/worker/request.py |
Makes Request.store_errors respect request-level ignore_result (via _ignore_result) rather than only task-level configuration. |
t/unit/tasks/test_trace.py |
Adds unit tests asserting request-level ignore_result precedence during tracing. |
ebd4c36 to
78ffc9f
Compare
78ffc9f to
7a4f8f6
Compare
|
please do not force push. |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 3 out of 3 changed files in this pull request and generated 1 comment.
Comments suppressed due to low confidence (1)
celery/app/trace.py:195
handle_error_state()now derivesignore_resultfrom the request override, but there isn’t a focused regression test asserting that request-levelignore_result=False/Truechanges whether failures are stored (i.e., thestore_errorsdecision). Could we add a unit test that exercises a failure path withtask.ignore_result=Trueand a request override ofignore_result=False(and the inverse) to ensure this behavior stays consistent?
def handle_error_state(self, task, req,
eager=False, call_errbacks=True):
ignore_result = get_actual_ignore_result(task, req)
if ignore_result:
store_errors = task.store_errors_even_if_ignored
elif eager and task.store_eager_result:
store_errors = True
else:
store_errors = not eager
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #10184 +/- ##
==========================================
+ Coverage 88.22% 88.23% +0.01%
==========================================
Files 153 153
Lines 19559 19569 +10
Branches 2246 2249 +3
==========================================
+ Hits 17256 17267 +11
+ Misses 2005 2004 -1
Partials 298 298
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. |
auvipy
left a comment
There was a problem hiding this comment.
also, please fix the merge conflicts
This commit adds the missing unit tests to verify the priority logic for 'ignore_result' within the Request class, ensuring it correctly falls back to the task property when not specified in the message.
4755766 to
3e80328
Compare
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated no new comments.
You can also share your feedback on Copilot code review. Take the survey.
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
You can also share your feedback on Copilot code review. Take the survey.
|
@ChickenBenny one last review before merge |
|
LGTM~ |
* fix: prioritize request ignore_result over task definition
* fix: add comment to explain logic of get_actual_ignore_result
* test: add missing tests for Request.ignore_result fallback
This commit adds the missing unit tests to verify the priority logic
for 'ignore_result' within the Request class, ensuring it correctly
falls back to the task property when not specified in the message.
* fix: pre-commit error
* fix: add missing tests for handle_error_state while req is none
* fix(worker): add test to ensure request.store_errors respects task.ignore_result
* fix(docs): clarify fallback behavior for ignore_result and store_errors
* fix(worker): handle ignore_result=None in request headers
---------
Co-authored-by: Asif Saif Uddin {"Auvi":"অভি"} <[email protected]>
* fix: prioritize request ignore_result over task definition
* fix: add comment to explain logic of get_actual_ignore_result
* test: add missing tests for Request.ignore_result fallback
This commit adds the missing unit tests to verify the priority logic
for 'ignore_result' within the Request class, ensuring it correctly
falls back to the task property when not specified in the message.
* fix: pre-commit error
* fix: add missing tests for handle_error_state while req is none
* fix(worker): add test to ensure request.store_errors respects task.ignore_result
* fix(docs): clarify fallback behavior for ignore_result and store_errors
* fix(worker): handle ignore_result=None in request headers
---------
Co-authored-by: Asif Saif Uddin {"Auvi":"অভি"} <[email protected]>
* fix: prioritize request ignore_result over task definition
* fix: add comment to explain logic of get_actual_ignore_result
* test: add missing tests for Request.ignore_result fallback
This commit adds the missing unit tests to verify the priority logic
for 'ignore_result' within the Request class, ensuring it correctly
falls back to the task property when not specified in the message.
* fix: pre-commit error
* fix: add missing tests for handle_error_state while req is none
* fix(worker): add test to ensure request.store_errors respects task.ignore_result
* fix(docs): clarify fallback behavior for ignore_result and store_errors
* fix(worker): handle ignore_result=None in request headers
---------
Co-authored-by: Asif Saif Uddin {"Auvi":"অভি"} <[email protected]>
This MR contains the following updates: | Package | Type | Update | Change | OpenSSF | |---|---|---|---|---| | [celery](https://docs.celeryq.dev/) ([source](https://github.com/celery/celery), [changelog](https://docs.celeryq.dev/en/stable/changelog.html)) | dependencies | patch | `5.6.2` → `5.6.3` | [](https://securityscorecards.dev/viewer/?uri=github.com/celery/celery) | --- ### Release Notes <details> <summary>celery/celery (celery)</summary> ### [`v5.6.3`](https://github.com/celery/celery/releases/tag/v5.6.3) [Compare Source](celery/celery@v5.6.2...v5.6.3) #### What's Changed - Fix Django worker recursion bug + defensive checks for pool\_cls.**module** by [@​maycuatroi1](https://github.com/maycuatroi1) in [#​10048](celery/celery#10048) - Docs: Update user\_preload\_options example to use click. by [@​jorsyk](https://github.com/jorsyk) in [#​10056](celery/celery#10056) - Fix invalid configuration key "bootstrap\_servers" in Kafka demo by [@​jorsyk](https://github.com/jorsyk) in [#​10060](celery/celery#10060) - Fix broken images on PyPI page by [@​Timour-Ilyas](https://github.com/Timour-Ilyas) in [#​10066](celery/celery#10066) - Remove broken reference. by [@​sueannioanis](https://github.com/sueannioanis) in [#​10071](celery/celery#10071) - Removed --dist=loadscope from smoke tests by [@​Nusnus](https://github.com/Nusnus) in [#​10073](celery/celery#10073) - Docs: Clarify task\_retry signal args may be None by [@​GangEunzzang](https://github.com/GangEunzzang) in [#​10076](celery/celery#10076) - Update example for Django by [@​sbc-khacnha](https://github.com/sbc-khacnha) in [#​10081](celery/celery#10081) - Make tests compatible with pymongo >= 4.16 by [@​cjwatson](https://github.com/cjwatson) in [#​10074](celery/celery#10074) - fix: source install of cassandra-driver by [@​Izzette](https://github.com/Izzette) in [#​10105](celery/celery#10105) - fix: register task cross-reference role in Sphinx extension by [@​veeceey](https://github.com/veeceey) in [#​10100](celery/celery#10100) - fix: avoid cycle detection in native delayed delivery by [@​Izzette](https://github.com/Izzette) in [#​10095](celery/celery#10095) - fix(asynpool): avoid AttributeError when proc lacks \_sentinel\_poll by [@​mriddle](https://github.com/mriddle) in [#​10086](celery/celery#10086) - fix dusk\_astronomical horizon sign (+18 -> -18) by [@​Mr-Neutr0n](https://github.com/Mr-Neutr0n) in [#​10121](celery/celery#10121) - Fix/10106 onupdate col use lambda func by [@​ChickenBenny](https://github.com/ChickenBenny) in [#​10108](celery/celery#10108) - Fix warm shutdown RuntimeError with eventlet>=0.37.0 ([#​10083](celery/celery#10083)) by [@​ChickenBenny](https://github.com/ChickenBenny) in [#​10123](celery/celery#10123) - Fix 10109 db backend connection health by [@​ChickenBenny](https://github.com/ChickenBenny) in [#​10124](celery/celery#10124) - Database Backend filter unsupport sql engine arguments with nullpool [#​7355](celery/celery#7355) by [@​ChickenBenny](https://github.com/ChickenBenny) in [#​10134](celery/celery#10134) - fix(beat): correct argument order in Service.**reduce** by [@​bysiber](https://github.com/bysiber) in [#​10137](celery/celery#10137) - ci: declare explicit read-only token permissions in workflow jobs by [@​Rohan5commit](https://github.com/Rohan5commit) in [#​10139](celery/celery#10139) - chore: 'boto3to' to 'boto3 to' by [@​cuiweixie](https://github.com/cuiweixie) in [#​10133](celery/celery#10133) - Database Backend: Add missing index on date\_done (Fixes [#​10097](celery/celery#10097)) by [@​ChickenBenny](https://github.com/ChickenBenny) in [#​10098](celery/celery#10098) - docs: fix typo in CONTRIBUTING.rst by [@​Rohan5commit](https://github.com/Rohan5commit) in [#​10141](celery/celery#10141) - Refer to Flower / Prometheus for monitoring by [@​WilliamDEdwards](https://github.com/WilliamDEdwards) in [#​10140](celery/celery#10140) - docs: remove duplicated words in broker and routing docs by [@​Rohan5commit](https://github.com/Rohan5commit) in [#​10146](celery/celery#10146) - docs: fix stale version reference and grammar in README by [@​kelsonbrito50](https://github.com/kelsonbrito50) in [#​10145](celery/celery#10145) - docs: fix wording in Celery 5.3 worker pool notes by [@​Rohan5commit](https://github.com/Rohan5commit) in [#​10149](celery/celery#10149) - docs: fix duplicated wording in 3.1 changelog entry by [@​Rohan5commit](https://github.com/Rohan5commit) in [#​10152](celery/celery#10152) - docs: fix changelog typo in context manager wording by [@​Rohan5commit](https://github.com/Rohan5commit) in [#​10144](celery/celery#10144) - Fix/10096 worker fails to reconnect after redis failover by [@​ChickenBenny](https://github.com/ChickenBenny) in [#​10151](celery/celery#10151) - Improve on\_after\_finalize signal documentation by [@​Br1an67](https://github.com/Br1an67) in [#​10155](celery/celery#10155) - Add non-commutative example to clarify partial arg ordering in canvas docs by [@​Br1an67](https://github.com/Br1an67) in [#​10157](celery/celery#10157) - Remove redundant test\_isa\_mapping test (fixes [#​10077](celery/celery#10077)) by [@​daniel7an](https://github.com/daniel7an) in [#​10103](celery/celery#10103) - Upgrade pytest-celery to >=1.3.0 and adopt PYTEST\_CELERY\_PKG build arg by [@​Nusnus](https://github.com/Nusnus) in [#​10162](celery/celery#10162) - Remove deprecated args from redis get\_connection call by [@​JaeHyuckSa](https://github.com/JaeHyuckSa) in [#​10036](celery/celery#10036) - Fix [#​6912](celery/celery#6912) rpc backend reconnection error by [@​ChickenBenny](https://github.com/ChickenBenny) in [#​10179](celery/celery#10179) - Fix NameError with TYPE\_CHECKING annotations on Python 3.14+ (PEP 649) by [@​drichardson](https://github.com/drichardson) in [#​10165](celery/celery#10165) - docs: Add elaboration on prefetch multiplier settings (worker\_prefetch\_multiplier) and worker\_eta\_task\_limit by [@​tsangwailam](https://github.com/tsangwailam) in [#​10181](celery/celery#10181) - Fix O(K²) message bloat in a chain of chords by [@​Borzik](https://github.com/Borzik) in [#​10171](celery/celery#10171) - Fix mock connection interfaces to prevent `TypeError` during exception handling by [@​ChickenBenny](https://github.com/ChickenBenny) in [#​10178](celery/celery#10178) - fix(trace): dispatch chain/callbacks on dedup fast-path for redelivered tasks by [@​aurangzaib048](https://github.com/aurangzaib048) in [#​10159](celery/celery#10159) - Extract `reconnect_on_error` to `BaseResultConsumer` by [@​ChickenBenny](https://github.com/ChickenBenny) in [#​10189](celery/celery#10189) - pep 649 by [@​ericbuehl](https://github.com/ericbuehl) in [#​10187](celery/celery#10187) - [Fix#9722](https://github.com/Fix/celery/issues/9722) friendly status errors for CLI by [@​ChickenBenny](https://github.com/ChickenBenny) in [#​10190](celery/celery#10190) - docs: clarify after\_return behavior for retried tasks by [@​KianAnbarestani](https://github.com/KianAnbarestani) in [#​10192](celery/celery#10192) - Add compression header to message protocol docs by [@​Br1an67](https://github.com/Br1an67) in [#​10156](celery/celery#10156) - docs: fix duplicated word in bootsteps comment by [@​Rohan5commit](https://github.com/Rohan5commit) in [#​10153](celery/celery#10153) - Remove outdated autoreloader section from extending docs by [@​Br1an67](https://github.com/Br1an67) in [#​10154](celery/celery#10154) - Fix: prioritize request ignore\_result over task definition by [@​patri27826](https://github.com/patri27826) in [#​10184](celery/celery#10184) - fix: clear the timer while catch the exception by [@​ChickenBenny](https://github.com/ChickenBenny) in [#​10218](celery/celery#10218) - Prepare for release: v5.6.3 by [@​Nusnus](https://github.com/Nusnus) in [#​10221](celery/celery#10221) #### New Contributors - [@​maycuatroi1](https://github.com/maycuatroi1) made their first contribution in [#​10048](celery/celery#10048) - [@​jorsyk](https://github.com/jorsyk) made their first contribution in [#​10056](celery/celery#10056) - [@​Timour-Ilyas](https://github.com/Timour-Ilyas) made their first contribution in [#​10066](celery/celery#10066) - [@​sueannioanis](https://github.com/sueannioanis) made their first contribution in [#​10071](celery/celery#10071) - [@​GangEunzzang](https://github.com/GangEunzzang) made their first contribution in [#​10076](celery/celery#10076) - [@​sbc-khacnha](https://github.com/sbc-khacnha) made their first contribution in [#​10081](celery/celery#10081) - [@​veeceey](https://github.com/veeceey) made their first contribution in [#​10100](celery/celery#10100) - [@​mriddle](https://github.com/mriddle) made their first contribution in [#​10086](celery/celery#10086) - [@​Mr-Neutr0n](https://github.com/Mr-Neutr0n) made their first contribution in [#​10121](celery/celery#10121) - [@​ChickenBenny](https://github.com/ChickenBenny) made their first contribution in [#​10108](celery/celery#10108) - [@​bysiber](https://github.com/bysiber) made their first contribution in [#​10137](celery/celery#10137) - [@​Rohan5commit](https://github.com/Rohan5commit) made their first contribution in [#​10139](celery/celery#10139) - [@​cuiweixie](https://github.com/cuiweixie) made their first contribution in [#​10133](celery/celery#10133) - [@​kelsonbrito50](https://github.com/kelsonbrito50) made their first contribution in [#​10145](celery/celery#10145) - [@​Br1an67](https://github.com/Br1an67) made their first contribution in [#​10155](celery/celery#10155) - [@​daniel7an](https://github.com/daniel7an) made their first contribution in [#​10103](celery/celery#10103) - [@​drichardson](https://github.com/drichardson) made their first contribution in [#​10165](celery/celery#10165) - [@​tsangwailam](https://github.com/tsangwailam) made their first contribution in [#​10181](celery/celery#10181) - [@​Borzik](https://github.com/Borzik) made their first contribution in [#​10171](celery/celery#10171) - [@​aurangzaib048](https://github.com/aurangzaib048) made their first contribution in [#​10159](celery/celery#10159) - [@​ericbuehl](https://github.com/ericbuehl) made their first contribution in [#​10187](celery/celery#10187) - [@​KianAnbarestani](https://github.com/KianAnbarestani) made their first contribution in [#​10192](celery/celery#10192) - [@​patri27826](https://github.com/patri27826) made their first contribution in [#​10184](celery/celery#10184) **Full Changelog**: <celery/celery@v5.6.2...v5.6.3> </details> --- - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box --- This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiI0My45OS4wIiwidXBkYXRlZEluVmVyIjoiNDMuOTkuMCIsInRhcmdldEJyYW5jaCI6Im1haW4iLCJsYWJlbHMiOlsiZGVwZW5kZW5jaWVzIiwicmVub3ZhdGUiXX0=--> See merge request swiss-armed-forces/cyber-command/cea/loom!419
Description
Fixes #9654
Previous if you set
apply_sync(ignore_result=False)with@app.task(ignore_result=True), it would failed to get the task results, because in thetrace.pywe only usetask.ignore_resultas the condition to decide whether to publish result.Add new function
get_actual_ignore_resultto handle the handle the priority lookup:Request > TaskChanges
get_actual_ignore_resultto handle the priority lookup:Request > Task.handle_error_stateto respect these overrides.publish_resultevaluation into the execution flow for better accuracy.Test with three condition
@app.task(ignore_result=True)withapply_sync(ignore_result=False)=> Expected Test ResultExist@app.task(ignore_result=False)withapply_sync(ignore_result=True)=> Expected Test ResultNot Existapp.conf.task_ignore_result = Truewithapply_sync(ignore_result=False)=> Expected Test ResultExistThis PR fixes an inconsistency where
ignore_result=Truetasks would still store error results if the request did not explicitly provide anignore_resultkey.ignore_resultkey, it defaulted toFalsein the worker request, causingstore_errorsto beTrue. This led to failure results being stored even for tasks decorated with@app.task(ignore_result=True).ignore_resultsetting. If a task is configured to ignore results, it will no longer store error results by default unless overridden by the request.User Impact: Users relying on "accidental" error persistence for tasks with
ignore_result=Truewill notice that these errors are no longer stored in the backend.