Add retry on recoverable exception for the backend#6122
Conversation
|
This pull request introduces 3 alerts and fixes 2 when merging bfe7785 into 704896a - view on LGTM.com new alerts:
fixed alerts:
|
bfe7785 to
1e20ea9
Compare
|
This pull request fixes 2 alerts when merging 1e20ea9 into 704896a - view on LGTM.com fixed alerts:
|
1e20ea9 to
3b06404
Compare
Codecov Report
@@ Coverage Diff @@
## master #6122 +/- ##
==========================================
- Coverage 83.36% 83.19% -0.17%
==========================================
Files 145 145
Lines 17283 17318 +35
Branches 2141 2147 +6
==========================================
Hits 14408 14408
- Misses 2651 2686 +35
Partials 224 224
Continue to review full report at Codecov.
|
|
This pull request introduces 2 alerts and fixes 3 when merging 3b06404 into 704896a - view on LGTM.com new alerts:
fixed alerts:
|
auvipy
left a comment
auvipy
left a comment
There was a problem hiding this comment.
great job so far! can you increase the coverage bit more?
3b06404 to
6c098b5
Compare
|
This pull request fixes 4 alerts when merging 6c098b5 into 52aef4b - view on LGTM.com fixed alerts:
|
This way, a backend implementation is able to take decisions based on current state to store meta in case of failures.
6c098b5 to
752dab5
Compare
|
This pull request introduces 2 alerts and fixes 4 when merging 752dab5 into 52aef4b - view on LGTM.com new alerts:
fixed alerts:
|
acks.late makes celery acknowledge messages only after processing and storing result on the backend. However, in case of backend unreachable, it will shadow a Retry exception and put the task as failed in the backend not retrying the task and acknoledging it on the broker. With this new result_backend_always_retry setting, if the backend exception is recoverable (to be defined per backend implementation), it will retry the backend operation with an exponential backoff.
if existing value in the backend is success, then do nothing. if it is a ready status, then update it only if new value is a ready status as well. else update it. This way, a SUCCESS cannot be overriden so that we do not loose results but any ready state other than success (FAILURE, REVOKED) can be overriden by another ready status (i.e. a SUCCESS)
|
flake8 issues need to be fixed |
752dab5 to
4e9815c
Compare
|
Done. |
|
This pull request introduces 2 alerts when merging 4e9815c into d3863d9 - view on LGTM.com new alerts:
|
acks.late makes celery acknowledge messages only after processing and
storing result on the backend.
However, in case of backend unreachable, it will shadow a Retry
exception and put the task as failed in the backend not retrying the
task and acknoledging it on the broker.
With this new result_backend_always_retry setting, if the backend
exception is recoverable (to be defined per backend implementation),
it will retry the backend operation with an exponential backoff.
In addition, make ES retry storing updates in a better way
if existing value in the backend is success, then do nothing.
if it is a ready status, then update it only if new value is a ready status as well.
else update it.
This way, a SUCCESS cannot be overriden so that we do not loose
results but any ready state other than success (FAILURE, REVOKED) can
be overriden by another ready status (i.e. a SUCCESS)