Repository navigation
Commit 78a66bd
committed
Don't fail a task whose SUCCESS was stored before the worker was lost
When the pool child stores SUCCESS and then dies (OOM kill, SIGKILL, a
crash in on_success, a task_postrun receiver or process_cleanup), the
parent gets WorkerLostError and Request.on_failure calls mark_as_failure
unconditionally. That fires the errbacks for a task that succeeded,
counts a chord header twice, and with reject_on_worker_lost requeues a
task that already ran.
Check the stored state first. If it is SUCCESS, acknowledge (acks_late)
and return without recording a failure. If the backend can't be read,
keep the existing failure handling.
Refs #107241 parent 6bc4220 commit 78a66bd
2 files changed
Lines changed: 69 additions & 0 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
705 | 705 | | |
706 | 706 | | |
707 | 707 | | |
| 708 | + | |
| 709 | + | |
| 710 | + | |
| 711 | + | |
| 712 | + | |
| 713 | + | |
| 714 | + | |
| 715 | + | |
| 716 | + | |
| 717 | + | |
| 718 | + | |
708 | 719 | | |
709 | 720 | | |
710 | 721 | | |
| |||
761 | 772 | | |
762 | 773 | | |
763 | 774 | | |
| 775 | + | |
| 776 | + | |
| 777 | + | |
| 778 | + | |
| 779 | + | |
| 780 | + | |
| 781 | + | |
764 | 782 | | |
765 | 783 | | |
766 | 784 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
962 | 962 | | |
963 | 963 | | |
964 | 964 | | |
| 965 | + | |
| 966 | + | |
| 967 | + | |
| 968 | + | |
| 969 | + | |
| 970 | + | |
| 971 | + | |
| 972 | + | |
| 973 | + | |
| 974 | + | |
| 975 | + | |
| 976 | + | |
| 977 | + | |
| 978 | + | |
| 979 | + | |
| 980 | + | |
| 981 | + | |
| 982 | + | |
| 983 | + | |
| 984 | + | |
| 985 | + | |
| 986 | + | |
| 987 | + | |
| 988 | + | |
| 989 | + | |
| 990 | + | |
| 991 | + | |
| 992 | + | |
| 993 | + | |
| 994 | + | |
| 995 | + | |
| 996 | + | |
| 997 | + | |
| 998 | + | |
| 999 | + | |
| 1000 | + | |
| 1001 | + | |
| 1002 | + | |
| 1003 | + | |
| 1004 | + | |
| 1005 | + | |
| 1006 | + | |
| 1007 | + | |
| 1008 | + | |
| 1009 | + | |
| 1010 | + | |
| 1011 | + | |
| 1012 | + | |
| 1013 | + | |
| 1014 | + | |
| 1015 | + | |
965 | 1016 | | |
966 | 1017 | | |
967 | 1018 | | |
| |||
0 commit comments