Hello,

We are currently using celery with Broker=RabbitMQ & Backend=Redis. Our setup runs on Kubernetes with auto-scaling enabled. (Our number of pods scale up / down based on resource utilization). We use the Forked Process mode to run our celery worker.

When the size of our cluster increases, everything works great. However, when the pods turn down, the tasks that were running on the pods get marked as FAILED. The sequence of actions is as follows:

1. Resource consumption decreases
2. Kubernetes realizes that pods that were scaled up now need to be turned down
3. Kubernetes sends SIGKILL signal to the pods that should turn down
4. Celery intercepts the signals and turns down all the Forked Processes
5. The tasks that were running on the processes return their execution back to the Main Process
6. The main process marks all the running tasks as FAILED

When the main process is the one that marks the running operations as FAILED, the retry code is never executed.

We do want to recover these tasks as they are part of a long running process. Celery should provide a way to recover these tasks.