Checklist
software -> celery:4.1.0 (latentcall) kombu:4.1.0 py:3.5.2
billiard:3.5.0.3 py-amqp:2.2.2
platform -> system:Linux arch:64bit imp:CPython
loader -> celery.loaders.app.AppLoader
settings -> transport:amqp results:redis://redis-prod/0
result_backend: 'redis://redis-prod/0'
task_track_started: True
task_acks_late: True
task_routes: {
'*': {'queue': 'queue1'}}
worker_send_task_events: True
result_serializer: 'json'
broker_connection_max_retries: 0
broker_heartbeat: 20
worker_pool_restarts: True
task_serializer: 'json'
worker_redirect_stdouts_level: 'INFO'
accept_content: ['json', 'pickle']
task_queue_max_priority: 10
broker_url: 'amqp://guest:********@rabbitmq-prod:5672//'
No, this issue is happening in release 4.1.0
Steps to reproduce
Not sure, it happens randomly.
Expected behavior
Workers should stay online in flower unless there is a problem. If there really was a persistent problem I would expect commands like
celery -A tasks control heartbeat
celery -A tasks status
to show the workers as offline as well.
Actual behavior
After several hours running, some workers appear as offline in flower and celery -A proj events --dump records no heartbeats coming from these workers. At the same time
celery -A tasks control heartbeat
celery -A tasks status
show all workers as online.
When I restart RabbitMQ with
rabbitmqctl stop_app && rabbitmqctl start_app
the workers return to the online state and at the same time, the other worker nodes log messages complaining about a "Substantial drift" of several hours in relation to the clocks of the nodes that were offline. So it seems that somewhere the missing heartbeats were bottled up.
Another way to make them online again is to delete the celeryev.* queues in RabbitMQ.
This problem affect both workers with the fork pool and the gevent pool.
Discussion
Ok, I know the described behavior above is pretty vague. Basically there are a zillion thing that could go wrong with the connection of the worker nodes to RabbitMQ. These worker nodes are deployed in Kubernetes pods in VMWare nodes, so there is a SNAT and DNAT going on. The VMs could be missing clock interrupts, or there could be problems with the conntrack tables, or the memory of the nodes could be exhausted. Put simply, it's expected that from time to time a worker node will loose connection to the broker. However, with worker heartbeats and socket KEEPALIVE (https://github.com/celery/py-amqp/blob/2304654c00e7fde13f32226a097a73eab7f7275f/amqp/transport.py#L165) it should be possible to detect connection problems and start recovery procedures.
I don't know exactly where in the celery/kombu/pq-amqp stack to look to debug this. I would appreciate any advice on that. In the meantime I'm going to activate CELERY_RDBSIG=1 and wait for workers to go offline again and try to get a stacktrace.
Maybe setting py-amqp socket read_timeout or write_timeout could be helpful in detecting stale TCP connections. Any thoughts on that? Another question that I have is how can I know which celeryev*. queue is owned by which worker? The connection descriptors in RabbitMQ are not helpful because of the SNAT.
Checklist
celery -A proj reportin the issue.masterbranch of Celery.No, this issue is happening in release 4.1.0
Steps to reproduce
Not sure, it happens randomly.
Expected behavior
Workers should stay online in flower unless there is a problem. If there really was a persistent problem I would expect commands like
to show the workers as offline as well.
Actual behavior
After several hours running, some workers appear as offline in flower and
celery -A proj events --dumprecords no heartbeats coming from these workers. At the same timeshow all workers as online.
When I restart RabbitMQ with
the workers return to the online state and at the same time, the other worker nodes log messages complaining about a "Substantial drift" of several hours in relation to the clocks of the nodes that were offline. So it seems that somewhere the missing heartbeats were bottled up.
Another way to make them online again is to delete the celeryev.* queues in RabbitMQ.
This problem affect both workers with the fork pool and the gevent pool.
Discussion
Ok, I know the described behavior above is pretty vague. Basically there are a zillion thing that could go wrong with the connection of the worker nodes to RabbitMQ. These worker nodes are deployed in Kubernetes pods in VMWare nodes, so there is a SNAT and DNAT going on. The VMs could be missing clock interrupts, or there could be problems with the conntrack tables, or the memory of the nodes could be exhausted. Put simply, it's expected that from time to time a worker node will loose connection to the broker. However, with worker heartbeats and socket KEEPALIVE (https://github.com/celery/py-amqp/blob/2304654c00e7fde13f32226a097a73eab7f7275f/amqp/transport.py#L165) it should be possible to detect connection problems and start recovery procedures.
I don't know exactly where in the celery/kombu/pq-amqp stack to look to debug this. I would appreciate any advice on that. In the meantime I'm going to activate CELERY_RDBSIG=1 and wait for workers to go offline again and try to get a stacktrace.
Maybe setting py-amqp socket read_timeout or write_timeout could be helpful in detecting stale TCP connections. Any thoughts on that? Another question that I have is how can I know which celeryev*. queue is owned by which worker? The connection descriptors in RabbitMQ are not helpful because of the SNAT.