[2025-03-24 08:12:59,761] [INFO] [celery.worker.consumer.connection] Connected to redis://:**@redis-prod-redis-ha-haproxy.database.svc.cluster.local:6379//
[2025-03-24 08:13:29,888] [WARNING] [celery.worker.consumer.consumer] consumer: Connection to broker lost. Trying to re-establish the connection...
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/celery/worker/consumer/consumer.py", line 340, in start
    blueprint.start(self)
  File "/usr/local/lib/python3.12/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/usr/local/lib/python3.12/site-packages/celery/worker/consumer/consumer.py", line 746, in start
    c.loop(*c.loop_args())
  File "/usr/local/lib/python3.12/site-packages/celery/worker/loops.py", line 97, in asynloop
    next(loop)
  File "/usr/local/lib/python3.12/site-packages/kombu/asynchronous/hub.py", line 373, in create_loop
    cb(*cbargs)
  File "/usr/local/lib/python3.12/site-packages/kombu/transport/redis.py", line 1352, in on_readable
    self.cycle.on_readable(fileno)
  File "/usr/local/lib/python3.12/site-packages/kombu/transport/redis.py", line 569, in on_readable
    chan.handlers[type]()
  File "/usr/local/lib/python3.12/site-packages/kombu/transport/redis.py", line 918, in _receive
    ret.append(self._receive_one(c))
               ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/kombu/transport/redis.py", line 928, in _receive_one
    response = c.parse_response()
               ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/redis/client.py", line 865, in parse_response
    response = self._execute(conn, try_read)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/redis/client.py", line 841, in _execute
    return conn.retry.call_with_retry(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/redis/retry.py", line 65, in call_with_retry
    fail(error)
  File "/usr/local/lib/python3.12/site-packages/redis/client.py", line 843, in <lambda>
    lambda error: self._disconnect_raise_connect(conn, error),
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/redis/client.py", line 830, in _disconnect_raise_connect
    raise error
  File "/usr/local/lib/python3.12/site-packages/redis/retry.py", line 62, in call_with_retry
    return do()
           ^^^^
  File "/usr/local/lib/python3.12/site-packages/redis/client.py", line 842, in <lambda>
    lambda: command(*args, **kwargs),
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/redis/client.py", line 863, in try_read
    return conn.read_response(disconnect_on_error=False, push_request=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/redis/connection.py", line 592, in read_response
    response = self._parser.read_response(disable_decoding=disable_decoding)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/redis/_parsers/hiredis.py", line 128, in read_response
    self.read_from_socket()
  File "/usr/local/lib/python3.12/site-packages/redis/_parsers/hiredis.py", line 90, in read_from_socket
    raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.
[2025-03-24 08:13:32,175] [WARNING] [py.warnings] /usr/local/lib/python3.12/site-packages/celery/worker/consumer/consumer.py:391: CPendingDeprecationWarning:
In Celery 5.1 we introduced an optional breaking change which
on connection loss cancels all currently executed tasks with late acknowledgement enabled.
These tasks cannot be acknowledged as the connection is gone, and the tasks are automatically redelivered
back to the queue. You can enable this behavior using the worker_cancel_long_running_tasks_on_connection_loss
setting. In Celery 5.1 it is set to False by default. The setting will be set to True by default in Celery 6.0.

  warnings.warn(CANCEL_TASKS_BY_DEFAULT, CPendingDeprecationWarning)

[2025-03-24 08:13:32,175] [INFO] [celery.worker.consumer.consumer] Temporarily reducing the prefetch count to 4 to avoid over-fetching since 8 tasks are currently being processed.
The prefetch count will be gradually restored to 16 as the tasks complete processing.