Skip to content

Celery warm shutdown with celery>=5.5.0b3 and eventlet>=0.37.0 #10083

Description

@AndreyObviously

Checklist

  • I have verified that the issue exists against the main branch of Celery.
  • This has already been asked to the discussions forum first.
  • I have read the relevant section in the
    contribution guide
    on reporting bugs.
  • I have checked the issues list
    for similar or identical bug reports.
  • I have checked the pull requests list
    for existing proposed fixes.
  • I have checked the commit log
    to find out if the bug was already fixed in the main branch.
  • I have included all related issues and possible duplicate issues
    in this issue (If there are none, check this box anyway).
  • I have tried to reproduce the issue with pytest-celery and added the reproduction script below.

Mandatory Debugging Information

  • I have included the output of celery -A proj report in the issue.
    (if you are not able to do this, then at least specify the Celery
    version affected).
  • I have verified that the issue exists against the main branch of Celery.
  • I have included the contents of pip freeze in the issue.
  • I have included all the versions of all the external dependencies required
    to reproduce this bug.

Optional Debugging Information

  • I have tried reproducing the issue on more than one Python version
    and/or implementation.
  • I have tried reproducing the issue on more than one message broker and/or
    result backend.
  • I have tried reproducing the issue on more than one version of the message
    broker and/or result backend.
  • I have tried reproducing the issue on more than one operating system.
  • I have tried reproducing the issue on more than one workers pool.
  • I have tried reproducing the issue with autoscaling, retries,
    ETA/Countdown & rate limits disabled.
  • I have tried reproducing the issue after downgrading
    and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

  • None

Possible Duplicates

  • None

Environment & Settings

Celery version:

celery report Output:

software -> celery:5.6.2 (recovery) kombu:5.6.2 py:3.9.25
            billiard:4.2.4 py-amqp:5.3.1
platform -> system:Linux arch:64bit, ELF
            kernel version:6.10.11-linuxkit imp:CPython
loader   -> celery.loaders.app.AppLoader
settings -> transport:amqp results:django-db

Steps to Reproduce

Required Dependencies

  • Python Version: 3.9.25
  • Minimal Celery Version: 5.5.0b3
  • Minimal Kombu Version: N/A or Unknown
  • Minimal Broker Version: N/A or Unknown
  • Minimal Result Backend Version: N/A or Unknown
  • Minimal OS and/or Kernel Version: N/A or Unknown
  • Minimal Broker Client Version: N/A or Unknown
  • Minimal Result Backend Client Version: N/A or Unknown
  • Minimal Eventlet Version: 0.37.0

Python Packages

pip freeze Output:

Other Dependencies

Details

eventlet>=0.37.0

Minimally Reproducible Test Case

In #9222 print was changed to os.write.
Eventlet in release 0.37.0 changed logic of os.read/write https://github.com/eventlet/eventlet/pull/975/changes.
This combination leads to exception during "warm shutdown"

Details Run

 celery -A myproj worker --pool=eventlet

worker will be started

 -------------- celery@9be1ca044182 v5.6.2 (recovery)
--- ***** ----- 
-- ******* ---- Linux-6.10.11-linuxkit-aarch64-with-glibc2.36 2026-02-02 17:45:29
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         myproj:0xffff997d2100
- ** ---------- .> transport:   amqp://guest:**@rabbitmq:5672//
- ** ---------- .> results:     
- *** --- * --- .> concurrency: 10 (eventlet)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery

then hit Ctrl+C. Exception will be raised(see Actual Behavior)

Expected Behavior

Details No exception. Same as for prefork(for example)

celery -A myproj worker --pool=eventlet

worker will be started

 -------------- celery@9be1ca044182 v5.6.2 (recovery)
--- ***** ----- 
-- ******* ---- Linux-6.10.11-linuxkit-aarch64-with-glibc2.36 2026-02-02 17:45:29
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         myproj:0xffff997d2100
- ** ---------- .> transport:   amqp://guest:**@rabbitmq:5672//
- ** ---------- .> results:     
- *** --- * --- .> concurrency: 10 (eventlet)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery

then hit Ctrl+C. No exception after this. Result should look like

^C
worker: Hitting Ctrl+C again will initiate cold shutdown, terminating all running tasks!

worker: Warm shutdown (MainProcess)

Actual Behavior

Details

^C[2026-02-02 17:54:32,127: CRITICAL/MainProcess] Unrecoverable error: RuntimeError('do not call blocking functions from the mainloop')
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/celery/worker/worker.py", line 203, in start
    self.blueprint.start(self)
  File "/usr/local/lib/python3.9/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/usr/local/lib/python3.9/site-packages/celery/bootsteps.py", line 365, in start
    return self.obj.start()
  File "/usr/local/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 346, in start
    blueprint.start(self)
  File "/usr/local/lib/python3.9/site-packages/celery/bootsteps.py", line 116, in start
    step.start(parent)
  File "/usr/local/lib/python3.9/site-packages/celery/worker/consumer/consumer.py", line 788, in start
    c.loop(*c.loop_args())
  File "/usr/local/lib/python3.9/site-packages/celery/worker/loops.py", line 136, in synloop
    _loop_cycle()
  File "/usr/local/lib/python3.9/site-packages/celery/worker/loops.py", line 125, in _loop_cycle
    connection.drain_events(timeout=2.0)
  File "/usr/local/lib/python3.9/site-packages/kombu/connection.py", line 343, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/kombu/transport/pyamqp.py", line 171, in drain_events
    return connection.drain_events(**kwargs)
  File "/usr/local/lib/python3.9/site-packages/amqp/connection.py", line 526, in drain_events
    while not self.blocking_read(timeout):
  File "/usr/local/lib/python3.9/site-packages/amqp/connection.py", line 531, in blocking_read
    frame = self.transport.read_frame()
  File "/usr/local/lib/python3.9/site-packages/amqp/transport.py", line 297, in read_frame
    frame_header = read(7, True)
  File "/usr/local/lib/python3.9/site-packages/amqp/transport.py", line 632, in _read
    s = recv(n - len(rbuf))
  File "/usr/local/lib/python3.9/site-packages/eventlet/greenio/base.py", line 352, in recv
    return self._recv_loop(self.fd.recv, b'', bufsize, flags)
  File "/usr/local/lib/python3.9/site-packages/eventlet/greenio/base.py", line 346, in _recv_loop
    self._read_trampoline()
  File "/usr/local/lib/python3.9/site-packages/eventlet/greenio/base.py", line 314, in _read_trampoline
    self._trampoline(
  File "/usr/local/lib/python3.9/site-packages/eventlet/greenio/base.py", line 206, in _trampoline
    return trampoline(fd, read=read, write=write, timeout=timeout,
  File "/usr/local/lib/python3.9/site-packages/eventlet/hubs/__init__.py", line 157, in trampoline
    return hub.switch()
  File "/usr/local/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 310, in switch
    return self.greenlet.switch()
  File "/usr/local/lib/python3.9/site-packages/eventlet/hubs/hub.py", line 362, in run
    self.wait(sleep_time)
  File "/usr/local/lib/python3.9/site-packages/eventlet/hubs/poll.py", line 80, in wait
    presult = self.do_poll(seconds)
  File "/usr/local/lib/python3.9/site-packages/eventlet/hubs/epolls.py", line 31, in do_poll
    return self.poll.poll(seconds)
  File "/usr/local/lib/python3.9/site-packages/celery/apps/worker.py", line 300, in _handle_request
    callback(worker)
  File "/usr/local/lib/python3.9/site-packages/celery/apps/worker.py", line 448, in on_SIGINT
    safe_say('worker: Hitting Ctrl+C again will initiate cold shutdown, terminating all running tasks!',
  File "/usr/local/lib/python3.9/site-packages/celery/apps/worker.py", line 82, in safe_say
    os.write(f.fileno(), f'\n{msg}\n'.encode())
  File "/usr/local/lib/python3.9/site-packages/eventlet/green/os.py", line 74, in write
    hubs.trampoline(fd, write=True)
  File "/usr/local/lib/python3.9/site-packages/eventlet/hubs/__init__.py", line 139, in trampoline
    raise RuntimeError('do not call blocking functions from the mainloop')
RuntimeError: do not call blocking functions from the mainloop

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions