Skip to content

Give up sending a worker-offline message if transport is not connected#6039

Merged
auvipy merged 2 commits into
celery:masterfrom
Alphadelta14:improvement/giveup-worker-offline
Apr 15, 2020
Merged

Give up sending a worker-offline message if transport is not connected#6039
auvipy merged 2 commits into
celery:masterfrom
Alphadelta14:improvement/giveup-worker-offline

Conversation

@Alphadelta14

@Alphadelta14 Alphadelta14 commented Apr 15, 2020

Copy link
Copy Markdown
Contributor

Description

If a celery worker is segmented from a cluster and is told to die, it will try very aggressively to connect to the transport that may or not exist at this point. This can add a bit of time to shutdown procedures and usually ends up just getting sigkill'd by the init/supervisor regardless.
This change makes it so that the worker-offline is still sent, but if it is lost, just accept it and die peacefully by setting retry=False on the worker-offline message.
In cases where a cluster is still fully operational from the segmented worker, it will track the worker as lost by heartbeats instead of the offline message.

@auvipy auvipy added this to the 4.4.x milestone Apr 15, 2020
@auvipy
auvipy merged commit 6be4523 into celery:master Apr 15, 2020
auvipy added a commit that referenced this pull request Jul 8, 2020
@thedrow @verhovsky
@auvipy @jainal09 @gsfish @dipAch @spengjie @chris-griffin @mhewedy @phated @tamsanh @eriktews @anilaratna2 @wyguy444 @galCohen88 @alfred-sa @paramkpr @sveniu @safwanrahman Aissaoui Anouar @qdzzyb2014 @Alireza2n @marcosmoyano @shenek @AndrewPix @fladi @dejlek @yannickschuchmann @matteius @tirkarthi @bernd-wechner @soldag @uddmorningsun @marfgold1 @Wooden-Robot @imomaliev @AlexRiina @xncbf @nixphix @martey @Seleznev-nvkz @Prodge @adyouri @Ixiodor @abhishekakamai @allanlei @M1ha-Shvn @hicay @michamos @chrisburr @tildedave @the-owl Michael Gaddis @epwalsh @TalRoni @lpsinger @neonbunny Abhishek @theirix @yukihira1992 @jpays-cnp @gward Alexa Griffith @heedong-jung @heedongsamsung @shreyanshk @georgedorn @Alphadelta14 @azimjohn @ysde @amirzia @nadflinn @nezhar @specialorange @aeijdenberg rafaelreuber @coderanger @clems4ever Mathieu Chataigner @eugeneyalansky @leonardsocialcodeinc @xinydev @watercrossing @hanchau shaoziwei @mchataigner @Anakael @ckdanny @Tirzono @dtilloy-criteo @anthonynsimon @lironhl @Darkheir @Kirade @singlaive Murphy Meng @whtsky @kwist-sgr @ldragan @nijel @radimsuckr @art-vasilyev @kakakikikeke-fork @baixuexue123 baixue @prashnts @AbdealiLoKo
Remove PY3 conditionals (#5954