Issue Summary
Celery 3.x does not allow limiting worker memory usage. This means one worker can use all of host's memory and affect function for other works and processes. Upgrading Celery to 4.x will allow adding --max-memory-per-child flag and limit the memory usage.
I understand that Mozilla is using Celery 4 with Redash, we'd like to hear about your experience and what it involved.
There is a discussion on Discourse at https://discuss.redash.io/t/upgrade-celery-to-4-x/1867
Please add your thoughts to that discussion.
@arikfr can you tag the Mozilla folks?
Issue Summary
Celery 3.x does not allow limiting worker memory usage. This means one worker can use all of host's memory and affect function for other works and processes. Upgrading Celery to 4.x will allow adding
--max-memory-per-childflag and limit the memory usage.I understand that Mozilla is using Celery 4 with Redash, we'd like to hear about your experience and what it involved.
There is a discussion on Discourse at https://discuss.redash.io/t/upgrade-celery-to-4-x/1867
Please add your thoughts to that discussion.
@arikfr can you tag the Mozilla folks?