Skip to content

Fix Celery worker --max-tasks-per-child for Celery 4.x.#3625

Merged
arikfr merged 4 commits into
masterfrom
celery-max-memory
Mar 27, 2019
Merged

Fix Celery worker --max-tasks-per-child for Celery 4.x.#3625
arikfr merged 4 commits into
masterfrom
celery-max-memory

Conversation

@jezdez

@jezdez jezdez commented Mar 22, 2019

Copy link
Copy Markdown
Contributor

What type of PR is this? (check all applicable)

  • Feature
  • Bug Fix

Description

This fixes the name of the --max-tasks-per-child parameter whose name was changed in Celery 4.x and adds a --max-memory-per-child of 1/4 of total memory.

Related Tickets & Documents

Port of mozilla#364

@jezdez jezdez added this to the v7.0.0 milestone Mar 22, 2019
@jezdez
jezdez requested a review from arikfr March 22, 2019 13:16
@ghost ghost assigned jezdez Mar 22, 2019
@ghost ghost added the in progress label Mar 22, 2019
@jezdez

jezdez commented Mar 22, 2019

Copy link
Copy Markdown
Contributor Author

/cc @jasonthomas

Comment thread bin/docker-entrypoint Outdated
Comment thread bin/docker-entrypoint Outdated
Comment thread bin/docker-entrypoint Outdated
worker() {
WORKERS_COUNT=${WORKERS_COUNT:-2}
QUEUES=${QUEUES:-queries,scheduled_queries,celery,schemas}
WORKER_EXTRA_OPTIONS=${WORKER_EXTRA_OPTIONS:}

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jasonthomas Is this a legitimate shell script?

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it is throwing a bad substitution error. It should be WORKER_EXTRA_OPTIONS=${WORKER_EXTRA_OPTIONS:-} instead.

@jezdez
jezdez requested review from arikfr and removed request for arikfr March 27, 2019 14:29
@jezdez jezdez modified the milestones: v7.0.0, Next Mar 27, 2019

@arikfr arikfr left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good assuming the WORKER_EXTRA_OPTIONS thing is legit.

@arikfr
arikfr merged commit 1933dee into master Mar 27, 2019
@arikfr
arikfr deleted the celery-max-memory branch March 27, 2019 19:08
@arikfr

arikfr commented Mar 27, 2019

Copy link
Copy Markdown
Member

Thanks.

harveyrendell pushed a commit to pushpay/redash that referenced this pull request Nov 14, 2019
* Fix Celery worker CLI parameter name that was changed in Celery 4.x.

* Set Celery worker --max-memory-per-child to 1/4th of total system memory.

* Review fixes.

* Review fixes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants