Introduce workers.celery and workers.kubernetes config sections#51460
Closed
Miretpl wants to merge 19 commits intoapache:mainfrom
Closed
Introduce workers.celery and workers.kubernetes config sections#51460Miretpl wants to merge 19 commits intoapache:mainfrom
Miretpl wants to merge 19 commits intoapache:mainfrom
Conversation
Miretpl
changed the title
Contributor
Author
|
Currently, the introduction of |
Miretpl
force-pushed
the
separate-workers-config
branch
from
94a1b5a to
12ba007
Compare
Contributor
Author
|
Both sections |
This was referenced Jun 26, 2025
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
github-actions
bot
added
the
stale
Contributor
Author
|
Not stale - I'm during rebase 😄 |
github-actions
bot
removed
the
stale
Miretpl
force-pushed
the
separate-workers-config
branch
from
401b8d1 to
b7067b0
Compare
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
github-actions
bot
added
the
stale
2 tasks
98 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
closes: #28880
Currently, in the Helm Chart, there is a workers field whose values are shared between Celery and Kubernetes executors workers. The only possible way to configure both separately is by using the
podTemplatefield, which overrides the whole pod-template-file content. However, there is no easy possibility to override part of the configuration or to share the same configuration between workers.This PR introduces the
workers.celeryandworkers.kubernetesfiles, which contain specific fields used by each executor worker. Every field specified under the above sections overrides the value of the respective fields under theworkerssection. In case of configuring the field underworkerssection, without configuring different values underworkers.celeryorworkers.kubernetessections, it will result in applying the change to both worker types. I added comments about deprecation to every field which was used only by one executor type. Also, I implemented the separation of service accounts used by pod-template-file and Airflow Celery workers with the usage ofworkers.useWorkerDedicatedServiceAccountsflag. By default, both workers will use a single common service account, and the fieldsworkers.celery.serviceAccountandworkers.kubernetes.serviceAccountwill have no effect.^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named
{pr_number}.significant.rstor{issue_number}.significant.rst, in airflow-core/newsfragments.