Description

Proposal:

Allow jobs deployed by the helm chart to be created without assuming that they will use the same environments as the pods they are deployed alongside.

Ways to accomplish this:

• Add new boolean values for the jobs, allowing the user to opt-out of using the environment provided by the custom_airflow_environment helper, and instead provide their own environment definition catered to these jobs.

• Allow the user to specify the environment on a per-resource level, failing back to the deployment-wide custom_airflow_environment if per-resource environments aren't provided.

Use case/motivation

The custom_airflow_environment helper is used in the creation of all Airflow pods and jobs (e.g. run-airflow-migrations and create-user-job jobs). There is no option to specify environments on a per-resource level, forcing all resources to share the same environment variables.

For pods, this is reasonable behavior as they should share environments. For jobs (which are immutable once the resource is created), this creates a difficult situation where the environment cannot be altered once the helm chart is deployed if using tools such as flux or argo. In practice this means that environments across the entire deployment cannot be changed without doing a full uninstall and reinstall of the helm chart.

These jobs accomplish very small, focused tasks when compared to the pods that they share their environments with - and I believe that they should not be forced to share the same environment variables, as it's very unlikely that they require the same environment definitions.

References to this helper in the job templates:
https://github.com/apache/airflow/blob/main/chart/templates/jobs/migrate-database-job.yaml#L100
https://github.com/apache/airflow/blob/main/chart/templates/jobs/create-user-job.yaml#L98

Note that there is no way to avoid the custom_airflow_environment helper from being applied to these templates.

Related issues

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct