Task execution failure with multiple executors

Official Helm Chart version

1.16.0 (latest released)

Apache Airflow version

2.10.5

Kubernetes Version

1.31

Helm Chart configuration

No response

Docker Image customizations

No response

What happened

I'm afraid that then new multiple executor configuration does not work at all.
In when the scheduler parses the dags tasks are configured with:

executor='KubernetesExecutor`

but then when a pod is launched an environment variable is injected:

AIRFLOW__CORE__EXECUTOR=LocalExecutor

which next results in a validation error:

│ Traceback (most recent call last):                                                                                                                                                                                                                                           │
│   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/dag.py", line 852, in validate_executor_field                                                                                                                                                       │
│     ExecutorLoader.lookup_executor_name_by_str(task.executor)                                                                                                                                                                                                                │
│   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/executors/executor_loader.py", line 215, in lookup_executor_name_by_str                                                                                                                                    │
│     raise UnknownExecutorException(f"Unknown executor being loaded: {executor_name_str}")                                                                                                                                                                                    │
│ airflow.exceptions.UnknownExecutorException: Unknown executor being loaded: KubernetesExecutor

What you think should happen instead

No response

How to reproduce

Configure the chart with CeleryExecutor,KubernetesExecutor and run any task

Anything else

No response

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

Looks like the issue is here:
https://github.com/apache/airflow/blob/main/chart/files/pod-template-file.kubernetes-helm-yaml#L96
I managed to work around it by injecting

k8s.V1Pod(
        spec=k8s.V1PodSpec(
            containers=[
                k8s.V1Container(
                    name="base",
                    env=[k8s.V1EnvVar(name="AIRFLOW__CORE__EXECUTOR", value="LocalExecutor,KubernetesExecutor")],
                )
            ],
        ),
    )

into pod_override but that does not look like an ideal solution

@o-nikolas

cc @o-nikolas

@jedcunningham

Yes, a fundamental thing about Airflow is that the configuration should be the same in all machines. The tasks are using executors that are not configured in the pod it seems. So this is an issue with the helm chart itself (CC @jedcunningham I know you're OOTO, but I don't know who the next in-line Helm PoC is).

I believe this issue stems from the fact that when a task is executed inside a Kubernetes Pod, the DAG attempts to load the executor via a parameter — and that’s the root cause of the problem. This isn’t an issue when using a single executor, but if you attempt to use multiple executors, each task must explicitly receive an executor parameter to specify which one to use.

For example, consider the following DAG file:

# dags/kubernetes_hello_world.py

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime

default_args = {
    'start_date': datetime(2023, 1, 1),
}

with DAG(
    dag_id='kubernetes_hello_world',
    default_args=default_args,
    schedule_interval=None,
    catchup=False,
    tags=['test'],
) as dag:
    hello_task = BashOperator(
        task_id='print_hello',
        executor='KubernetesExecutor',  # This is the root cause of the issue!
        bash_command='echo "Hello, World from KubernetesExecutor!"'
    )

In this case, the scheduler dispatches the task to be executed via the KubernetesExecutor, which pushes it into the task queue. When a Pod is spun up based on the pod_template, the executor is typically set using the environment variable AIRFLOW__CORE__EXECUTOR, which defaults to LocalExecutor.

If the task didn't have an executor parameter explicitly passed, then the executor would be determined based on this environment variable by airflow.executors.executor_loader. However, because the DAG explicitly passes executor='KubernetesExecutor', the task inside the Pod attempts to re-load and execute via KubernetesExecutor — causing it to spin up yet another Pod.

This results in the following error inside the container:

airflow.exceptions.UnknownExecutorException: Unknown executor being loaded: KubernetesExecutor

Therefore, I think tasks that are created and run via a pod_template should ignore any executor parameter defined in the DAG and instead rely solely on the AIRFLOW__CORE__EXECUTOR environment variable for executor selection. Until this is handled in code, it seems to me that using multiple executors in practice is not really feasible.

If the maintainers agree with this approach, I’d love to submit a PR to contribute a fix for this. Would that be okay?

@ms32035

Before complex changes, I just created a workaround PR to close this issue based on the idea of @ms32035

But I think this is quite fundamental change in the way how K8S executor works. It's always worked in the way (from what I remember) - that when POD was run by K8S executor, the LocalExecutor has been set unconditonally - because fundamentally we did not want to again trigger the executor when run - so the task in K8S executor is executed effectively by the LocalExecutor.

We see that it's still using a local executor inside the pod, and the check that's causing the root issue is purely a validation, but I might be wrong.

@ihnokim

@ihnokim The issue is really not that complex. We simply have code in dag validation logic (when the dag is loaded) to check that all executors requested by tasks within the dag are present and configured (since we don't want users to get stuck trying to run tasks on executors that don't exist). Airflow assumes configuration is the same across all hosts, many things will break if it's not consistent (this was the biggest feedback @jedcunningham gave us ages ago when we started executor work).
So in this case when the new pod starts, it loads the dag, the validation logic runs, the configuration is not correct, so we fail.

Your PR to create a consistent config (at least for core.executor) is not a workaround, it's a perfectly fine fix 👍

@potiuk

@potiuk @ms32035 @o-nikolas Thank you for your kind response!
As you explained, I was able to confirm through testing that after passing the validation inside the pod, it runs like it is using the local executor as expected.
I hope the PR I submitted helps reduce confusion for other Helm chart users 👍

@ihnokim

@ihnokim Can you please finish with the PR? Highly appreciated

@potiuk

Last I looked there were approvals on the PR but also a request for changes blocking is merge (CC @potiuk). Would love to get that one done!

Navigation Menu

Repository navigation

Task execution failure with multiple executors #48667

Official Helm Chart version

Apache Airflow version

Kubernetes Version

Helm Chart configuration

Docker Image customizations

What happened

What you think should happen instead

How to reproduce

Anything else

Are you willing to submit PR?

Code of Conduct

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Task execution failure with multiple executors #48667

Description

Official Helm Chart version

Apache Airflow version

Kubernetes Version

Helm Chart configuration

Docker Image customizations

What happened

What you think should happen instead

How to reproduce

Anything else

Are you willing to submit PR?

Code of Conduct

Activity

boring-cyborg commented on Apr 2, 2025

ms32035 commented on Apr 2, 2025

eladkal commented on Apr 4, 2025

o-nikolas commented on Apr 4, 2025

ihnokim commented on Apr 18, 2025

ihnokim commented on Apr 18, 2025

potiuk commented on Apr 18, 2025

ms32035 commented on Apr 18, 2025

o-nikolas commented on Apr 18, 2025

ihnokim commented on Apr 19, 2025

morooshka commented on Jun 6, 2025

o-nikolas commented on Jun 6, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions