Skip to content

KPO deferable "random" false fail #31335

@raphaelauv

Description

@raphaelauv

Apache Airflow version

2.6.1

What happened

With the KPO and only in deferrable I have "random" false fail

the dag

from pendulum import today
from airflow import DAG
from airflow.providers.cncf.kubernetes.operators.pod import KubernetesPodOperator

dag = DAG(
    dag_id="kubernetes_dag",
    schedule_interval="0 0 * * *",
    start_date=today("UTC").add(days=-1)
)

with dag:
    cmd = "echo toto && sleep 22 && echo finish"

    KubernetesPodOperator.partial(
        task_id="task-one",
        namespace="default",
        kubernetes_conn_id="kubernetes_default",
        config_file="/opt/airflow/include/.kube/config",  # bug of deferrable corrected in 6.2.0
        name="airflow-test-pod",
        image="alpine:3.16.2",
        cmds=["sh", "-c", cmd],
        is_delete_operator_pod=True,
        deferrable=True,
        get_logs=True,
    ).expand(env_vars=[{"a": "a"} for _ in range(8)])

Screenshot from 2023-05-17 02-01-03

the log of the task in error :

dag_id=kubernetes_dag_run_id=scheduled__2023-05-16T00_00_00+00_00_task_id=task-one_map_index=2_attempt=1.log

What you think should happen instead

KPO should not fail ( in deferable ) if the container succesfully run in K8S

How to reproduce

If I remove the sleep 22 from the cmd then I do not see any more random task fails

Operating System

ubuntu 22.04

Versions of Apache Airflow Providers

apache-airflow-providers-cncf-kubernetes==6.1.0

Deployment

Docker-Compose

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions