Repository navigation
KubernetesPodOperator Mark Task Failed after SUCCESS Randomly #21420
Description
Apache Airflow version
2.2.3 (latest released)
What happened
We are using KubernetesPodOperator to schedule basically all of our tasks.
Sometimes, randomly some TASK finish and are marked as SUCCESS and then immediately marked FAILED.
....
2022-02-08, 06:09:51 CET - Bootstrap - INFO - *********************************************************
2022-02-08, 06:09:51 CET - Bootstrap - INFO - *** FIN ***
2022-02-08, 06:09:51 CET - Bootstrap - INFO - *********************************************************
Event: consore-biologieclinique-interface-0.4.f3afe897abf646c6b22642cb594b8745 had an event of type Succeeded
Event with job id consore-biologieclinique-interface-0.4.f3afe897abf646c6b22642cb594b8745 Succeeded
Event: consore-biologieclinique-interface-0.4.f3afe897abf646c6b22642cb594b8745 had an event of type Succeeded
Event with job id consore-biologieclinique-interface-0.4.f3afe897abf646c6b22642cb594b8745 Succeeded
Marking task as SUCCESS. dag_id=consore-by-window, task_id=consore-biologieclinique-interface-0.4, execution_date=20220207T215900, start_date=20220208T040444, end_date=20220208T040953
Task exited with return code 0
Marking task as FAILED. dag_id=consore-by-window, task_id=consore-biologieclinique-interface-0.4, execution_date=20220207T215900, start_date=20220208T040444, end_date=20220208T040953
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'execution_date' from the template is deprecated and will be removed in a future version. Please use 'data_interval_start' or 'logical_date' instead.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'next_ds' from the template is deprecated and will be removed in a future version. Please use '{{ data_interval_end | ds }}' instead.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'next_ds_nodash' from the template is deprecated and will be removed in a future version. Please use '{{ data_interval_end | ds_nodash }}' instead.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'next_execution_date' from the template is deprecated and will be removed in a future version. Please use 'data_interval_end' instead.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'prev_ds' from the template is deprecated and will be removed in a future version.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'prev_ds_nodash' from the template is deprecated and will be removed in a future version.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'prev_execution_date' from the template is deprecated and will be removed in a future version.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'prev_execution_date_success' from the template is deprecated and will be removed in a future version. Please use 'prev_data_interval_start_success' instead.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'tomorrow_ds' from the template is deprecated and will be removed in a future version.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'tomorrow_ds_nodash' from the template is deprecated and will be removed in a future version.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'yesterday_ds' from the template is deprecated and will be removed in a future version.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/context.py:152 AirflowContextDeprecationWarning: Accessing 'yesterday_ds_nodash' from the template is deprecated and will be removed in a future version.
/opt/bitnami/airflow/venv/lib/python3.8/site-packages/airflow/utils/email.py:102 PendingDeprecationWarning: Fetching SMTP credentials from configuration variables will be deprecated in a future release. Please set credentials using a connection instead.
section/key [smtp/smtp_user] not found in config
Email alerting: attempt 1
...
What you expected to happen
What i expect is that the TASK is Marked as SUCCESS only .
...
2022-02-08, 06:09:51 CET - Bootstrap - INFO - *********************************************************
2022-02-08, 06:09:51 CET - Bootstrap - INFO - *** FIN ***
2022-02-08, 06:09:51 CET - Bootstrap - INFO - *********************************************************
Event: consore-biologieclinique-interface-0.4.f3afe897abf646c6b22642cb594b8745 had an event of type Succeeded
Event with job id consore-biologieclinique-interface-0.4.f3afe897abf646c6b22642cb594b8745 Succeeded
Event: consore-biologieclinique-interface-0.4.f3afe897abf646c6b22642cb594b8745 had an event of type Succeeded
Event with job id consore-biologieclinique-interface-0.4.f3afe897abf646c6b22642cb594b8745 Succeeded
Marking task as SUCCESS. dag_id=consore-by-window, task_id=consore-biologieclinique-interface-0.4, execution_date=20220207T215900, start_date=20220208T040444, end_date=20220208T040953
Task exited with return code 0
...
How to reproduce
I don't know how to reproduce.
Ideas :
- Deploy airflow 2.2.3
- Create DAG with KubernetesPodOperator
- Schedule a task
Operating System
debian-buster
Versions of Apache Airflow Providers
can't have that
Deployment
Official Apache Airflow Helm Chart
Deployment details
using the bitnami helm https://artifacthub.io/packages/helm/bitnami/airflow
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
raphaelauv commentedon Feb 8, 2022
what version of https://pypi.org/project/apache-airflow-providers-cncf-kubernetes/#history ?
armandleopold commentedon Feb 14, 2022
@raphaelauv I am not really familiar with how bitnami builds its images :
https://github.com/bitnami/bitnami-docker-airflow/blob/master/2/debian-10/prebuildfs/opt/bitnami/.bitnami_components.json#L7
raphaelauv commentedon Feb 14, 2022
you can go inside the container and do
pip listarmandleopold commentedon Feb 14, 2022
armandleopold commentedon Feb 14, 2022
This is strange :
The task is "Failed" but don't appears as failed in the front-end DAG view
Ho no ok, it's because the last DAG run finished Successfully.