KubernetesPodOperator xcom sidecar stuck in running #22318
Description
Apache Airflow version
2.2.4 (latest released)
What happened
When the main container errors and failed to write a return.json file, the xcom sidecar hangs and doesn't exit properly with an empty return.json.
This is a problem because we want to suppress the following error, as the reason the pod failed should not be that xcom failed.
[2022-03-16, 17:08:07 UTC] {pod_manager.py:342} INFO - Running command... cat /airflow/xcom/return.json
[2022-03-16, 17:08:07 UTC] {pod_manager.py:349} INFO - stderr from command: cat: can't open '/airflow/xcom/return.json': No such file or directory
[2022-03-16, 17:08:07 UTC] {pod_manager.py:342} INFO - Running command... kill -s SIGINT 1
[2022-03-16, 17:08:08 UTC] {kubernetes_pod.py:417} INFO - Deleting pod: test.20882a4c607d418d94e87231214d34c0
[2022-03-16, 17:08:08 UTC] {taskinstance.py:1718} ERROR - Task failed with exception
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 385, in execute
result = self.extract_xcom(pod=self.pod)
File "/usr/local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/operators/kubernetes_pod.py", line 360, in extract_xcom
result = self.pod_manager.extract_xcom(pod)
File "/usr/local/lib/python3.8/site-packages/airflow/providers/cncf/kubernetes/utils/pod_manager.py", line 337, in extract_xcom
raise AirflowException(f'Failed to extract xcom from pod: {pod.metadata.name}')
airflow.exceptions.AirflowException: Failed to extract xcom from pod: test.20882a4c607d418d94e87231214d34c0
and have the KubernetesPodOperator exit gracefully
What you think should happen instead
sidecar should exit with an empty xcom return value
How to reproduce
KubernetesPodOperator with command mkdir -p /airflow/xcom;touch /airflow/xcom/return.json; cat a >> /airflow/xcom/return.json
Operating System
Versions of Apache Airflow Providers
No response
Deployment
Other Docker-based deployment
Deployment details
No response
Anything else
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct