Repository navigation
Completed Kubernetes Pods not cleared up #57553
Description
Apache Airflow Provider(s)
cncf-kubernetes
Versions of Apache Airflow Providers
Using Python 3.12.11 environment at:
apache-airflow-providers-amazon==9.2.0
apache-airflow-providers-cncf-kubernetes==10.1.0
apache-airflow-providers-common-compat==1.3.0
apache-airflow-providers-common-io==1.5.0
apache-airflow-providers-common-sql==1.21.0
apache-airflow-providers-databricks==7.0.0
apache-airflow-providers-elasticsearch==6.0.0
apache-airflow-providers-fab==1.5.2
apache-airflow-providers-ftp==3.12.0
apache-airflow-providers-google==12.0.0
apache-airflow-providers-http==5.0.0
apache-airflow-providers-imap==3.8.0
apache-airflow-providers-openlineage==2.0.0
apache-airflow-providers-postgres==6.0.0
apache-airflow-providers-sftp==5.0.0
apache-airflow-providers-smtp==1.9.0
apache-airflow-providers-sqlite==4.0.0
apache-airflow-providers-ssh==4.0.0Apache Airflow version
2.10.5 (also present in main)
Operating System
Rocky Linux
Deployment
Other 3rd-party Helm chart
Deployment details
Airflow Scheduler deployment with a single replica in an Airflow namespace. Executor pods are created in the same namespace.
What happened
When a task starts, an executor pod is created. During execution, the scheduler pod restarts. The task finishes before the new scheduler pod becomes ready. As a result, the executor pod completes but isn’t cleaned up because the scheduler was restarting. The completed pod remains orphaned — it isn’t adopted by the new scheduler instance and stays in the namespace indefinitely.
What you think should happen instead
The orphaned and completed tasks should be adopted by the new scheduler and deleted.
How to reproduce
- Create a deployment for an Airflow scheduler
- Run a
KubernetesPodOperatortask ensuring the config option ofdelete_worker_podsis set toTrue(which is the default). This task needs to be long enough for the next step - Scale the scheduler deployment to zero before the task completes
- Once the task completes scale the deployment back to one
During this time the task pod should be left indefinitely in a completed state and not adopted by the new scheduler pod.
It may take a try or two to get the timing right so that the task completes during the restart.
Anything else
We have several deployments throughout the day with hundreds of tasks scheduling. Due to this we are seeing tens of pods being orphaned daily.
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
Rustix69 commentedon Oct 31, 2025
@ryandutton Can I pick up this issue ??
ryandutton commentedon Oct 31, 2025
I'm happy to leave it with you, let me know if I can provide anymore information.
ryandutton commentedon Nov 3, 2025
This issue started occurring when I upgraded from 2.9.3 to 2.10.5. I've reviewed the different between the two versions and there was a slight change to the
_list_podsmethod. The previous version included a k8s call wrapped in a try catch. The comment above included a link to this issue kubernetes-client/python#2155. This is an observation rather than identifying the problem.EDIT: I've ran the
_list_podscode locally using the matching versions and it appears to identify the complete pod correctly so I don't think it's relating to issue/pr I linked above.ryandutton commentedon Nov 4, 2025
Issue:
A section of code appears to be unreachable for completed pods. The
adopt_or_reset_orphaned_tasksmethod, which runs periodically in the scheduler, is responsible for adopting both launched and completed pods.A change introduced in this commit prevents
_adopt_completed_podsfrom being called when completed pods exist but no adoptable (launched) pods are present. The results of this logic are stored intis_to_adopt_or_reset.Proposed fix:
Decouple the call to
_adopt_completed_podsfrom the query that determines whether launched tasks are available for adoption.I was able to resolve this locally by invoking self._adopt_completed_pods within the Kubernetes executor’s start method.
ryandutton commentedon Nov 5, 2025
This issue is also present in
main