feat: add zombie KPO pod cleanup to KubernetesExecutor by gingeekrishna · Pull Request #67030 · apache/airflow

gingeekrishna · 2026-05-16T09:01:12Z

Summary

When a KubernetesPodOperator (KPO) task reaches a terminal state, the scheduler signals its pod to terminate. However, sidecar containers that ignore SIGTERM (such as the xcom sidecar or custom log-aggregation sidecars) can keep the pod in Running state indefinitely, leaking resources and confusing monitoring dashboards.

This PR adds a periodic zombie-pod cleanup to KubernetesExecutor that:

Identifies KPO pods that are still Running in Kubernetes but whose TaskInstance has already reached a terminal state (zombie pods)
Also catches pods from stale retry attempts (pod try_number < active TI try_number)
Force-deletes them with grace_period_seconds=0 to bypass stuck sidecar containers

Two new [kubernetes_executor] config options gate the feature:

clean_zombie_kpo_pods (boolean, default True) — enable/disable the cleanup scan
zombie_kpo_pod_cleanup_interval (integer seconds, default 300) — how often to scan

EksPodOperator and GKEStartPodOperator are intentionally excluded: their pods run in external clusters where this kube_client has no authority.

Changes

providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py — add _cleanup_zombie_kpo_pods() method and sync() integration
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/kube_config.py — read the two new config options
providers/cncf/kubernetes/provider.yaml — register the two new config options

Test plan

Gen-AI disclosure

Yes — Claude Code (Anthropic)

Important

🛠️ Maintainer triage note for @gingeekrishna · by @potiuk · 2026-07-28 17:13 UTC

This draft PR has been inactive since the last triage note, with no response from the author. Closing to keep the queue clean:

You are welcome to reopen this PR when you resume work, or open a new one addressing the issues previously raised.
If you pick it back up, please rebase onto the current main branch first.

The ball is in your court — you've been assigned to this PR. Reopen or open a fresh PR once addressed — no rush.

See the Pull Request quality criteria for how to fix each item. There is no rush.

_{Automated triage — may be imperfect; a maintainer takes the next look. We use this two-stage triage process so maintainers' limited time goes to the conversation with you.}

boring-cyborg · 2026-05-16T09:01:20Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
Here are some useful points:

Pay attention to the quality of your code (ruff, mypy and type annotations). Our prek-hooks will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example Dag that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: [email protected]
Slack: https://s.apache.org/airflow-slack

jscheffl · 2026-05-16T09:19:20Z

This PR lags any description. Can you please add any intention?

jscheffl · 2026-05-16T11:45:09Z

Note: Before merging main hoping that simple static checks fix themself: Please check contribution docs how to run Prek that will fox most of the findings of CI: https://github.com/apache/airflow/blob/main/contributing-docs/README.rst

potiuk · 2026-05-24T18:15:06Z

@gingeekrishna A few things need addressing before review — see our Pull Request quality criteria.

❌ CI fails: CI image checks / Static checks (and possibly other checks — see the Checks tab for the full list).

No rush.

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

Drafted-by: Claude Code (Opus 4.7); reviewed by @potiuk before posting

potiuk · 2026-06-09T08:27:27Z

@gingeekrishna Converting to draft — this PR does not yet meet our Pull Request quality criteria.

❌ Merge conflicts with main. See docs.

See the linked criteria for how to fix each item, then mark the PR "Ready for review". This is not a rejection — just an invitation to bring the PR up to standard. No rush.

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

gingeekrishna · 2026-06-09T09:53:55Z

This PR lags any description. Can you please add any intention?

Apologies for that, I just updated the right description

jscheffl

I am not a K8sExecutor user so my experience is a bit limited. Would ask some other maintainer for opinion.

gingeekrishna · 2026-06-09T20:52:26Z

I am not a K8sExecutor user so my experience is a bit limited. Would ask some other maintainer for opinion.

Thank you @jscheffl!

eladkal · 2026-06-12T07:11:58Z

Please fix the tests

FAILED tests/kubernetes_tests/test_kubernetes_pod_operator.py::TestKubernetesPodOperatorSystem::test_full_pod_spec - kubernetes.client.exceptions.NotFoundException: (404)
Reason: Not Found
HTTP response headers: HTTPHeaderDict({'Audit-Id': 'de64a3d2-dbab-4e4a-befd-9ec7094844de', 'Cache-Control': 'no-cache, private', 'Content-Type': 'application/json', 'X-Kubernetes-Pf-Flowschema-Uid': 'a5a8be79-1bdd-4dda-a9ac-f32865598441', 'X-Kubernetes-Pf-Prioritylevel-Uid': '2961fe89-97fb-4331-a496-064497325545', 'Date': 'Thu, 11 Jun 2026 13:45:07 GMT', 'Content-Length': '202'})
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"test-pod-bbut043z\" not found","reason":"NotFound","details":{"name":"test-pod-bbut043z","kind":"pods"},"code":404}

gingeekrishna · 2026-06-12T19:58:54Z

Hi @eladkal — fixed! The test failure had two root causes in _cleanup_zombie_kpo_pods:

No phase filter — the pod list included all pods (Pending, Completed, newly submitted), not just Running ones as the feature intends. Added field_selector="status.phase=Running" to scope it correctly.
No age guard — a pod with no matching TaskInstance in the DB (including system-test pods like dag_id="dag") was immediately treated as a zombie and force-deleted. Added a minimum-age check: pods younger than zombie_kpo_pod_cleanup_interval seconds are now skipped.

Also rebased onto main to clean up the merge commits from the diff.

gingeekrishna · 2026-06-16T19:09:53Z

Hi @eladkal — fixed! The test failure had two root causes in _cleanup_zombie_kpo_pods:

No phase filter — the pod list included all pods (Pending, Completed, newly submitted), not just Running ones as the feature intends. Added field_selector="status.phase=Running" to scope it correctly.

No age guard — a pod with no matching TaskInstance in the DB (including system-test pods like dag_id="dag") was immediately treated as a zombie and force-deleted. Added a minimum-age check: pods younger than zombie_kpo_pod_cleanup_interval seconds are now skipped.

Also rebased onto main to clean up the merge commits from the diff.

Hi @eladkal

is that my PR good to merge now?

Signed-off-by: gingeekrishna <[email protected]>

… only Scoping the `task_instance` query by `dag_id.in_(pod_dag_ids)` bounds the result set to DAGs that actually have live KPO pods rather than scanning all non-terminal KPO TIs across the entire table. Signed-off-by: Radhakrishnan Pachyappan <[email protected]>

…d age guard Two bugs in _cleanup_zombie_kpo_pods caused spurious deletions: 1. No phase filter: the pod list included Pending/Completed pods and newly submitted pods, not just Running ones as the feature intends. Added field_selector="status.phase=Running" to _list_pods call. 2. No minimum-age guard: a pod created seconds ago with no matching TaskInstance yet (submit→schedule gap, or system-test pods) was immediately treated as a zombie. Now pods younger than zombie_kpo_pod_cleanup_interval seconds are skipped. Fixes the TestKubernetesPodOperatorSystem::test_full_pod_spec failure where the CI KubernetesExecutor was force-deleting the test pod because dag_id="dag" had no matching TaskInstance in the CI database. Also adds test_young_pod_skipped_by_age_guard and sets a realistic creation_timestamp in _make_pod so all zombie cleanup tests exercise the age guard correctly.

- Run ruff format on kubernetes_executor.py (dict arg expansion) - Remove datetime.timezone import that collided with airflow.sdk.timezone import later in the file, causing MyPy incompatible-import error; replace datetime.now(timezone.utc) with timezone.utcnow() to match existing test code style Signed-off-by: RadhaKrishnan Rajendran <[email protected]> Signed-off-by: gingeekrishna <[email protected]>

Resolve import conflict in kubernetes_executor.py: keep timezone (added by this branch) alongside dataclass (added by upstream).

potiuk · 2026-07-20T10:15:31Z

@gingeekrishna This PR has been converted to draft because it does not yet meet our Pull Request quality criteria.

Issues found:

❌ Build docs: Failing: CI image checks / Build documentation (--docs-only), CI image checks / Build documentation (--spellcheck-only). Run breeze build-docs locally to reproduce.
❌ Pre-commit / static checks: Failing: CI image checks / Static checks. Run prek run --from-ref main --stage pre-commit locally to reproduce and fix.
❌ Provider tests: Failing: Provider distributions tests / Compat 2.11.1:P3.10:, Provider distributions tests / Compat 3.0.6:P3.10:, Provider distributions tests / Compat 3.1.8:P3.10:, Provider distributions tests / Compat 3.2.2:P3.10:. Run breeze testing providers-tests --test-type "Providers[<provider>]" locally.

Note: Your branch is 85 commits behind main. Some check failures may be caused by changes in the base branch rather than by your PR. Please rebase your branch and push again to get up-to-date CI results.

What to do next:

Fix each issue listed above.
Make sure static checks pass locally (prek run --from-ref main --stage pre-commit).
Mark the PR as "Ready for review" when you're done.

Converting a PR to draft is not a rejection — it is an invitation to bring the PR up to the project's standards so that maintainer review time is spent productively. There is no rush — take your time and work at your own pace. We appreciate your contribution and are happy to wait for updates. If you have questions, feel free to ask on the Airflow Slack.

Note: This comment was drafted by an AI-assisted triage tool and may contain mistakes. Once you have addressed the points above, an Apache Airflow maintainer — a real person — will take the next look at your PR. We use this two-stage triage process so that our maintainers' limited time is spent where it matters most: the conversation with you.

Copilot AI review requested due to automatic review settings May 16, 2026 09:01

gingeekrishna requested review from hussein-awala, jedcunningham and jscheffl as code owners May 16, 2026 09:01

boring-cyborg Bot added area:providers kind:documentation provider:cncf-kubernetes Kubernetes (k8s) provider related issues labels May 16, 2026

gingeekrishna mentioned this pull request May 16, 2026

KPO pods can hang if sidecar never exits #58968

Open

1 of 2 tasks

jscheffl reviewed May 16, 2026

View reviewed changes

Comment thread providers/cncf/kubernetes/docs/changelog.rst Resolved

potiuk marked this pull request as draft June 9, 2026 08:27

gingeekrishna marked this pull request as ready for review June 9, 2026 09:20

gingeekrishna changed the title ~~Add zombie KPO pod cleanup feature for KubernetesPodOperator~~ feat: add zombie KPO pod cleanup to KubernetesExecutor Jun 9, 2026

gingeekrishna force-pushed the feature-58968-kpo-pod branch 3 times, most recently from 8bd6398 to 42fc62c Compare June 9, 2026 09:51

gingeekrishna requested a review from jscheffl June 9, 2026 09:52

jscheffl reviewed Jun 9, 2026

View reviewed changes

Comment thread ...iders/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py Resolved

Comment thread ...iders/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py Resolved

gingeekrishna force-pushed the feature-58968-kpo-pod branch from bd48312 to 357a17b Compare June 12, 2026 19:57

potiuk assigned gingeekrishna Jun 17, 2026

potiuk marked this pull request as draft June 18, 2026 21:47

gingeekrishna added 3 commits June 27, 2026 13:41

Add zombie KPO pod cleanup feature for KubernetesPodOperator

359e6f0

Signed-off-by: gingeekrishna <[email protected]>

gingeekrishna force-pushed the feature-58968-kpo-pod branch from 83e78e5 to 4a5e7b2 Compare June 27, 2026 08:12

gingeekrishna marked this pull request as ready for review June 27, 2026 08:13

gingeekrishna added 2 commits July 5, 2026 18:50

Merge upstream/main into feature-58968-kpo-pod

93bb839

Resolve import conflict in kubernetes_executor.py: keep timezone (added by this branch) alongside dataclass (added by upstream).

potiuk marked this pull request as draft July 20, 2026 10:15

potiuk closed this Jul 28, 2026

Conversation

gingeekrishna commented May 16, 2026 • edited by potiuk Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Test plan

Gen-AI disclosure

Uh oh!

boring-cyborg Bot commented May 16, 2026

Uh oh!

Uh oh!

jscheffl commented May 16, 2026

Uh oh!

jscheffl commented May 16, 2026

Uh oh!

potiuk commented May 24, 2026

Uh oh!

potiuk commented Jun 9, 2026

Uh oh!

gingeekrishna commented Jun 9, 2026

Uh oh!

jscheffl left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

gingeekrishna commented Jun 9, 2026

Uh oh!

eladkal commented Jun 12, 2026

Uh oh!

gingeekrishna commented Jun 12, 2026

Uh oh!

gingeekrishna commented Jun 16, 2026

Uh oh!

potiuk commented Jul 20, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

gingeekrishna commented May 16, 2026 •

edited by potiuk

Loading