Skip to content

Fail KubernetesPodOperator when on_kill interrupted the pod - #72606

Open
itz-puneet wants to merge 2 commits into
apache:mainfrom
itz-puneet:kpo-interrupted-not-success
Open

itz-puneet wants to merge 2 commits into
apache:mainfrom
itz-puneet:kpo-interrupted-not-success

Conversation

@itz-puneet

@itz-puneet itz-puneet commented Sep 6, 2026 •

Copy link
Copy Markdown

closes: #71202

The bug

Under KubernetesExecutor there are two pods: the per-task pod the executor creates, and the child
pod KubernetesPodOperator.execute() creates for the user's workload. When both are interrupted
within about a second of each other, the task instance is recorded as success even though the
child pod never finished. A second user confirmed the same behaviour on their production clusters in
the issue thread.

The sequence:

  1. SIGTERM reaches the task process, and the runner calls on_kill().
  2. on_kill() sets self._killed and deletes the child pod.
  3. The wait inside execute_sync — await_pod_completion, following container logs — returns
    normally rather than raising, because the pod it was watching has simply gone away.
  4. The finally block runs post_complete_action → cleanup, and cleanup returns early on
    self._killed before reaching any of its failure signalling.
  5. execute_sync falls through to its return, so the operator reports success.

cleanup's early return is correct on its own terms — it exists so the pod is not deleted a second
time, since a duplicate delete can raise and trigger a spurious retry. The gap is that skipping the
deletion also skips the only place that would have failed the task.

The fix

Raise from execute_sync when the body completed while _killed is set.

The placement matters and is the reason this is not done inside cleanup: cleanup is called from a
finally block, so raising there would replace an exception that is already propagating. When the
body failed on its own, that failure is what the operator should surface — it already fails the task,
with a more specific reason. Putting the check after the finally means it only fires on the path
that would otherwise have returned success. cleanup keeps its early return unchanged, so the
double-delete it guards against still cannot happen.

Tests

Two regression tests in test_pod.py:

  • test_execute_sync_fails_when_on_kill_ran_during_the_wait — drives the exact sequence above, with
    the wait returning normally after on_kill() fires. Fails on main (the call returns instead of
    raising), passes with this change.
  • test_execute_sync_keeps_the_original_error_when_on_kill_ran — the body raises after on_kill();
    asserts the original exception still propagates rather than being replaced. This is what pins the
    placement of the check.

Neither test needs a cluster.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Claude Opus 5) following the guidelines

Under KubernetesExecutor the per-task pod and the KubernetesPodOperator
child pod can be interrupted within about a second of each other. SIGTERM
reaches the task process, the runner calls on_kill(), which sets _killed
and deletes the child pod, and the wait in execute_sync then returns
normally because the pod it was watching has gone away.

cleanup() returns early once _killed is set, so it never reaches its
failure signalling. execute_sync then fell through to its return and the
task instance was recorded as success, even though the workload never
finished.

Raise instead when the body completed while _killed is set. The check
sits after the finally block deliberately: when the body raised, that
exception keeps propagating and already fails the task with its own
reason, so this cannot mask it. cleanup() keeps its early return, so the
pod is still not deleted twice.

Closes: apache#71202
Copilot AI lite review requested due to automatic review settings September 6, 2026 23:29
@itz-puneet
itz-puneet requested a review from jscheffl as a code owner September 6, 2026 23:29
@boring-cyborg boring-cyborg Bot added area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues labels Sep 6, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

It introduces a new direct raise AirflowException(...) (disallowed by project policy / prek hook) and the tests/docstrings should be adjusted to comply with the same guidelines.

Once you've addressed the issues Copilot identified, you can request another Copilot review.

Pull request overview

Fixes an edge case in KubernetesPodOperator.execute_sync() where a task could be finalized as success even though the child pod was deleted by on_kill() during the completion wait (common under KubernetesExecutor when both task pod and child pod are interrupted close together).

Changes:

  • Add a post-finally guard in execute_sync() to fail the task if on_kill() ran during the wait path and the method would otherwise return success.
  • Add regression tests covering (a) the “wait returns normally after on_kill” success-bug path and (b) preserving the original exception when the body already failed.
File summaries
File Description
providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py Adds a post-cleanup failure check when _killed is set to prevent false-success outcomes.
providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod.py Adds two unit regression tests reproducing the interrupted-pod success bug and ensuring exceptions aren’t masked.
Review details
  • Files reviewed: 2/2 changed files
  • Comments generated: 4
  • Review effort level: Lite

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment on lines +841 to +845
pod = MagicMock()
pod.metadata.name = "sleep-worker"
pod.metadata.namespace = "default"
pod.status.phase = PodPhase.RUNNING
return pod

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving this one as is: the surrounding module builds pod mocks with bare MagicMock() throughout, and spec_set on V1Pod plus its nested status/metadata objects would make these two tests inconsistent with the rest of the file. Happy to switch if a maintainer would rather standardise on it.

The check-no-new-airflow-exceptions prek hook caps the number of
"raise AirflowException" occurrences per file against
generated/known_airflow_exceptions.txt, and directs new code to a
dedicated exception class instead. Both files here were over their limit:
operators/pod.py allows 3 and had 4, and the test module allows none.

Add PodInterrupted alongside the other pod exceptions and raise that. The
test asserting the interrupted path now expects it by type, and the test
covering an already-failing body raises RuntimeError, which makes the
point better anyway: any in-flight exception survives, not just an
Airflow one.

Also fall back to self.name when self.pod is unset, so the message cannot
read "Pod None", and drop the issue reference from the test docstring.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers provider:cncf-kubernetes Kubernetes (k8s) provider related issues

Projects

None yet

2 participants