Repository navigation
Fail KubernetesPodOperator when on_kill interrupted the pod - #72606
itz-puneet wants to merge 2 commits into
Conversation
Under KubernetesExecutor the per-task pod and the KubernetesPodOperator child pod can be interrupted within about a second of each other. SIGTERM reaches the task process, the runner calls on_kill(), which sets _killed and deletes the child pod, and the wait in execute_sync then returns normally because the pod it was watching has gone away. cleanup() returns early once _killed is set, so it never reaches its failure signalling. execute_sync then fell through to its return and the task instance was recorded as success, even though the workload never finished. Raise instead when the body completed while _killed is set. The check sits after the finally block deliberately: when the body raised, that exception keeps propagating and already fails the task with its own reason, so this cannot mask it. cleanup() keeps its early return, so the pod is still not deleted twice. Closes: apache#71202
There was a problem hiding this comment.
🟡 Changes recommended
It introduces a new direct raise AirflowException(...) (disallowed by project policy / prek hook) and the tests/docstrings should be adjusted to comply with the same guidelines.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Fixes an edge case in KubernetesPodOperator.execute_sync() where a task could be finalized as success even though the child pod was deleted by on_kill() during the completion wait (common under KubernetesExecutor when both task pod and child pod are interrupted close together).
Changes:
- Add a post-
finallyguard inexecute_sync()to fail the task ifon_kill()ran during the wait path and the method would otherwise return success. - Add regression tests covering (a) the “wait returns normally after on_kill” success-bug path and (b) preserving the original exception when the body already failed.
File summaries
| File | Description |
|---|---|
| providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/operators/pod.py | Adds a post-cleanup failure check when _killed is set to prevent false-success outcomes. |
| providers/cncf/kubernetes/tests/unit/cncf/kubernetes/operators/test_pod.py | Adds two unit regression tests reproducing the interrupted-pod success bug and ensuring exceptions aren’t masked. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 4
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| pod = MagicMock() | ||
| pod.metadata.name = "sleep-worker" | ||
| pod.metadata.namespace = "default" | ||
| pod.status.phase = PodPhase.RUNNING | ||
| return pod |
There was a problem hiding this comment.
Leaving this one as is: the surrounding module builds pod mocks with bare MagicMock() throughout, and spec_set on V1Pod plus its nested status/metadata objects would make these two tests inconsistent with the rest of the file. Happy to switch if a maintainer would rather standardise on it.
The check-no-new-airflow-exceptions prek hook caps the number of "raise AirflowException" occurrences per file against generated/known_airflow_exceptions.txt, and directs new code to a dedicated exception class instead. Both files here were over their limit: operators/pod.py allows 3 and had 4, and the test module allows none. Add PodInterrupted alongside the other pod exceptions and raise that. The test asserting the interrupted path now expects it by type, and the test covering an already-failing body raises RuntimeError, which makes the point better anyway: any in-flight exception survives, not just an Airflow one. Also fall back to self.name when self.pod is unset, so the message cannot read "Pod None", and drop the issue reference from the test docstring.
closes: #71202
The bug
Under
KubernetesExecutorthere are two pods: the per-task pod the executor creates, and the childpod
KubernetesPodOperator.execute()creates for the user's workload. When both are interruptedwithin about a second of each other, the task instance is recorded as success even though the
child pod never finished. A second user confirmed the same behaviour on their production clusters in
the issue thread.
The sequence:
on_kill().on_kill()setsself._killedand deletes the child pod.execute_sync—await_pod_completion, following container logs — returnsnormally rather than raising, because the pod it was watching has simply gone away.
finallyblock runspost_complete_action→cleanup, andcleanupreturns early onself._killedbefore reaching any of its failure signalling.execute_syncfalls through to itsreturn, so the operator reports success.cleanup's early return is correct on its own terms — it exists so the pod is not deleted a secondtime, since a duplicate delete can raise and trigger a spurious retry. The gap is that skipping the
deletion also skips the only place that would have failed the task.
The fix
Raise from
execute_syncwhen the body completed while_killedis set.The placement matters and is the reason this is not done inside
cleanup:cleanupis called from afinallyblock, so raising there would replace an exception that is already propagating. When thebody failed on its own, that failure is what the operator should surface — it already fails the task,
with a more specific reason. Putting the check after the
finallymeans it only fires on the paththat would otherwise have returned success.
cleanupkeeps its early return unchanged, so thedouble-delete it guards against still cannot happen.
Tests
Two regression tests in
test_pod.py:test_execute_sync_fails_when_on_kill_ran_during_the_wait— drives the exact sequence above, withthe wait returning normally after
on_kill()fires. Fails onmain(the call returns instead ofraising), passes with this change.
test_execute_sync_keeps_the_original_error_when_on_kill_ran— the body raises afteron_kill();asserts the original exception still propagates rather than being replaced. This is what pins the
placement of the check.
Neither test needs a cluster.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Claude Opus 5) following the guidelines