Skip to content

Commit c9d83d6

Browse files
committed
Add deep analysis files for 3 open Airflow issues
Analysis of: - apache#56271: KubernetesExecutor broken in 3.1.0 (multi-executor validation) - apache#59707: --celery-hostname causes stuck tasks (stale broker connections) - apache#56149: DockerOperator on_kill ignores auto_remove='force' https://claude.ai/code/session_01GEn4wLCsXxwCHTaPrWjfBe
1 parent 598c9a9 commit c9d83d6

3 files changed

Lines changed: 664 additions & 0 deletions

File tree

Lines changed: 214 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,214 @@
1+
# Issue #56149: DockerOperator on_kill Does Not Respect auto_remove='force'
2+
3+
## Problem Description
4+
5+
When a DockerOperator task configured with `auto_remove='force'` is killed via the Airflow UI
6+
(e.g., "Mark as Failed" or "Clear"), the `on_kill()` method only **stops** the container but does
7+
not **remove** it. This leaves orphaned containers on the Docker host.
8+
9+
### Symptoms
10+
11+
1. Container remains in `Exited (137)` state after task kill
12+
2. Subsequent DAG runs with the same `container_name` fail with **409 Conflict** because the
13+
stopped container still exists
14+
3. Manual `docker rm` is required to clean up
15+
16+
### Reproduction
17+
18+
```python
19+
from airflow.sdk import DAG
20+
from airflow.providers.docker.operators.docker import DockerOperator
21+
22+
with DAG("docker_test", ...) as dag:
23+
task = DockerOperator(
24+
task_id="long_running",
25+
image="alpine:latest",
26+
command=["sleep", "300"],
27+
container_name="my_container",
28+
auto_remove="force", # Should force-remove on any exit
29+
docker_url="unix:///var/run/docker.sock",
30+
)
31+
```
32+
33+
Steps:
34+
1. Trigger the DAG and wait for the task to enter `running` state
35+
2. In the Airflow UI, select "Clear" with "Mark as Failed" checked
36+
3. Run `docker ps -a` on the host
37+
4. **Expected**: Container is removed
38+
5. **Actual**: Container shows `Exited (137)` and persists
39+
40+
---
41+
42+
## Root Cause Analysis
43+
44+
### The Broken Code Path: `on_kill()`
45+
46+
**File**: `providers/docker/src/airflow/providers/docker/operators/docker.py:525-531`
47+
48+
```python
49+
def on_kill(self) -> None:
50+
if self.hook.client_created:
51+
self.log.info("Stopping docker container")
52+
if self.container is None:
53+
self.log.info("Not attempting to kill container as it was not created")
54+
return
55+
self.cli.stop(self.container["Id"])
56+
# ← Missing: no auto_remove check, no container removal
57+
```
58+
59+
The `on_kill()` method only calls `self.cli.stop()`. It completely ignores the `auto_remove`
60+
setting.
61+
62+
### The Working Code Path: Normal Execution `finally` Block
63+
64+
**File**: `providers/docker/src/airflow/providers/docker/operators/docker.py:457-461`
65+
66+
```python
67+
finally:
68+
if self.auto_remove == "success":
69+
self.cli.remove_container(self.container["Id"])
70+
elif self.auto_remove == "force":
71+
self.cli.remove_container(self.container["Id"], force=True)
72+
```
73+
74+
During normal execution (success or failure), the `_run_image_with_mounts()` method's `finally`
75+
block properly handles container removal based on the `auto_remove` setting.
76+
77+
### Why the `finally` Block Doesn't Save Us
78+
79+
When `on_kill()` is triggered:
80+
81+
1. The task runner sends a kill signal to the running task
82+
2. `on_kill()` is called, which stops the container
83+
3. The `_run_image_with_mounts()` method may still be running (waiting on `self.cli.wait()`)
84+
4. The `wait()` call may return/raise after the container is stopped
85+
5. The `finally` block **may** execute, but by this point the container is already stopped with
86+
exit code 137
87+
6. The `finally` block tries to remove the container, but depending on timing and the
88+
`auto_remove` value:
89+
- `"success"`: Won't remove (task didn't succeed)
90+
- `"force"`: Would remove, but **race condition** — the `finally` block may not execute
91+
if the process is killed before reaching it
92+
93+
In practice, the `finally` block execution is **not guaranteed** when the task is killed
94+
externally, making `on_kill()` the correct place to handle cleanup.
95+
96+
---
97+
98+
## Architecture of the DockerOperator
99+
100+
### Container Lifecycle
101+
102+
```
103+
DockerOperator.execute()
104+
│
105+
▼
106+
_run_image()
107+
│
108+
▼
109+
_run_image_with_mounts()
110+
│
111+
├─→ self.cli.create_container(...) # auto_remove=False (line 405)
112+
│ └─→ Docker daemon does NOT auto-remove
113+
│ └─→ Airflow manages container lifecycle
114+
│
115+
├─→ self.cli.start(container)
116+
│
117+
├─→ try:
118+
│ └─→ self.cli.wait(container, timeout)
119+
│ └─→ Check exit code
120+
│ └─→ Return result
121+
│
122+
└─→ finally: # Normal cleanup (line 457)
123+
├─→ auto_remove == "success" → remove_container()
124+
└─→ auto_remove == "force" → remove_container(force=True)
125+
126+
127+
Kill signal received:
128+
│
129+
▼
130+
on_kill() # Signal-based cleanup (line 525)
131+
│
132+
└─→ self.cli.stop(container) # Only stops, doesn't remove
133+
└─→ Container enters "Exited (137)"
134+
└─→ ⚠️ Container persists on disk
135+
```
136+
137+
### Key Design Decisions
138+
139+
1. **`auto_remove=False` at container creation** (line 405): The Docker daemon's auto-remove
140+
feature is intentionally disabled. Airflow manages the container lifecycle to control when
141+
removal happens (based on success/failure/force settings).
142+
143+
2. **Three `auto_remove` modes** (line 255):
144+
- `"never"` (default): Container persists after execution
145+
- `"success"`: Remove only on successful completion
146+
- `"force"`: Always remove, regardless of exit status
147+
148+
3. **`on_kill()` callback**: Invoked by the task runner (supervisor) when the task receives a
149+
kill signal (SIGTERM/SIGKILL). It's the last chance to clean up resources.
150+
151+
---
152+
153+
## Proposed Solution
154+
155+
### Fix: Add `auto_remove` Handling to `on_kill()`
156+
157+
```python
158+
def on_kill(self) -> None:
159+
if self.hook.client_created:
160+
self.log.info("Stopping docker container")
161+
if self.container is None:
162+
self.log.info("Not attempting to kill container as it was not created")
163+
return
164+
self.cli.stop(self.container["Id"])
165+
if self.auto_remove == "force":
166+
try:
167+
self.cli.remove_container(self.container["Id"], force=True)
168+
except APIError:
169+
self.log.warning("Failed to remove container %s", self.container["Id"])
170+
```
171+
172+
**Key decisions**:
173+
- Only `"force"` triggers removal in `on_kill()` — `"success"` should not remove on kill since
174+
the task did not succeed
175+
- Wrap in `try/except APIError` to handle race conditions (container may already be removed by
176+
the `finally` block or Docker daemon)
177+
178+
**Effort**: Tiny — 5 lines added to `on_kill()`.
179+
180+
**File to modify**:
181+
- `providers/docker/src/airflow/providers/docker/operators/docker.py` — `on_kill()` at line 525
182+
183+
**Existing PR**: Draft PR #63737 addresses this issue.
184+
185+
---
186+
187+
## Blind Spots and Potential Problems
188+
189+
1. **Race condition with `finally` block**: If both `on_kill()` and the `finally` block execute,
190+
there will be a double-remove attempt. The `try/except APIError` handles this, but we should
191+
verify that `APIError` (from `docker.errors`) is the correct exception for "container not
192+
found" scenarios. `docker.errors.NotFound` (a subclass of `APIError`) is more precise.
193+
194+
2. **`DockerSwarmOperator`**: This is a separate operator at
195+
`providers/docker/src/airflow/providers/docker/operators/docker_swarm.py`. It may have its
196+
own `on_kill()` with the same issue. Should be checked.
197+
198+
3. **Timing of stop vs. remove**: `self.cli.stop()` sends SIGTERM and waits for the container to
199+
exit (with a timeout). If the container hasn't fully stopped when `remove_container(force=True)`
200+
is called, the `force=True` parameter ensures removal regardless. This should be safe.
201+
202+
4. **Container not yet created**: The `self.container is None` check already handles the case
203+
where `on_kill()` is called before the container is created.
204+
205+
5. **Network cleanup**: Stopping/removing a container should clean up associated networks, but
206+
if the container is part of a custom network, there may be edge cases. This is unlikely to be
207+
affected by this fix.
208+
209+
6. **Test coverage**: Tests should cover:
210+
- `on_kill()` with `auto_remove="force"` — container should be removed
211+
- `on_kill()` with `auto_remove="success"` — container should NOT be removed
212+
- `on_kill()` with `auto_remove="never"` — container should NOT be removed
213+
- `on_kill()` when container is already removed (race condition) — no error
214+
- `on_kill()` when container was never created — no error (existing behavior)

0 commit comments

Comments
 (0)