|
| 1 | +# Issue #56149: DockerOperator on_kill Does Not Respect auto_remove='force' |
| 2 | + |
| 3 | +## Problem Description |
| 4 | + |
| 5 | +When a DockerOperator task configured with `auto_remove='force'` is killed via the Airflow UI |
| 6 | +(e.g., "Mark as Failed" or "Clear"), the `on_kill()` method only **stops** the container but does |
| 7 | +not **remove** it. This leaves orphaned containers on the Docker host. |
| 8 | + |
| 9 | +### Symptoms |
| 10 | + |
| 11 | +1. Container remains in `Exited (137)` state after task kill |
| 12 | +2. Subsequent DAG runs with the same `container_name` fail with **409 Conflict** because the |
| 13 | + stopped container still exists |
| 14 | +3. Manual `docker rm` is required to clean up |
| 15 | + |
| 16 | +### Reproduction |
| 17 | + |
| 18 | +```python |
| 19 | +from airflow.sdk import DAG |
| 20 | +from airflow.providers.docker.operators.docker import DockerOperator |
| 21 | + |
| 22 | +with DAG("docker_test", ...) as dag: |
| 23 | + task = DockerOperator( |
| 24 | + task_id="long_running", |
| 25 | + image="alpine:latest", |
| 26 | + command=["sleep", "300"], |
| 27 | + container_name="my_container", |
| 28 | + auto_remove="force", # Should force-remove on any exit |
| 29 | + docker_url="unix:///var/run/docker.sock", |
| 30 | + ) |
| 31 | +``` |
| 32 | + |
| 33 | +Steps: |
| 34 | +1. Trigger the DAG and wait for the task to enter `running` state |
| 35 | +2. In the Airflow UI, select "Clear" with "Mark as Failed" checked |
| 36 | +3. Run `docker ps -a` on the host |
| 37 | +4. **Expected**: Container is removed |
| 38 | +5. **Actual**: Container shows `Exited (137)` and persists |
| 39 | + |
| 40 | +--- |
| 41 | + |
| 42 | +## Root Cause Analysis |
| 43 | + |
| 44 | +### The Broken Code Path: `on_kill()` |
| 45 | + |
| 46 | +**File**: `providers/docker/src/airflow/providers/docker/operators/docker.py:525-531` |
| 47 | + |
| 48 | +```python |
| 49 | +def on_kill(self) -> None: |
| 50 | + if self.hook.client_created: |
| 51 | + self.log.info("Stopping docker container") |
| 52 | + if self.container is None: |
| 53 | + self.log.info("Not attempting to kill container as it was not created") |
| 54 | + return |
| 55 | + self.cli.stop(self.container["Id"]) |
| 56 | + # ← Missing: no auto_remove check, no container removal |
| 57 | +``` |
| 58 | + |
| 59 | +The `on_kill()` method only calls `self.cli.stop()`. It completely ignores the `auto_remove` |
| 60 | +setting. |
| 61 | + |
| 62 | +### The Working Code Path: Normal Execution `finally` Block |
| 63 | + |
| 64 | +**File**: `providers/docker/src/airflow/providers/docker/operators/docker.py:457-461` |
| 65 | + |
| 66 | +```python |
| 67 | +finally: |
| 68 | + if self.auto_remove == "success": |
| 69 | + self.cli.remove_container(self.container["Id"]) |
| 70 | + elif self.auto_remove == "force": |
| 71 | + self.cli.remove_container(self.container["Id"], force=True) |
| 72 | +``` |
| 73 | + |
| 74 | +During normal execution (success or failure), the `_run_image_with_mounts()` method's `finally` |
| 75 | +block properly handles container removal based on the `auto_remove` setting. |
| 76 | + |
| 77 | +### Why the `finally` Block Doesn't Save Us |
| 78 | + |
| 79 | +When `on_kill()` is triggered: |
| 80 | + |
| 81 | +1. The task runner sends a kill signal to the running task |
| 82 | +2. `on_kill()` is called, which stops the container |
| 83 | +3. The `_run_image_with_mounts()` method may still be running (waiting on `self.cli.wait()`) |
| 84 | +4. The `wait()` call may return/raise after the container is stopped |
| 85 | +5. The `finally` block **may** execute, but by this point the container is already stopped with |
| 86 | + exit code 137 |
| 87 | +6. The `finally` block tries to remove the container, but depending on timing and the |
| 88 | + `auto_remove` value: |
| 89 | + - `"success"`: Won't remove (task didn't succeed) |
| 90 | + - `"force"`: Would remove, but **race condition** — the `finally` block may not execute |
| 91 | + if the process is killed before reaching it |
| 92 | + |
| 93 | +In practice, the `finally` block execution is **not guaranteed** when the task is killed |
| 94 | +externally, making `on_kill()` the correct place to handle cleanup. |
| 95 | + |
| 96 | +--- |
| 97 | + |
| 98 | +## Architecture of the DockerOperator |
| 99 | + |
| 100 | +### Container Lifecycle |
| 101 | + |
| 102 | +``` |
| 103 | +DockerOperator.execute() |
| 104 | + │ |
| 105 | + ▼ |
| 106 | +_run_image() |
| 107 | + │ |
| 108 | + ▼ |
| 109 | +_run_image_with_mounts() |
| 110 | + │ |
| 111 | + ├─→ self.cli.create_container(...) # auto_remove=False (line 405) |
| 112 | + │ └─→ Docker daemon does NOT auto-remove |
| 113 | + │ └─→ Airflow manages container lifecycle |
| 114 | + │ |
| 115 | + ├─→ self.cli.start(container) |
| 116 | + │ |
| 117 | + ├─→ try: |
| 118 | + │ └─→ self.cli.wait(container, timeout) |
| 119 | + │ └─→ Check exit code |
| 120 | + │ └─→ Return result |
| 121 | + │ |
| 122 | + └─→ finally: # Normal cleanup (line 457) |
| 123 | + ├─→ auto_remove == "success" → remove_container() |
| 124 | + └─→ auto_remove == "force" → remove_container(force=True) |
| 125 | +
|
| 126 | +
|
| 127 | +Kill signal received: |
| 128 | + │ |
| 129 | + ▼ |
| 130 | +on_kill() # Signal-based cleanup (line 525) |
| 131 | + │ |
| 132 | + └─→ self.cli.stop(container) # Only stops, doesn't remove |
| 133 | + └─→ Container enters "Exited (137)" |
| 134 | + └─→ ⚠️ Container persists on disk |
| 135 | +``` |
| 136 | + |
| 137 | +### Key Design Decisions |
| 138 | + |
| 139 | +1. **`auto_remove=False` at container creation** (line 405): The Docker daemon's auto-remove |
| 140 | + feature is intentionally disabled. Airflow manages the container lifecycle to control when |
| 141 | + removal happens (based on success/failure/force settings). |
| 142 | + |
| 143 | +2. **Three `auto_remove` modes** (line 255): |
| 144 | + - `"never"` (default): Container persists after execution |
| 145 | + - `"success"`: Remove only on successful completion |
| 146 | + - `"force"`: Always remove, regardless of exit status |
| 147 | + |
| 148 | +3. **`on_kill()` callback**: Invoked by the task runner (supervisor) when the task receives a |
| 149 | + kill signal (SIGTERM/SIGKILL). It's the last chance to clean up resources. |
| 150 | + |
| 151 | +--- |
| 152 | + |
| 153 | +## Proposed Solution |
| 154 | + |
| 155 | +### Fix: Add `auto_remove` Handling to `on_kill()` |
| 156 | + |
| 157 | +```python |
| 158 | +def on_kill(self) -> None: |
| 159 | + if self.hook.client_created: |
| 160 | + self.log.info("Stopping docker container") |
| 161 | + if self.container is None: |
| 162 | + self.log.info("Not attempting to kill container as it was not created") |
| 163 | + return |
| 164 | + self.cli.stop(self.container["Id"]) |
| 165 | + if self.auto_remove == "force": |
| 166 | + try: |
| 167 | + self.cli.remove_container(self.container["Id"], force=True) |
| 168 | + except APIError: |
| 169 | + self.log.warning("Failed to remove container %s", self.container["Id"]) |
| 170 | +``` |
| 171 | + |
| 172 | +**Key decisions**: |
| 173 | +- Only `"force"` triggers removal in `on_kill()` — `"success"` should not remove on kill since |
| 174 | + the task did not succeed |
| 175 | +- Wrap in `try/except APIError` to handle race conditions (container may already be removed by |
| 176 | + the `finally` block or Docker daemon) |
| 177 | + |
| 178 | +**Effort**: Tiny — 5 lines added to `on_kill()`. |
| 179 | + |
| 180 | +**File to modify**: |
| 181 | +- `providers/docker/src/airflow/providers/docker/operators/docker.py` — `on_kill()` at line 525 |
| 182 | + |
| 183 | +**Existing PR**: Draft PR #63737 addresses this issue. |
| 184 | + |
| 185 | +--- |
| 186 | + |
| 187 | +## Blind Spots and Potential Problems |
| 188 | + |
| 189 | +1. **Race condition with `finally` block**: If both `on_kill()` and the `finally` block execute, |
| 190 | + there will be a double-remove attempt. The `try/except APIError` handles this, but we should |
| 191 | + verify that `APIError` (from `docker.errors`) is the correct exception for "container not |
| 192 | + found" scenarios. `docker.errors.NotFound` (a subclass of `APIError`) is more precise. |
| 193 | + |
| 194 | +2. **`DockerSwarmOperator`**: This is a separate operator at |
| 195 | + `providers/docker/src/airflow/providers/docker/operators/docker_swarm.py`. It may have its |
| 196 | + own `on_kill()` with the same issue. Should be checked. |
| 197 | + |
| 198 | +3. **Timing of stop vs. remove**: `self.cli.stop()` sends SIGTERM and waits for the container to |
| 199 | + exit (with a timeout). If the container hasn't fully stopped when `remove_container(force=True)` |
| 200 | + is called, the `force=True` parameter ensures removal regardless. This should be safe. |
| 201 | + |
| 202 | +4. **Container not yet created**: The `self.container is None` check already handles the case |
| 203 | + where `on_kill()` is called before the container is created. |
| 204 | + |
| 205 | +5. **Network cleanup**: Stopping/removing a container should clean up associated networks, but |
| 206 | + if the container is part of a custom network, there may be edge cases. This is unlikely to be |
| 207 | + affected by this fix. |
| 208 | + |
| 209 | +6. **Test coverage**: Tests should cover: |
| 210 | + - `on_kill()` with `auto_remove="force"` — container should be removed |
| 211 | + - `on_kill()` with `auto_remove="success"` — container should NOT be removed |
| 212 | + - `on_kill()` with `auto_remove="never"` — container should NOT be removed |
| 213 | + - `on_kill()` when container is already removed (race condition) — no error |
| 214 | + - `on_kill()` when container was never created — no error (existing behavior) |
0 commit comments