Repository navigation
Commit d1b1f74
fix(openlineage): self-heal ProcessPoolExecutor on BrokenProcessPool (#67400)
* fix(openlineage): self-heal ProcessPoolExecutor on BrokenProcessPool
When a child process in the OpenLineage listener's ProcessPoolExecutor
terminates abruptly, concurrent.futures marks the pool as permanently
broken. Every subsequent submission raises BrokenProcessPool and lineage
data stops flowing until the scheduler is restarted.
This adds self-healing: submit_callable now catches BrokenProcessPool,
shuts down the broken executor, creates a fresh one, and retries the
submission so lineage reporting recovers automatically.
Closes #67283
* fix(openlineage): sort imports to fix ruff I001 static check
ruff I001: `from concurrent.futures.process import BrokenProcessPool`
must follow `from concurrent.futures import ProcessPoolExecutor`
* fix(openlineage): collapse log.warning to single line for ruff-format
The warning message fits within the line-length limit so it should not be
split across three lines.
---------
Co-authored-by: Anmol Mishra <[email protected]>1 parent 99816d8 commit d1b1f74
2 files changed
Lines changed: 40 additions & 1 deletion
File tree
- providers/openlineage
- src/airflow/providers/openlineage/plugins
- tests/unit/openlineage/plugins
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
20 | 20 | | |
21 | 21 | | |
22 | 22 | | |
| 23 | + | |
23 | 24 | | |
24 | 25 | | |
25 | 26 | | |
| |||
1049 | 1050 | | |
1050 | 1051 | | |
1051 | 1052 | | |
1052 | | - | |
| 1053 | + | |
| 1054 | + | |
| 1055 | + | |
| 1056 | + | |
| 1057 | + | |
| 1058 | + | |
| 1059 | + | |
1053 | 1060 | | |
1054 | 1061 | | |
1055 | 1062 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2128 | 2128 | | |
2129 | 2129 | | |
2130 | 2130 | | |
| 2131 | + | |
| 2132 | + | |
| 2133 | + | |
| 2134 | + | |
| 2135 | + | |
| 2136 | + | |
| 2137 | + | |
| 2138 | + | |
| 2139 | + | |
| 2140 | + | |
| 2141 | + | |
| 2142 | + | |
| 2143 | + | |
| 2144 | + | |
| 2145 | + | |
| 2146 | + | |
| 2147 | + | |
| 2148 | + | |
| 2149 | + | |
| 2150 | + | |
| 2151 | + | |
| 2152 | + | |
| 2153 | + | |
| 2154 | + | |
| 2155 | + | |
| 2156 | + | |
| 2157 | + | |
| 2158 | + | |
| 2159 | + | |
| 2160 | + | |
| 2161 | + | |
| 2162 | + | |
2131 | 2163 | | |
2132 | 2164 | | |
2133 | 2165 | | |
| |||
0 commit comments