Repository navigation
Commit d01ff90
committed
Don't deactivate DAG bundles owned by other dag-processors
`DagBundlesManager.sync_bundles_to_db()` marks every stored bundle that is
not in the calling process's config as inactive. This assumes the caller
sees the complete bundle configuration.
When multiple dag-processors are each configured with a partial config (one
bundle per processor, a natural way to shard parsing), each processor treats
the other processors' bundles as "no longer found in config" and disables
them. Processor A disables B's bundle, B disables A's, and they flip
`dag_bundle.active` on every parse cycle -- the deployment never converges
and continuously logs "DAG bundle ... is no longer found in config and has
been disabled" for bundles that are actively configured elsewhere.
Add a `deactivate_missing` flag (default `True`, preserving existing
single-processor behavior) and have `DagFileProcessorManager.sync_bundles()`
pass `deactivate_missing=False` when the processor was started with a bundle
filter (`--bundle-name` / `bundle_names_to_parse`), i.e. when it only owns a
subset of the bundles.
closes: apache#69963
related: apache#69698
Generated-by: Claude Code following the guidelines1 parent 10e310e commit d01ff90
4 files changed
Lines changed: 87 additions & 2 deletions
File tree
- airflow-core
- src/airflow/dag_processing
- bundles
- tests/unit/dag_processing
- bundles
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
271 | 271 | | |
272 | 272 | | |
273 | 273 | | |
274 | | - | |
| 274 | + | |
275 | 275 | | |
276 | 276 | | |
277 | 277 | | |
| |||
280 | 280 | | |
281 | 281 | | |
282 | 282 | | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
283 | 291 | | |
284 | 292 | | |
285 | 293 | | |
| |||
356 | 364 | | |
357 | 365 | | |
358 | 366 | | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
359 | 372 | | |
360 | 373 | | |
361 | 374 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
333 | 333 | | |
334 | 334 | | |
335 | 335 | | |
336 | | - | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
337 | 339 | | |
338 | 340 | | |
339 | 341 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
119 | 119 | | |
120 | 120 | | |
121 | 121 | | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
122 | 130 | | |
123 | 131 | | |
124 | 132 | | |
| |||
217 | 225 | | |
218 | 226 | | |
219 | 227 | | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
220 | 276 | | |
221 | 277 | | |
222 | 278 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
295 | 295 | | |
296 | 296 | | |
297 | 297 | | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
298 | 312 | | |
299 | 313 | | |
300 | 314 | | |
| |||
0 commit comments