Skip to content

Commit c9e190f

Browse files
authored
Add todo re exception handling in dagrun create loop (#59121)
In SchedulerJobRunner, in _create_dag_runs, we do except Exception: ... continue but this does not work when the exception is the result of a db error; in that case you need to roll the transaction back. To fix this we need either smaller transactions (e.g. one per dag run create attempt) or savepoints. either way it's a bit of a refactor. Issue: #59120
1 parent 95adec4 commit c9e190f

1 file changed

Lines changed: 4 additions & 0 deletions

File tree

‎airflow-core/src/airflow/jobs/scheduler_job_runner.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1831,6 +1831,10 @@ def _create_dag_runs(self, dag_models: Collection[DagModel], session: Session) -
18311831
# and continue to the next serdag.
18321832
except Exception:
18331833
self.log.exception("Failed creating DagRun for %s", serdag.dag_id)
1834+
# todo: continuing here does not work because session needs rollback
1835+
# but you need either to make smaller transactions and commit after every dag run
1836+
# or to use savepoints.
1837+
# https://github.com/apache/airflow/issues/59120
18341838
continue
18351839
if self._should_update_dag_next_dagruns(
18361840
serdag,

0 commit comments

Comments
 (0)