Skip to content

Commit a33aade

Browse files
committed
Add todo re exception handling in dagrun create loop
In SchedulerJobRunner, in _create_dag_runs, we do except Exception: ... continue but this does not work when the exception is the result of a db error; in that case you need to roll the transaction back. To fix this we need either smaller transactions (e.g. one per dag run create attempt) or savepoints. either way it's a bit of a refactor. Issue: apache#59120
1 parent d51e572 commit a33aade

1 file changed

Lines changed: 4 additions & 0 deletions

File tree

‎airflow-core/src/airflow/jobs/scheduler_job_runner.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1829,6 +1829,10 @@ def _create_dag_runs(self, dag_models: Collection[DagModel], session: Session) -
18291829
# and continue to the next dag.
18301830
except Exception:
18311831
self.log.exception("Failed creating DagRun for %s", dag.dag_id)
1832+
# todo: continuing here does not work because session needs rollback
1833+
# but you need either to make smaller transactions and commit after every dag run
1834+
# or to use savepoints.
1835+
# https://github.com/apache/airflow/issues/59120
18321836
continue
18331837
if self._should_update_dag_next_dagruns(
18341838
dag,

0 commit comments

Comments
 (0)