Repository navigation
Conversation
A DagRun created pinned to a dag version (bundle_version set, created_dag_version_id populated) becomes unresolvable if that dag_version row is later deleted: the FK's ondelete="set null" clears created_dag_version_id while bundle_version stays set, so the pinned resolver returns None (it only falls back to the latest version for unpinned runs). _start_queued_dagruns then could not resolve the serialized DAG and skipped the run with "DAG '...' not found in serialized_dag table" on every loop - the run was neither started nor failed and stayed QUEUED indefinitely, with no failure signal and a misleading log (the DAG is in serialized_dag; the pinned version is not). Fail such a run explicitly, mirroring how a SCHEDULED task instance whose serialized DAG cannot be found is already failed in the same loop. Add a regression test that the pinned-unresolvable run reaches FAILED instead of being skipped forever, plus a contrast test that an unpinned run with the same NULL created_dag_version_id still falls back to the latest version and starts. Signed-off-by: kjh0623 <[email protected]>
|
The failing "Tests (AMD)" job is not related to this change — it's a CI/MySQL DB-init issue, not a real test failure. All 12 failures are Migration Could a committer please re-run the failed jobs? 🙏 |
check-newsfragments-are-valid requires non-significant newsfragments to be a single line; the detailed explanation belongs in the PR description, not the newsfragment. Signed-off-by: kjh0623 <[email protected]>
check-newsfragment-pr-number requires the file to be named after the PR, not the issue it fixes: 70056.bugfix.rst -> 70113.bugfix.rst. Signed-off-by: kjh0623 <[email protected]>
potiuk
added
the
ready for maintainer review
|
Closing this. #71696 fixed the cause on main: |
closes: #70056
Problem
A
DagRuncreated pinned to a dag version (bundle_versionset,created_dag_version_idpopulated) becomes unresolvable if thatdag_versionrow is later deleted. The FK'sondelete="set null"clearscreated_dag_version_idwhilebundle_versionstays set, so the pinned resolver inDBDagBag._version_from_dag_runreturnsNone— it only falls back to the latest version for unpinned runs._start_queued_dagrunsthen can't resolve the serialized DAG and hitsif not dag: log.error(...); continue, so the run is neither started nor failed. It is re-selected and skipped on every scheduler loop forever, with no failure signal, no metric, and a misleading log (DAG '...' not found in serialized_dag table— the DAG is inserialized_dag; only the pinned version is gone). No timeout applies to a QUEUED run in this state; in production a run sat QUEUED for 8+ hours until manually cleared.Fix
Fail the run explicitly when its pinned version can't be resolved, mirroring how the same function already handles a SCHEDULED task instance whose serialized DAG can't be found (that path sets the TI to
FAILEDand moves on). The error message now names the unresolvedcreated_dag_version_id/bundle_version.This is option 1 from the issue. It doesn't change resolution for healthy or unpinned runs — only runs that are already unstartable get an explicit terminal state instead of being skipped indefinitely.
Tests
test_queued_dagrun_with_unresolvable_pinned_version_is_failed— reproduces the pinned+NULL state and asserts the run reachesFAILEDacross repeated loops instead of stayingQUEUED.test_queued_dagrun_without_bundle_version_falls_back_to_latest— contrast: same NULLcreated_dag_version_idbut unpinned still falls back to the latest serialized version and starts (RUNNING), so the fix doesn't touch that path.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code following the guidelines