Fix DagRun.dag_versions after partial run_on_latest_version clear by dgvj-work · Pull Request #71459 · apache/airflow

dgvj-work · 2026-08-11T19:42:26Z

Fix DagRun.dag_versions silently dropping older versions still in use after a run_on_latest_version clear.

For bundle-versioned runs the property previously returned only created_dag_version. Clearing with run_on_latest_version=True bumps that pointer to latest while uncleared TaskInstances or archived attempts keep the older dag_version_id, so the Grid UI could show a single version chip while tasks were still executing under mixed versions.

Changes

DagRun.dag_versions now derives versions from a single DISTINCT query over current and archived attempts (include_all_attempts=True) whenever the run is attached to a session. New routes get this automatically; no per-route helper call is needed.
attach_dag_versions_to_runs is the batch form of the same query, used by the list and grid endpoints. get_dag_run and patch_dag_run no longer call it. Bulk clear calls it before serializing the cleared runs.
The bundle_version shortcut that returned only created_dag_version is removed, so a run cleared with run_on_latest_version=True reports every version still present on its attempts. created_dag_version is used only when no attempt carries a version.
Rebased onto the attempt-archiving change (TaskInstanceHistory removed).

closes: #71454

Testing

uv run --project airflow-core pytest airflow-core/tests/unit/models/test_dagrun.py -k "dag_versions" -q
uv run --project airflow-core pytest airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_dag_run.py -k "unloaded or bundle_version or dag_versions" -q
uv run --project airflow-core pytest airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_grid.py -q

Was generative AI tooling used to co-author this PR?

Yes (please specify the tool below)

Generated-by: Cursor

Read the Pull Request Guidelines for more information.

pierrejeambrun

~~We also have #71425 to address the same issue.~~
Edit: #71425 isn't solving that issue.

@ephraimbuddy @dstandish wdyt?

This doesn't solve the contract problem mentioned in the other PR but can help sove the immediate reported issues with minimal effort. While we figure out the interface we want.

pierrejeambrun

A few nits, looking good to me.

dgvj-work · 2026-09-06T03:51:45Z

Added coverage for a full run_on_latest_version clear as well.

After every TI is cleared, current tasks are all on latest and the older version only remains on TIH. dag_versions and attach_dag_versions_to_runs now have a test for that path, next to the existing partial-clear case.

@ephraimbuddy this should be ready for another look.

dgvj-work · 2026-09-11T03:05:12Z

@ephraimbuddy could you please take another look at this ?

dgvj-work · 2026-09-14T03:06:25Z

get_dag_run() now prefetches dag versions with the DISTINCT helper, same as the list/grid path. A large mapped run no longer materializes every TI/TIH row just to serialize versions.

@ephraimbuddy this should be ready for another look.

dgvj-work · 2026-09-16T02:22:16Z

The GET test now asserts serialization leaves TI/TIH collections unloaded on a bundled run with history, so it fails if the prefetch is dropped.

@ephraimbuddy this should be ready for another look.

dgvj-work · 2026-10-08T03:58:41Z

patch_dag_run() now uses the same DISTINCT prefetch as GET, after the mutations. The PATCH test asserts serialization leaves TI/TIH collections unloaded on a bundled run with history. The model tests reload by primary key.

@ephraimbuddy this should be ready for another look.

dgvj-work · 2026-10-30T04:33:57Z

Rebased onto current main (attempt archiving: TIH is gone) and moved the DISTINCT lookup into DagRun.dag_versions itself, so clear_dag_run / clear_dag_runs / trigger_dag_run / materialize_asset get it without a per-route helper.

The query uses include_all_attempts=True so archived attempts stay visible after a full run_on_latest_version clear.

@ephraimbuddy would you mind another look?

Stop special-casing bundle-versioned runs through created_dag_version_id. Always derive versions from the run's TaskInstance / history rows so mixed versions after a partial clear are reported correctly. Closes apache#71454 Signed-off-by: Digvijay <[email protected]>

attach_dag_versions_to_runs previously skipped runs with bundle_version, assuming dag_versions used created_dag_version. After removing that shortcut, prefetch all runs so Grid/UI keeps the lightweight path and still sees mixed TI versions. Signed-off-by: Digvijay <[email protected]>

Shorten the dag_versions comment, drop the issue number from the test, and name the case for bundled DAGs. Signed-off-by: Digvijay <[email protected]>

A full clear moves every current TI to latest, so the older version only remains on TIH. Cover that path and keep prefetch from skipping bundle-versioned runs. Signed-off-by: Digvijay <[email protected]>

get_dag_run() skipped attach_dag_versions_to_runs, so opening a large mapped run materialized every TI/TIH row just to return versions. Signed-off-by: Digvijay <[email protected]>

Query count stayed flat even when each query loaded every TI/TIH row. Serialize a bundle-versioned run with history and assert those collections stay unloaded. Signed-off-by: Digvijay <[email protected]>

A note-only update still serializes dag_versions, so without the DISTINCT helper it walks every TI/TIH row. Reload tests by primary key because run_id is only unique within a Dag. Signed-off-by: Digvijay <[email protected]>

Session-attached runs now use one DISTINCT query over current and archived attempts, so clear, trigger, and other single-run routes serialize versions without walking TI collections. include_all_attempts=True is required after attempt archiving so a full run_on_latest_version clear still surfaces the older version. Signed-off-by: Digvijay <[email protected]>

Annotate fetch_dag_versions_for_runs, give the transient DagVersion a real id, and expire TI collections before asserting serialization so the check measures the DISTINCT fallback rather than leftover identity-map state from clear. Signed-off-by: Digvijay <[email protected]>

ephraimbuddy

Looks good. Moving the lookup into the property covers the clear routes, and include_all_attempts=True keeps the full-clear case. A few small comments inline.

Collapse the version fetch into one query, restore the lighter GET load, and keep only the response checks that still cover archived attempts. Signed-off-by: Digvijay <[email protected]>

dgvj-work · 2026-11-09T06:18:55Z

Addressed the latest review:

Version lookup is one statement (DISTINCT ids joined to DagVersion, bundle included). version_data stays out of the DISTINCT because it is JSON.
get_dag_run() no longer eager-loads created_dag_version.
Dropped the PATCH, single-clear, and trigger serialization tests and the helper that only re-checked dag_versions. GET and bulk clear still assert both versions in the response.
Transient-run test sets bundle_version. _dag_bag_from_serialized uses spec=DBDagBag.
Merged current main.

@ephraimbuddy this should be ready for another look.

Bulk clear serializes dag_versions, so attach the batch prefetch before building the response. Runs with no attempt version report created_dag_version. Signed-off-by: Digvijay <[email protected]>

ephraimbuddy · 2026-11-09T09:11:40Z

+            # Transient run (e.g. built in memory in tests): use the in-memory relationships.
+            versions = dict.fromkeys(list(self._historical_dag_versions) + list(self._ti_dag_versions))
+        result = sorted((dv for dv in versions if dv is not None), key=lambda dv: dv.id)
+        if not result and self.created_dag_version is not None:


When does this fire? Runs from before Dag versioning have created_dag_version_id NULL too, since 0047 adds the column without a backfill, so the docstring example never reaches it. Runs created since then get a version on their TIs at creation. This also changes unpinned runs, which returned [] here before. If it's for runs whose TIs were cleaned up, please say that in the docstring; otherwise I'd drop it.

dgvj-work requested review from XD-DENG, ashb, bugraoz93, choo121600, ephraimbuddy, henry3260, jason810496, pierrejeambrun, rawwar and shubhamraj-git as code owners August 11, 2026 19:42

pierrejeambrun reviewed Aug 27, 2026

View reviewed changes

pierrejeambrun approved these changes Aug 27, 2026

View reviewed changes

Comment thread airflow-core/tests/unit/models/test_dagrun.py Resolved

Comment thread airflow-core/tests/unit/models/test_dagrun.py Resolved

Comment thread airflow-core/src/airflow/models/dagrun.py Resolved

ephraimbuddy reviewed Sep 3, 2026

View reviewed changes

Comment thread airflow-core/tests/unit/models/test_dagrun.py Resolved

dgvj-work requested a review from ephraimbuddy September 11, 2026 03:04

ephraimbuddy reviewed Sep 11, 2026

View reviewed changes

Comment thread airflow-core/src/airflow/models/dagrun.py Resolved

ephraimbuddy reviewed Sep 14, 2026

View reviewed changes

Comment thread airflow-core/tests/unit/api_fastapi/core_api/routes/public/test_dag_run.py Resolved

dgvj-work force-pushed the fix/dag-versions-partial-run-on-latest branch from c3d7810 to 9c4a219 Compare September 16, 2026 02:36

ephraimbuddy reviewed Sep 18, 2026

View reviewed changes

Comment thread airflow-core/src/airflow/models/dagrun.py Resolved

Comment thread airflow-core/tests/unit/models/test_dagrun.py Resolved

ephraimbuddy reviewed Oct 21, 2026

View reviewed changes

Comment thread airflow-core/src/airflow/api_fastapi/core_api/routes/public/dag_run.py Resolved

dgvj-work force-pushed the fix/dag-versions-partial-run-on-latest branch from 6bd4fbe to da51f3c Compare October 30, 2026 04:33

dgvj-work added 4 commits October 29, 2026 23:35

Address review nits on bundled dag_versions tests

1774067

Shorten the dag_versions comment, drop the issue number from the test, and name the case for bundled DAGs. Signed-off-by: Digvijay <[email protected]>

Add full run_on_latest_version clear coverage for dag_versions

fbdcb05

A full clear moves every current TI to latest, so the older version only remains on TIH. Cover that path and keep prefetch from skipping bundle-versioned runs. Signed-off-by: Digvijay <[email protected]>

dgvj-work added 4 commits October 29, 2026 23:35

Prefetch dag versions on single DagRun GET

190d9d3

get_dag_run() skipped attach_dag_versions_to_runs, so opening a large mapped run materialized every TI/TIH row just to return versions. Signed-off-by: Digvijay <[email protected]>

Assert get_dag_run leaves TI collections unloaded

747a7a4

Query count stayed flat even when each query loaded every TI/TIH row. Serialize a bundle-versioned run with history and assert those collections stay unloaded. Signed-off-by: Digvijay <[email protected]>

Prefetch dag versions on DagRun PATCH

1343627

A note-only update still serializes dag_versions, so without the DISTINCT helper it walks every TI/TIH row. Reload tests by primary key because run_id is only unique within a Dag. Signed-off-by: Digvijay <[email protected]>

dgvj-work force-pushed the fix/dag-versions-partial-run-on-latest branch from da51f3c to 9e1b3ce Compare October 30, 2026 04:35

github-actions Bot added the need workflow approval label Oct 30, 2026

safaehar mentioned this pull request Oct 30, 2026

Prefetch dag_versions in the get Dag run endpoint #76031

Open

1 task

ephraimbuddy approved these changes Nov 3, 2026

View reviewed changes

github-actions Bot removed the need workflow approval label Nov 3, 2026

ephraimbuddy approved these changes Nov 3, 2026

View reviewed changes

dgvj-work added 2 commits November 9, 2026 00:16

Address review on dag_versions lookup and route tests

c8296b3

Collapse the version fetch into one query, restore the lighter GET load, and keep only the response checks that still cover archived attempts. Signed-off-by: Digvijay <[email protected]>

Merge main into fix/dag-versions-partial-run-on-latest

a685da2

Prefetch versions on bulk clear and fall back when attempts have none

e43dcdb

Bulk clear serializes dag_versions, so attach the batch prefetch before building the response. Runs with no attempt version report created_dag_version. Signed-off-by: Digvijay <[email protected]>

dgvj-work force-pushed the fix/dag-versions-partial-run-on-latest branch from 6fe210f to e43dcdb Compare November 9, 2026 06:32

github-actions Bot added need workflow approval and removed need workflow approval labels Nov 9, 2026

ephraimbuddy approved these changes Nov 9, 2026

View reviewed changes

ephraimbuddy enabled auto-merge (squash) November 9, 2026 09:18

Conversation

dgvj-work commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes

Testing

Was generative AI tooling used to co-author this PR?

Uh oh!

pierrejeambrun left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pierrejeambrun left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dgvj-work commented Sep 6, 2026

Uh oh!

dgvj-work commented Sep 11, 2026

Uh oh!

Uh oh!

dgvj-work commented Sep 14, 2026

Uh oh!

Uh oh!

dgvj-work commented Sep 16, 2026

Uh oh!

Uh oh!

Uh oh!

dgvj-work commented Oct 8, 2026

Uh oh!

Uh oh!

dgvj-work commented Oct 30, 2026

Uh oh!

ephraimbuddy left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dgvj-work commented Nov 9, 2026

Uh oh!

ephraimbuddy Nov 9, 2026

Choose a reason for hiding this comment

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

dgvj-work commented Aug 11, 2026 •

edited

Loading

pierrejeambrun left a comment •

edited

Loading