Repository navigation
[v3-4-test] Fix scheduler memory spike when a Dag run has a large conf (#73907) - #76252
Merged
Merged
Conversation
shahar1
force-pushed
the
backport-73907-v3-4-test
branch
from
November 1, 2026 20:20
b73d085 to
a0f4c10
Compare
vatsrahul1001
force-pushed
the
v3-4-test
branch
from
November 4, 2026 09:12
10eae22 to
e63e6bb
Compare
vatsrahul1001
requested review from
amoghrajesh,
bugraoz93,
choo121600,
dheerajturaga,
ephraimbuddy,
gopidesupavan,
henry3260,
hussein-awala,
jason810496,
jedcunningham,
jscheffl,
o-nikolas,
pierrejeambrun,
rawwar,
shubhamraj-git and
vatsrahul1001
as code owners
November 4, 2026 09:12
apache#73907) * Add tests for the run conf loaded per task instance when scheduling TaskInstance.dag_run is a joined eager load, so a task-instance query that also selects dag_run.conf returns and decodes one copy of the conf per row. A scheduling pass does that twice: fetch_task_instances reads every task instance of the run, and _get_ready_tis re-reads the ones still waiting on their upstreams. A large conf on a run with many mapped task instances multiplies scheduler memory this way. * Fix scheduler memory spike when a Dag run has a large conf TaskInstance.dag_run is a joined eager load, so every row of a task-instance query also carries the run's conf. The scheduler fetches all task instances of each running Dag run on every loop, and re-reads the ones still waiting on their upstreams, so a large conf was decoded once per task instance: a 1 MB conf on a run with 500 mapped task instances took the scheduler from 150 MB to over 3 GB. Neither query needs the conf up front: the few callers that read it can load it on access, and the re-check only compares states. (cherry picked from commit ce01a73)
shahar1
added
the
use fork ci
shahar1
force-pushed
the
backport-73907-v3-4-test
branch
from
November 4, 2026 18:48
a0f4c10 to
5ae0532
Compare
github-actions
Bot
added
the
fork ci required
Contributor
|
Fork CI: green. CI for the latest commit passed in your fork. The pull request is ready for maintainer review. Pull requests from contributors run their CI in the contributor's own fork. GitHub Actions is free and has no minute limit on public forks, while the CI capacity of the Apache Software Foundation is shared by every ASF project and is finite. How this works: fork CI for external contributors. This comment was posted automatically by the fork CI monitor from templates drafted with an AI-assisted tool, so it may contain mistakes. An Apache Airflow maintainer, a real person, will review the pull request once its fork CI is green. Why the first pass is automated. |
github-actions
Bot
added
the
ready for maintainer review
Contributor
|
Hi maintainer, this PR was merged without a milestone set.
|
vatsrahul1001
pushed a commit
that referenced
this pull request
Nov 9, 2026
#73907) (#76252) * Add tests for the run conf loaded per task instance when scheduling TaskInstance.dag_run is a joined eager load, so a task-instance query that also selects dag_run.conf returns and decodes one copy of the conf per row. A scheduling pass does that twice: fetch_task_instances reads every task instance of the run, and _get_ready_tis re-reads the ones still waiting on their upstreams. A large conf on a run with many mapped task instances multiplies scheduler memory this way. * Fix scheduler memory spike when a Dag run has a large conf TaskInstance.dag_run is a joined eager load, so every row of a task-instance query also carries the run's conf. The scheduler fetches all task instances of each running Dag run on every loop, and re-reads the ones still waiting on their upstreams, so a large conf was decoded once per task instance: a 1 MB conf on a run with 500 mapped task instances took the scheduler from 150 MB to over 3 GB. Neither query needs the conf up front: the few callers that read it can load it on access, and the re-check only compares states. (cherry picked from commit ce01a73) Co-authored-by: Lucas Pimentel <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #73907 to
v3-4-test.The cherry-pick conflicted only in
DagRun.fetch_task_instances:v3-4-testdoes not have thecolumnsparameter frommain, so the backport keeps the existing query and adds.defer(DagRun.conf)to itsjoinedload(TI.dag_run). The_get_ready_tischange and the two tests apply unchanged.The two new tests in
airflow-core/tests/unit/models/test_dagrun.pyfail without the change and pass with it.(cherry picked from commit ce01a73)
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines