Repository navigation
Conversation
A Dag run triggered with a large conf makes scheduler memory scale with the number of task instances in the run. TaskInstance.dag_run is a joined eager load, so the run's conf comes back on every row of every task-instance query the scheduling loop makes, and a mapped task with several hundred instances repeats a multi-megabyte conf on every loop while the run is alive. Closes: apache#71267
bf9ce13 to
4bcc197
Compare
potiuk
added
the
closed because of open PR limit
|
Hello @rjgoyln - thank you for your contributions to Apache Airflow! The Airflow community has introduced a limit of 5 open pull requests at a time for contributors without write access to the repository. You currently have 24 open pull requests, so - as a one-time step of introducing the limit - we closed the ones where maintainers have not engaged yet:
These pull requests stay open because maintainers are already engaged in them - they count towards your limit:
This is not a judgement of you or of your changes. We never told contributors before that opening many pull requests at once was a problem, so there is nothing to feel bad about - and nothing is lost: your branches, commits and the review history stay where they are. What we ask you to do is to make your first prioritization decision: choose which of the pull requests above matter most to you, and reopen them (up to 5 open at a time, including the ones still open) with the "Reopen pull request" button or While your pull requests are waiting for review, the most valuable thing you can do is help in other ways - reviewing other contributors' pull requests, helping with issues, and taking part in the discussions on the devlist and Slack. Why we introduced the limit, what it means for you and how to reopen or restore a pull request is explained in https://github.com/apache/airflow/blob/main/contributing-docs/32_open_pull_request_limit.rst. Drafted-by: Claude Code (Opus 5); reviewed by @potiuk before posting |
Summary
A Dag run triggered with a large conf makes scheduler memory scale with the number of task instances in the run: while a mapped task is being scheduled the scheduler grows to several times its usual footprint, and recovers only once the run finishes.
TaskInstance.dag_runis a joined eager load, so the run'sconfcomes back on every row a task-instance query returns, even though the ORM keeps a singleDagRunobject — and nothing in the scheduling loop reads it. Scheduling a run with 200 mapped task instances and a 0.9 MB conf peaks at 372 MiB of heap, against 3.7 MiB once the column leaves those queries, at an identical query count.Change
DagRun.fetch_task_instancesand the scheduler's critical-section query deferDagRun.conf._get_ready_tis, the mapped-dependency check and mapped expansion skip thedag_runjoin entirely.A query whose callers use the run keeps it and drops only the column; one that reads nothing but task-instance state drops the join.
confstill loads on demand.The callback paths keep the column: they serialize
ti.dag_runinto the execution APIDagRundatamodel, so a batch of finishing task instances still fetches one copy each.Behavior change
Reading
ti.dag_run.confafter the loading session has closed now raisesDetachedInstanceErrorinstead of returning the eagerly loaded value. No caller in the tree does this.closes: #71267
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5) following the guidelines