You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Batch and memoize upstream map-length lookups for mapped stub arg bindings #70548
Since #69757, ti_run derives per-map-index arg bindings for mapped @task.stub tasks via SchedulerDictOfListsExpandInput.resolve_expansion_sub_indexes → _get_map_lengths (airflow-core/src/airflow/models/expandinput.py). _get_map_lengths carries a pre-existing TODO:
# TODO: This initiates one database call for each XComArg. Would it be# more efficient to do one single db call and unpack the value here?
Upstream map lengths are immutable once populated for a DagRun, yet every ti_run of every map index re-queries them: a stub expanded over K XComArg kwargs producing N map indexes issues N×K identical get_task_map_length queries per DagRun, on the hot ti_run route. (Stubs expanding over a single kwarg short-circuit and skip the lookups; literal kwargs cost len() only.)
What needs to happen
Batch the per-kwarg length lookups into a single query (addressing the pre-existing _get_map_lengths TODO).
Consider memoizing resolved lengths per (dag_id, run_id, task_id) so repeated ti_run calls for sibling map indexes stop re-querying.
Acceptance criteria
One ti_run of a mapped stub issues at most one map-length query.
Activity