You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
provide the functionality that a Dag depends on an asset, but waits for more than one partition to be ready. (e.g., a Dag only fires when all the 31 partitions of the month arrive)
Introduce RollupMapper and the initial window types (Month, Week, Day), plus the scheduler plumbing that holds the downstream until all expected upstream keys are present, then fires once.
Committer
I acknowledge that I am a maintainer/committer of the Apache Airflow project.
Maybe not a v1 thing, but I can see it being a nice-to-have to essentially have a decay of how many partitions you are okay being missing, as a function of how far past some job date timestamp the universe time is. In other words, ideally wait for all 31 partitions of the month, but if we're 3 days past due, be okay with, say, 28/31 for practical purposes. Is this something that you think will be easy to bolt on either in userland or in official Airflowland, or would that be a more involved re-design based on how you guys are thinking about solving this?
It seems trivial to change to_downstream to an iterable. It is used only exactly once to create downstream dag runs, so we really only need a real use case. Not sure who to ask yet.
Also, not for 3.2, but I’m wondering if we can avoid to_upstream entirely. The problem I have with it is to_downstream and to_upstream must be consistent with each other, otherwise Airflow would crash and burn. Maybe it would be possible to keep upstream-downstream relationship in the database when to_downstream is called so it can be used instead? Not sure yet.
A few concrete scenarios for to_downstream(self) -> str | Iterable[str]:
Weekly model training --> daily inference: ML model that's expensive to train (only needs retraining weekly) but runs inference daily against fresh data. When weekly partition completes, it needs to enable 7 daily inference partitions.
Weekly data deliver --> daily processing: External vendor provides weekly data files (contractually they only deliver weekly), but business needs daily processing/reporting. When the weekly file arrives, kick of 7 daily processing jobs for the week that just completed.
Weekly aggregations --> daily dashboard: Expensive weekly rollups that get servedthrough a daily-sliced dashboard or API.
For these to work, to_downstream("2026-W10") would need to return something like: ["2026-03-04", "2026-03-05", "2026-03-06", "2026-03-07", "2026-03-08", "2026-03-09", "2026-03-10"]
The tricky part is there are two different temporal relationships:
Backward looking: Weekly W10 (March 4-10) enables 7 daily partitions WITHIN that week (March 4-10). All 7 would trigger at once when weekly completes.
Forward-looking: Weekly W10 enables the NEXT 7 daily partitions (March 11-17). They trigger according to their own schedule as each data arrives.
One-to-many would also be interesting for segment partitions. E.g. creating a daily report, with X partitioned Dag runs all running at the same time each creating the report for a different department. Currently this would be solved with DTM but has the disadvantage that you can't easily rerun the whole Dag just for one department (if, for example, the data for R&D would often be incomplete etc).
From what I understand this and other segment-based partitions would depend on the ability to "attach" partitions from within upstream Dags, similar to how you can attach an extra to an AssetEvent right now.
Also worth thinking through what if the daily report for Marketing needs data from 2 different locations, e.g. asset1 partitioned with "Marketing" and asset2 partitioned with marketing.
Activity
Lee-W commentedon Dec 11, 2025
part of #59005, we'll extract features from the PoC branch one by one and close these related issues.
FrankPortman commentedon Dec 23, 2025
Maybe not a v1 thing, but I can see it being a nice-to-have to essentially have a decay of how many partitions you are okay being missing, as a function of how far past some job date timestamp the universe time is. In other words, ideally wait for all 31 partitions of the month, but if we're 3 days past due, be okay with, say, 28/31 for practical purposes. Is this something that you think will be easy to bolt on either in userland or in official Airflowland, or would that be a more involved re-design based on how you guys are thinking about solving this?
Lee-W commentedon Feb 5, 2026
https://github.com/apache/airflow/pull/59005/changes#diff-072241d81275cd4b4b867f51025e9dca800610069305cffeb4b77ad45e135557R1708
Lee-W commentedon Feb 6, 2026
Or should we remove
to_upstreamfor the time being (3.2)uranusjr commentedon Feb 6, 2026
Chatted with @Lee-W and @dstandish earlier. For 3.2 let’s
to_upstreamfor now. Remove to_upstream contract from PartitionMapper #61706to_downstreamto return multiple partition keys to enable one-to-many mapping Allow to_downstream() to return more than one key #62346uranusjr commentedon Feb 10, 2026
PR to remove
to_upstream: #61706.It seems trivial to change
to_downstreamto an iterable. It is used only exactly once to create downstream dag runs, so we really only need a real use case. Not sure who to ask yet.1 remaining item
uranusjr commentedon Feb 23, 2026
Also, not for 3.2, but I’m wondering if we can avoid
to_upstreamentirely. The problem I have with it isto_downstreamandto_upstreammust be consistent with each other, otherwise Airflow would crash and burn. Maybe it would be possible to keep upstream-downstream relationship in the database whento_downstreamis called so it can be used instead? Not sure yet.uranusjr commentedon Feb 23, 2026
See #62346
Lee-W commentedon Feb 24, 2026
thoughts from @cmarteepants
A few concrete scenarios for
to_downstream(self) -> str | Iterable[str]:For these to work,
to_downstream("2026-W10")would need to return something like:["2026-03-04", "2026-03-05", "2026-03-06", "2026-03-07", "2026-03-08", "2026-03-09", "2026-03-10"]The tricky part is there are two different temporal relationships:
Backward looking: Weekly W10 (March 4-10) enables 7 daily partitions WITHIN that week (March 4-10). All 7 would trigger at once when weekly completes.
Forward-looking: Weekly W10 enables the NEXT 7 daily partitions (March 11-17). They trigger according to their own schedule as each data arrives.
TJaniF commentedon Feb 24, 2026
Adding to the above list by @cmarteepants and @Lee-W:
One-to-many would also be interesting for segment partitions. E.g. creating a daily report, with X partitioned Dag runs all running at the same time each creating the report for a different department. Currently this would be solved with DTM but has the disadvantage that you can't easily rerun the whole Dag just for one department (if, for example, the data for R&D would often be incomplete etc).
From what I understand this and other segment-based partitions would depend on the ability to "attach" partitions from within upstream Dags, similar to how you can attach an extra to an AssetEvent right now.
Something like
Also worth thinking through what if the daily report for Marketing needs data from 2 different locations, e.g. asset1 partitioned with "Marketing" and asset2 partitioned with marketing.
uranusjr commentedon Feb 25, 2026
Moving this back to the backlog for now since we’ve done what we want for 3.2. The rest will be in 3.3.
[-]Implement rollup / windowing[/-][+]Implement rollup[/+][-]Implement rollup[/-][+]Implement rollup (many-to-one partition mapper)[/+]