Skip to content

Implement rollup (many-to-one partition mapper) #59294

Closed

Description

@dstandish

Body

provide the functionality that a Dag depends on an asset, but waits for more than one partition to be ready. (e.g., a Dag only fires when all the 31 partitions of the month arrive)

Introduce RollupMapper and the initial window types (Month, Week, Day), plus the scheduler plumbing that holds the downstream until all expected upstream keys are present, then fires once.

Committer

  • I acknowledge that I am a maintainer/committer of the Apache Airflow project.

Activity

  1. added
    kind:metaHigh-level information important to the community
    on Dec 11, 2025
  2. Lee-W commented on Dec 11, 2025

    @Lee-W
    Member

    part of #59005, we'll extract features from the PoC branch one by one and close these related issues.

  3. FrankPortman commented on Dec 23, 2025

    @FrankPortman

    Maybe not a v1 thing, but I can see it being a nice-to-have to essentially have a decay of how many partitions you are okay being missing, as a function of how far past some job date timestamp the universe time is. In other words, ideally wait for all 31 partitions of the month, but if we're 3 days past due, be okay with, say, 28/31 for practical purposes. Is this something that you think will be easy to bolt on either in userland or in official Airflowland, or would that be a more involved re-design based on how you guys are thinking about solving this?

  4. moved this from Todo to Backlog in AIP-76 Asset Partitioningon Jan 21, 2026
  5. moved this from Backlog to Todo in AIP-76 Asset Partitioningon Feb 6, 2026
  6. moved this from Todo to Backlog in AIP-76 Asset Partitioningon Feb 6, 2026
  7. Lee-W commented on Feb 6, 2026

    @Lee-W
    Member

    Or should we remove to_upstream for the time being (3.2)

  8. uranusjr commented on Feb 6, 2026

    @uranusjr
    Member

    Chatted with @Lee-W and @dstandish earlier. For 3.2 let’s

  9. moved this from Backlog to Todo in AIP-76 Asset Partitioningon Feb 6, 2026
  10. uranusjr commented on Feb 10, 2026

    @uranusjr
    Member

    PR to remove to_upstream: #61706.

    It seems trivial to change to_downstream to an iterable. It is used only exactly once to create downstream dag runs, so we really only need a real use case. Not sure who to ask yet.

  11. 1 remaining item

  12. uranusjr commented on Feb 23, 2026

    @uranusjr
    Member

    Also, not for 3.2, but I’m wondering if we can avoid to_upstream entirely. The problem I have with it is to_downstream and to_upstream must be consistent with each other, otherwise Airflow would crash and burn. Maybe it would be possible to keep upstream-downstream relationship in the database when to_downstream is called so it can be used instead? Not sure yet.

  13. uranusjr commented on Feb 23, 2026

    @uranusjr
    Member

    See #62346

  14. moved this from Todo to In Progress in AIP-76 Asset Partitioningon Feb 23, 2026
  15. Lee-W commented on Feb 24, 2026

    @Lee-W
    Member

    thoughts from @cmarteepants

    A few concrete scenarios for to_downstream(self) -> str | Iterable[str]:

    • Weekly model training --> daily inference: ML model that's expensive to train (only needs retraining weekly) but runs inference daily against fresh data. When weekly partition completes, it needs to enable 7 daily inference partitions.
    • Weekly data deliver --> daily processing: External vendor provides weekly data files (contractually they only deliver weekly), but business needs daily processing/reporting. When the weekly file arrives, kick of 7 daily processing jobs for the week that just completed.
    • Weekly aggregations --> daily dashboard: Expensive weekly rollups that get servedthrough a daily-sliced dashboard or API.

    For these to work, to_downstream("2026-W10") would need to return something like: ["2026-03-04", "2026-03-05", "2026-03-06", "2026-03-07", "2026-03-08", "2026-03-09", "2026-03-10"]

    The tricky part is there are two different temporal relationships:

    Backward looking: Weekly W10 (March 4-10) enables 7 daily partitions WITHIN that week (March 4-10). All 7 would trigger at once when weekly completes.
    Forward-looking: Weekly W10 enables the NEXT 7 daily partitions (March 11-17). They trigger according to their own schedule as each data arrives.

  16. moved this from In Progress to Backlog in AIP-76 Asset Partitioningon Feb 24, 2026
  17. TJaniF commented on Feb 24, 2026

    @TJaniF
    Contributor

    Adding to the above list by @cmarteepants and @Lee-W:

    One-to-many would also be interesting for segment partitions. E.g. creating a daily report, with X partitioned Dag runs all running at the same time each creating the report for a different department. Currently this would be solved with DTM but has the disadvantage that you can't easily rerun the whole Dag just for one department (if, for example, the data for R&D would often be incomplete etc).

    From what I understand this and other segment-based partitions would depend on the ability to "attach" partitions from within upstream Dags, similar to how you can attach an extra to an AssetEvent right now.

    Something like

    my_asset_1 = Asset("asset1")
     
    @task(outlets=[my_asset_1])
    def t1():
        yield Partitions(my_asset_1, ["R&D", "Marketing", "Finance"])
     
        return "hello :)"

    Also worth thinking through what if the daily report for Marketing needs data from 2 different locations, e.g. asset1 partitioned with "Marketing" and asset2 partitioned with marketing.

  18. uranusjr commented on Feb 25, 2026

    @uranusjr
    Member

    Moving this back to the backlog for now since we’ve done what we want for 3.2. The rest will be in 3.3.

  19. self-assigned this
    on Mar 30, 2026
  20. moved this from Backlog to In Progress in AIP-76 Asset Partitioningon Apr 2, 2026
  21. changed the title [-]Implement rollup / windowing[/-] [+]Implement rollup[/+] on Apr 15, 2026
  22. changed the title [-]Implement rollup[/-] [+]Implement rollup (many-to-one partition mapper)[/+] on Apr 24, 2026
  23. moved this from In Progress to In Review in AIP-76 Asset Partitioningon Apr 28, 2026
  24. added this to the Airflow 3.3.0 milestone on Jun 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

kind:featureFeature Requestskind:metaHigh-level information important to the community

Type

No type

Projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions