short_circuit operator ignore_downstream_trigger_rules not working on mapped tasks #55225

Activity

1. dv-matthew-crown

   added

   kind:bugThis is a clearly a bugThis is a clearly a bug

   area:core

   needs-triagelabel for new issues that we didn't triage yetlabel for new issues that we didn't triage yet

   on Sep 3, 2025
2. boring-cyborg commented on Sep 3, 2025

   

   boring-cyborgBot

   on Sep 3, 2025 – with boring-cyborg

   More actions

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
3. dosubot

   added

   area:dynamic-task-mappingAIP-42AIP-42

   on Sep 3, 2025
4. suhail-zemoso commented on Sep 4, 2025

   

   suhail-zemoso

   on Sep 4, 2025

   Contributor

   More actions

   I would like to work on this issue--Let me know if it's being picked by someone else.
5. vatsrahul1001

   added

   priority:mediumBug that should be fixed before next release but would not block a releaseBug that should be fixed before next release but would not block a release

   affected_version:3.0Issues Reported for 3.0Issues Reported for 3.0

   and removed

   needs-triagelabel for new issues that we didn't triage yetlabel for new issues that we didn't triage yet

   on Sep 4, 2025
6. suhail-zemoso commented on Sep 5, 2025

   

   suhail-zemoso

   on Sep 5, 2025

   Contributor

   More actions

   @dv-matthew-crown
   If you add trigger_rule="all_success" to the mapped task (dummy3 and published), we can achieve the expected behavior, which is that dummy2, dummy3, and published will be skipped.

   
7. dv-matthew-crown commented on Sep 5, 2025

   

   dv-matthew-crown

   on Sep 5, 2025

   Author

   More actions

   My mapped tasks require trigger rule none_failed when running normally, so this would not be a solution in my opinion
8. dv-matthew-crown commented on Sep 5, 2025

   

   dv-matthew-crown

   on Sep 5, 2025

   Author

   More actions

   @suhail-zemoso what is also interesting, is that despite the mapped tasks failing the overall DAG is marked as successful
9. phanikumv

   assigned

   vatsrahul1001

   on Oct 27, 2025
10. vatsrahul1001 commented on Oct 29, 2025

    

    vatsrahul1001

    on Oct 29, 2025

    Contributor

    More actions

    Not able to reproduce this with version 3.1.1. Closing the issue. Please feel free to reopen if you’re still facing it.
11. vatsrahul1001

    closed this as completedon Oct 29, 2025
12. Chais commented on Sep 30, 2026

    

    Chais

    on Sep 30, 2026

    More actions

    Commenting because I see this misbehaviour in 3.3.0. In fact it looks like short circuiting doesn't work as soon as mapped tasks are involved.
    Minimal example:

    from datetime import datetime
    from random import random, sample
    from string import ascii_letters
    from typing import List
    from zoneinfo import ZoneInfo
    
    from airflow.sdk import dag, task, task_group
    
    
    @task.short_circuit()
    def no() -> bool:
        return False
    
    
    @task.short_circuit()
    def fifty_fifty() -> bool:
        return random() > 0.5
    
    
    @task.python()
    def printstr(value: str) -> None:
        print(value)
    
    
    @task.python()
    def get_rand_strs(s: int, k: int) -> List[str]:
        return ["".join(sample(ascii_letters, k)) for _ in range(s)]
    
    
    @dag(
        "playground",
        description="Try random things",
        schedule=None,
        start_date=datetime(2024, 11, 1, 0, 0, 0, 0, ZoneInfo("Europe/Berlin")),
        catchup=False,
        dag_display_name="Playground",
    )
    def playground():
        @task_group()
        def group_1(somestr: str):
            (
                fifty_fifty()
                >> printstr(somestr)  # Should be skipped about half the time, but isn't
            )
    
        @task_group()
        def group_2(somestr: str):
            printstr(f"{somestr} again")  # Should also be skipped about half the time, isn't either
    
        strings = get_rand_strs(20, 10)
        group_1.expand(somestr=strings) >> group_2.expand(somestr=strings)
        no() >> printstr("Should be skipped and is.")
    
    
    playground()
13. dv-matthew-crown commented on Sep 30, 2026

    

    dv-matthew-crown

    on Sep 30, 2026

    Author

    More actions

    yep, i have been seeing this consistently since i reported and your minimal example shows the same state - @vatsrahul1001 can we reopen ?
14. kaxil

    reopened this on Sep 30, 2026
15. kaxil commented on Sep 30, 2026

    

    kaxil

    on Sep 30, 2026

    Member

    More actions

    Reopening: both reports reproduce on main, and they are two separate bugs.

    1. Unmapped short-circuit with a mapped task further downstream (the original report). The skip request only matched the map_index=-1 row of a mapped task. A task mapped over a literal list already has task instances 0..n-1 when the run is created, so none of them were skipped, and none_failed let them run. This has been the case since 3.0. Fix: Fix short-circuit not skipping already expanded mapped tasks #73960.
    2. Short-circuit inside a mapped task group (the example above). Since fix: LatestOnlyOperator not working if just before a Dynamic Task M... #62287 (3.2.0), NotPreviouslySkippedDep reads the skip decision at map_index=-1 unless the parent is a mapped operator. A task inside a mapped task group writes that decision per map index, so it was never found. Fix: Fix short-circuit inside a mapped task group not skipping downstream #73959.

    In the playground example, group_2.printstr will be skipped for every map index once any group_1.printstr is skipped, not about half the time. group_2 is a separate mapped task group, so each of its instances depends on all group_1.printstr instances, and the default all_success rule skips it.
16. kaxil

    closed this as completedin #73960on Sep 30, 2026