Skip to content

Comments

AIP-72: Supporting Pulling multiple XCOM values#45509

Merged
kaxil merged 2 commits intoapache:mainfrom
astronomer:AIP72-multi-xcompull
Jan 10, 2025
Merged

AIP-72: Supporting Pulling multiple XCOM values#45509
kaxil merged 2 commits intoapache:mainfrom
astronomer:AIP72-multi-xcompull

Conversation

@amoghrajesh
Copy link
Contributor

closes: #45243

We can just handle xcoms for multiple task ids in the client directly by reusing the same task runner machinery.

DAG:

from airflow import DAG
from airflow.providers.standard.operators.python import PythonOperator

def push_to_xcom(**kwargs):
    value = "Hello, XCom!"
    return value

def push_to_xcom2(**kwargs):
    value = "Hello, XCom2!"
    return value

def pull_from_xcom(**kwargs):
    ti = kwargs['ti']
    xcom_value = ti.xcom_pull(task_ids=["push_xcom_task", "push_xcom_task2"])
    print(f"Retrieved XCom Value: {xcom_value}")


with DAG(
    'xcom_example',
    schedule=None,
    catchup=False,
) as dag:

    push_xcom_task = PythonOperator(
        task_id='push_xcom_task',
        python_callable=push_to_xcom,
    )

    push_xcom_task2 = PythonOperator(
        task_id='push_xcom_task2',
        python_callable=push_to_xcom2,
    )

    pull_xcom_task = PythonOperator(
        task_id='pull_xcom_task',
        python_callable=pull_from_xcom,
    )

    push_xcom_task >> push_xcom_task2 >> pull_xcom_task

Task1:
image

Task2:
image

Task3: (gets xcoms pushed by 1 and 2)
image


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@amoghrajesh
Copy link
Contributor Author

Not too sure where to add the tests for this.

Copy link
Member

@ashb ashb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How/where to test: add some tests that create a RunteimTaskInstance method and call the functions on it.

kaxil added a commit to astronomer/airflow that referenced this pull request Jan 9, 2025
This test was missed and will help apache#45509
kaxil added a commit that referenced this pull request Jan 9, 2025
This test was missed and will help #45509
amoghrajesh and others added 2 commits January 10, 2025 01:09
Apply suggestions from code review

cleanup and adding tests

Co-Authored-By: Kaxil Naik <[email protected]>
@kaxil kaxil force-pushed the AIP72-multi-xcompull branch from b589847 to e958a5d Compare January 9, 2025 19:45
Copy link
Member

@kaxil kaxil left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving, but you should take a look too :D since I added the test -- to verify that you are happy with it.

@kaxil kaxil merged commit e8f3be8 into apache:main Jan 10, 2025
44 checks passed
@kaxil kaxil deleted the AIP72-multi-xcompull branch January 10, 2025 12:57
@kaxil
Copy link
Member

kaxil commented Jan 10, 2025

@amoghrajesh I merged this to tick off items :) -- but feel free to change anything in follow-up

karenbraganz pushed a commit to karenbraganz/airflow that referenced this pull request Jan 13, 2025
karenbraganz pushed a commit to karenbraganz/airflow that referenced this pull request Jan 13, 2025
HariGS-DB pushed a commit to HariGS-DB/airflow that referenced this pull request Jan 16, 2025
HariGS-DB pushed a commit to HariGS-DB/airflow that referenced this pull request Jan 16, 2025
Copy link
Contributor Author

@amoghrajesh amoghrajesh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @kaxil.
Yes it looks good.

dauinh pushed a commit to dauinh/airflow that referenced this pull request Jan 24, 2025
got686-yandex pushed a commit to got686-yandex/airflow that referenced this pull request Jan 30, 2025
got686-yandex pushed a commit to got686-yandex/airflow that referenced this pull request Jan 30, 2025
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request May 28, 2025
This test was missed and will help apache/airflow#45509

GitOrigin-RevId: 5b523dac778c8eb5abad9c912f7ebaf08abdbe0b
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Sep 23, 2025
This test was missed and will help apache/airflow#45509

GitOrigin-RevId: 5b523dac778c8eb5abad9c912f7ebaf08abdbe0b
kosteev pushed a commit to GoogleCloudPlatform/composer-airflow that referenced this pull request Oct 21, 2025
This test was missed and will help apache/airflow#45509

GitOrigin-RevId: 5b523dac778c8eb5abad9c912f7ebaf08abdbe0b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support pulling multiple XCom values

3 participants