10,837 questions
0
votes
0
answers
14
views
Airflow BranchPythonOperator works incorrectly when called in the following ways in the DAG
I implemented the following pipeline:
DAG Graph
As seen from the graph, "tracker" and "tracker_second" are BranchPythonOperators. I am passing them a list of tasks that need to be ...
0
votes
0
answers
11
views
pass on value returned from a function to a dag
In Airflow, is there a way where we can pass the value returned from a function to a dag?
I have been searching about it in the internet but couldnt find anything.
- 734
0
votes
0
answers
23
views
error "google.api_core.exceptions.InvalidArgument: 400 Cluster name is required" while trying to use airflow DataprocSubmitJobOperator
I'm trying to use the Dataproc submit job operator from Airflow (https://airflow.apache.org/docs/apache-airflow-providers-google/stable/_api/airflow/providers/google/cloud/operators/dataproc/index....
0
votes
0
answers
18
views
Use Jinja template for table name inside SQLExecuteQueryOperator params or parameter
I am using SQLExecuteQuery Operator to query a table, which name is dynamically generated (e.g. staging_{{ts_nodash}}).
I know I can, but I don't want to use the jinja template inside sql (example ...
- 63
0
votes
0
answers
25
views
Airflow - How to pass xcom variable into EventBridgePutEventsOperator
I am trying to pass a list of dictionaries to EventBridgePutEventsOperator but am getting Parameter validation failed: Invalid type for parameter Entries, value: [{...}], type: <class 'str'>, ...
1
vote
1
answer
35
views
How to correctly pair elements from two XCom lists in Airflow TriggerDagRunOperator.expand()?
I am using Apache Airflow and trying to trigger multiple DAGs from within another DAG using TriggerDagRunOperator.expand().
I have two lists being returned from an upstream task via XCom:
confs → A ...
- 129
0
votes
0
answers
21
views
Airflow catchup still doesn't work properly in 2.10
I have this DAG:
dag = DAG(dag_id='orch_stop',
description='set_orch_limits_to_zero',
catchup=False,
schedule=CronTriggerTimetable('45 20 * * 5', timezone='Europe/Moscow')...
0
votes
0
answers
15
views
Airflow DAG Failing in UI but Succeeding via CLI
I’m trying to make an HTTPS call from an Airflow DAG.
I've set up the connection with the appropriate Connection ID, Connection Type, and Host (URL).
When I run the following command from the terminal,...
0
votes
0
answers
19
views
Airflow (2.10.4) : How to customise run id for all run types something manual__20250227T023001, scheduled_20250227T023001
I am trying to customise the run_id for a dag, wehter it be triggered manually or scheduled one.
Tried customising TriggerDagRunOperator but getting errors also first time run_id is default pattern ...
- 391
0
votes
1
answer
12
views
How to access the response (export Manifest) from Airflow DynamoDBToS3Operator
How do I access the response of DynamoDBToS3Operator, so that I can pass the get the export location. I found out that response['ExportManifest'] contains the path to manifest_summary.json (and ...
- 1
0
votes
0
answers
31
views
How to robustly handle blips in the DAG folder
I'm running airflow 2 (with Redis and Celery) where the dags folder is set to /drive1/myuser/airflow/dags. This dags folder contains a symlink that looks like my_dags -> /drive2/myrepos/...
- 113
1
vote
1
answer
26
views
dependent tasks in different dags not triggering
I have two dags in which i want to trigger another task in a dag once the first one finishes.
Parent dag:
from airflow import DAG
from datetime import datetime, timedelta
from airflow.operators....
- 734
0
votes
0
answers
17
views
Apache Airflow @task_group error - AttributeError: 'function' object has no attribute 'update_relative'
Hello I am using Apache Airflow (2.2.3) taskflow api @task_group operator.
I keep coming into the same error when trying to use @task_group:
AttributeError: 'function' object has no attribute '...
- 111
0
votes
0
answers
26
views
Airflow - GitSync - No DAGs available
According to the official documentation, this should be enough:
dags:
mountPath: ~
persistence:
enabled: false
gitSync:
enabled: true
repo: https://github.com/organisation/my-project....
- 310
0
votes
0
answers
12
views
Plugin issue after airflow upgrade from version 1.9 to version 2.10.2
Hello,
We have upgraded our airflow from version 1.9 to version 2.10.2. there was a plugin called peb_plugin. And peb_plugin has hook, appbuilder, static, templates and all other folders.
We tried to ...
- 29