dag_run.conf is NoneType when running example_trigger_target_dag from example_dags #9852
Description
Apache Airflow version: 1.10.9 & 1.10.11
Environment:
- Cloud provider or hardware configuration: Local PC running Docker on Windows through WSL2
- OS (e.g. from /etc/os-release): Debian GNU/Linux 10 (buster)
- Kernel (e.g.
uname -a): Linux ba34d8b76277 4.19.104-microsoft-standard Improving the search functionality in the graph view #1 SMP Wed Feb 19 06:37:35 UTC 2020 x86_64 GNU/Linux - Install tools: Docker + Python 3.7
- Others: PostgreSQL + Local Executor
What happened:
I'm trying to run the example dag that triggers another dag. When i run the example, the run is unsuccessful because dag_run.conf is NoneType rather than the conf passed in the controller dag.
Moreover, the dag runs fine if run from the CLI
What you expected to happen:
The conf from the trigger dag should not be NoneType
What do you think went wrong?
Unsure - previous JIRA post suggests this has something to do with pickling. However, that post refers to SQLite being the root of the issue whereas i'm using Postgres
How to reproduce it:
Running the example dag for triggering other dags on the above configuration - I suspect this is not a common issue as I do not find a lot of resources related to this error on git, jira, stackoverflow etc
Anything else we need to know:
Exact replica of https://issues.apache.org/jira/browse/AIRFLOW-46 However, the JIRA post suggests that this only happens on SQLite whereas this is happening on Postgres for me
How often does this happen?
Problem occurs every time I run the example_dag for triggering another dag
Any relevant logs to include?
DAG Run Log
[2020-07-16 01:12:15,968] {{taskinstance.py:670}} INFO - Dependencies all met for <TaskInstance: example_trigger_target_dag.run_this 2020-07-16T01:12:12.248293+00:00 [queued]>
[2020-07-16 01:12:16,018] {{taskinstance.py:670}} INFO - Dependencies all met for <TaskInstance: example_trigger_target_dag.run_this 2020-07-16T01:12:12.248293+00:00 [queued]>
[2020-07-16 01:12:16,018] {{taskinstance.py:880}} INFO -
--------------------------------------------------------------------------------
[2020-07-16 01:12:16,018] {{taskinstance.py:881}} INFO - Starting attempt 1 of 1
[2020-07-16 01:12:16,018] {{taskinstance.py:882}} INFO -
--------------------------------------------------------------------------------
[2020-07-16 01:12:16,071] {{taskinstance.py:901}} INFO - Executing <Task(PythonOperator): run_this> on 2020-07-16T01:12:12.248293+00:00
[2020-07-16 01:12:16,073] {{standard_task_runner.py:54}} INFO - Started process 575 to run task
[2020-07-16 01:12:16,093] {{standard_task_runner.py:77}} INFO - Running: ['airflow', 'run', 'example_trigger_target_dag', 'run_this', '2020-07-16T01:12:12.248293+00:00', '--job_id', '25', '--pool', 'default_pool', '--raw', '-sd', 'DAGS_FOLDER/test_dag_2.py', '--cfg_path', '/tmp/tmp0tn7w7v2']
[2020-07-16 01:12:16,093] {{standard_task_runner.py:78}} INFO - Job 25: Subtask run_this
[2020-07-16 01:12:16,167] {{logging_mixin.py:112}} INFO - Running %s on host %s <TaskInstance: example_trigger_target_dag.run_this 2020-07-16T01:12:12.248293+00:00 [running]> 4b821a8e9fcc
[2020-07-16 01:12:16,186] {{logging_mixin.py:112}} INFO - <class 'airflow.models.dagrun.DagRun'>
[2020-07-16 01:12:16,186] {{taskinstance.py:1150}} ERROR - 'NoneType' object is not subscriptable
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 984, in _run_raw_task
result = task_copy.execute(context=context)
File "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 113, in execute
return_value = self.execute_callable()
File "/usr/local/lib/python3.7/site-packages/airflow/operators/python_operator.py", line 118, in execute_callable
return self.python_callable(*self.op_args, **self.op_kwargs)
File "/usr/local/airflow/dags/test_dag_2.py", line 44, in run_this_func
print("Remotely received value of {} for key=message".format(context["dag_run"].conf["message"]))
TypeError: 'NoneType' object is not subscriptable
[2020-07-16 01:12:16,191] {{taskinstance.py:1194}} INFO - Marking task as FAILED. dag_id=example_trigger_target_dag, task_id=run_this, execution_date=20200716T011212, start_date=20200716T011215, end_date=20200716T011216
[2020-07-16 01:12:20,926] {{local_task_job.py:102}} INFO - Task exited with return code 1