First, I found this bug with a very simple Dockerfile:
FROM quay.io/astronomer/ap-airflow-dev:main-onbuild
Launched via astro dev start
But I wanted to be sure that it still existed with the very latest code, so I replicated it a second time, this time with the dockerfile shown in this gist. I saw the same behavior both times.
The commit I installed from was cff7d9194f549d801947f47dfce4b5d6870bfaaa
To replicate:
- Unpause
simple_dataset_resetandsimple_dataset_sink - Run
simple_dataset_reset - (quickly) unpase all 16 dags called
source_* - wait for two minutes
For reasons I don't totally understand, source_0 fails with this error:
File "/usr/local/airflow/dags/ready_set_go.py", line 61, in increment_source
cursor.execute(update_template.format(column="source"))
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "pg_type_typname_nsp_index"
DETAIL: Key (typname, typnamespace)=(counter, 2200) already exists.
Maybe that's a problem with my code in the DAG, I don't know. But the rest of the failures indicate an airflow problem (see somefail.png).
The number of failures remains more or less stable each time I do it (four or five) but the same dags don't always fail. When they do fail, they show this in their task logs:
[2022-07-21, 06:04:17 UTC] {logging_mixin.py:115} INFO - [(4, 0)]
[2022-07-21, 06:06:00 UTC] {python.py:173} INFO - Done. Returned value was: None
[2022-07-21, 06:06:00 UTC] {taskinstance.py:1388} INFO - Marking task as SUCCESS. dag_id=source_12, task_id=increment_source, execution_date=19700101T000000, start_date=20220721T060417, end_date=20220721T060600
[2022-07-21, 06:06:00 UTC] {standard_task_runner.py:97} ERROR - Failed to execute job 8 for task increment_source ((psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "dataset_dag_run_queue_pkey"
DETAIL: Key (dataset_id, target_dag_id)=(1, simple_dataset_sink) already exists.
[SQL: INSERT INTO dataset_dag_run_queue (dataset_id, target_dag_id, created_at) VALUES (%(dataset_id)s, %(target_dag_id)s, %(created_at)s)]
[parameters: {'dataset_id': 1, 'target_dag_id': 'simple_dataset_sink', 'created_at': datetime.datetime(2022, 7, 21, 6, 6, 0, 131730, tzinfo=Timezone('UTC'))}]
(Background on this error at: https://sqlalche.me/e/14/gkpj); 375)
[2022-07-21, 06:06:00 UTC] {local_task_job.py:163} INFO - Task exited with return code 1
