Skip to content

Instantly share code, notes, and snippets.

@MatrixManAtYrService
Last active July 21, 2022 06:20
Show Gist options
  • Select an option

  • Save MatrixManAtYrService/b5e58be0949eab9180608d0760288d4d to your computer and use it in GitHub Desktop.

Select an option

Save MatrixManAtYrService/b5e58be0949eab9180608d0760288d4d to your computer and use it in GitHub Desktop.
Too many tasks ending at once and writing to the same dataset

First, I found this bug with a very simple Dockerfile:

FROM quay.io/astronomer/ap-airflow-dev:main-onbuild

Launched via astro dev start

But I wanted to be sure that it still existed with the very latest code, so I replicated it a second time, this time with the dockerfile shown in this gist. I saw the same behavior both times.

The commit I installed from was cff7d9194f549d801947f47dfce4b5d6870bfaaa

To replicate:

  1. Unpause simple_dataset_reset and simple_dataset_sink
  2. Run simple_dataset_reset
  3. (quickly) unpase all 16 dags called source_*
  4. wait for two minutes

For reasons I don't totally understand, source_0 fails with this error:

  File "/usr/local/airflow/dags/ready_set_go.py", line 61, in increment_source
    cursor.execute(update_template.format(column="source"))
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "pg_type_typname_nsp_index"
DETAIL:  Key (typname, typnamespace)=(counter, 2200) already exists.

Maybe that's a problem with my code in the DAG, I don't know. But the rest of the failures indicate an airflow problem (see somefail.png).

The number of failures remains more or less stable each time I do it (four or five) but the same dags don't always fail. When they do fail, they show this in their task logs:

[2022-07-21, 06:04:17 UTC] {logging_mixin.py:115} INFO - [(4, 0)]
[2022-07-21, 06:06:00 UTC] {python.py:173} INFO - Done. Returned value was: None
[2022-07-21, 06:06:00 UTC] {taskinstance.py:1388} INFO - Marking task as SUCCESS. dag_id=source_12, task_id=increment_source, execution_date=19700101T000000, start_date=20220721T060417, end_date=20220721T060600
[2022-07-21, 06:06:00 UTC] {standard_task_runner.py:97} ERROR - Failed to execute job 8 for task increment_source ((psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "dataset_dag_run_queue_pkey"
DETAIL:  Key (dataset_id, target_dag_id)=(1, simple_dataset_sink) already exists.

[SQL: INSERT INTO dataset_dag_run_queue (dataset_id, target_dag_id, created_at) VALUES (%(dataset_id)s, %(target_dag_id)s, %(created_at)s)]
[parameters: {'dataset_id': 1, 'target_dag_id': 'simple_dataset_sink', 'created_at': datetime.datetime(2022, 7, 21, 6, 6, 0, 131730, tzinfo=Timezone('UTC'))}]
(Background on this error at: https://sqlalche.me/e/14/gkpj); 375)
[2022-07-21, 06:06:00 UTC] {local_task_job.py:163} INFO - Task exited with return code 1
FROM quay.io/astronomer/ap-airflow-dev:main-onbuild
USER root
RUN apt update && apt install -y git npm
RUN git clone https://github.com/apache/airflow /tmp/airflowrepo
RUN chown -R astro /tmp/airflowrepo
RUN chown -R astro /usr/local
USER astro
ENV AIRFLOW_REF="main"
# prep nvm
RUN curl https://raw.githubusercontent.com/creationix/nvm/master/install.sh | bash
RUN bash -c '. "$HOME/.nvm/nvm.sh" ; set -e ; nvm install node ; npm install -g yarn'
# prep repo
RUN cd /tmp/airflowrepo && git checkout ${AIRFLOW_REF}
# install newer version
RUN bash -c '. "$HOME/.nvm/nvm.sh" ;\
set -e ;\
cd /tmp/airflowrepo ;\
pip install --upgrade pip gitpython ;\
python setup.py compile_assets bdist_wheel ;\
'
RUN rm /usr/local/share/astronomer-pip-constraints.txt && touch /usr/local/share/astronomer-pip-constraints.txt
RUN cd /tmp/airflowrepo && pip install dist/$(ls dist)
from airflow import Dataset
from datetime import datetime, timedelta
from airflow.decorators import task, dag
from airflow import settings
import pause
import psycopg2
import textwrap
# when simple_dataset_source updates this dataset
# simple_dataset_sink will run
counter = Dataset("counter")
# we're using this table as a dataset
# airflow doesn't know about it, it just looks at the above object
update_template = textwrap.dedent(
"""
BEGIN;
-- make sure the table exists
CREATE TABLE IF NOT EXISTS counter (
source INT NOT NULL,
sink INT NOT NULL
);
-- seed it if it's newly created
INSERT INTO counter (source, sink)
SELECT 0, 0
WHERE NOT EXISTS (SELECT * FROM counter);
-- increment based on coaller
UPDATE counter
SET {column} = {column} + 1;
COMMIT;
"""
)
def get_cursor():
"get a cursor on the airflow database"
url = settings.engine.url
host = url.host or "localhost"
port = str(url.port or "5432")
user = url.username or "postgres"
password = url.password or "postgres"
schema = url.database
conn = psycopg2.connect(
host=host, user=user, port=port, password=password, dbname=schema
)
conn.autocommit = True
cursor = conn.cursor()
return cursor
@task(outlets=[counter])
def increment_source():
cursor = get_cursor()
cursor.execute(update_template.format(column="source"))
cursor.execute("SELECT * FROM counter;")
print(cursor.fetchall())
# if many of these are happening at once, have them all complete at the same moment
# to maximize the likelihood of a collision
now = datetime.now()
pause_until_minutes = now.minute + 2
pause.until(
datetime(now.year, now.month, now.day, now.hour, pause_until_minutes, 0)
)
@task
def increment_sink():
cursor = get_cursor()
cursor.execute(update_template.format(column="sink"))
cursor.execute("SELECT * FROM counter;")
print(cursor.fetchall())
@task
def reset():
cursor = get_cursor()
cursor.execute("DROP TABLE IF EXISTS counter;")
# run this before the test
@dag(start_date=datetime(1970, 1, 1), schedule_interval=None)
def simple_dataset_reset():
reset()
# unpause this
def source_factory(i):
@dag(
dag_id=f"source_{i}",
start_date=datetime(1970, 1, 1),
schedule_interval=timedelta(days=365 * 30),
)
def source():
increment_source()
return source()
# expect this to run
@dag(schedule_on=[counter], start_date=datetime(1970, 1, 1))
def simple_dataset_sink():
increment_sink()
# task logs should say 1,1
# source ran once
# sink ran once
# the reset dag
reset_dag = simple_dataset_reset()
# 16 source dags
for i in range(16):
globals()[str(i)] = source_factory(i)
# one sink dag
sink = simple_dataset_sink()
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment