Skip to content

Airflow hangs on "Found x duplicates in table task_fail. Will attempt to move them" #23472

@ghost

Description

Apache Airflow version

2.3.0 (latest released)

What happened

After upgrade from 2.2.2 to 2.3.0, when I am trying to do airflow db upgrade all I get is following log and then airflow hangs:
Found 16 duplicates in table task_fail. Will attempt to move them.

What you think should happen instead

Airflow should not hang at this log and properly move these duplicates

How to reproduce

  • Run Airflow 2.2.2
  • Create duplicates in table task_fail
  • Update Airflow to 2.3.0 and try upgrading DB

Operating System

Debian 10

Versions of Apache Airflow Providers

apache-airflow-providers-apache-cassandra==2.0.1 \
apache-airflow-providers-apache-hive==2.0.2 \
apache-airflow-providers-celery==2.1.0 \
apache-airflow-providers-cncf-kubernetes==2.0.2 \
apache-airflow-providers-ftp==2.0.1 \
apache-airflow-providers-http==2.0.1 \
apache-airflow-providers-imap==2.0.1 \
apache-airflow-providers-jdbc==2.0.1 \
apache-airflow-providers-mysql==2.1.1 \
apache-airflow-providers-papermill==2.2.3 \
apache-airflow-providers-postgres==2.2.0 \
apache-airflow-providers-sftp==2.1.1 \
apache-airflow-providers-sqlite==2.0.1 \
apache-airflow-providers-ssh==2.1.1 \
apache-airflow-providers-google==5.1.0 \
apache-airflow-providers-apache-beam==3.1.0 \
apache-airflow-providers-apache-spark==2.0.1 \

Deployment

Other Docker-based deployment

Deployment details

Google GKE
2 Instances of Airflow processes running(HA configuration)

Anything else

Problem occurs every time I am trying to update database.

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions