Repository navigation
Getting Duplicate entry issue in dag_tag #50815
Description
Apache Airflow version
Other Airflow 2 version (please specify below)
If "Other Airflow 2 version" selected, which one?
2.5.0
What happened?
When updating the tags for a dag for upper-case to lower-case the scheduler give this info and the Dag does not appear on the UI.
sqlalchemy.exc.PendingRollbackError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (MySQLdb._exceptions.IntegrityError) (1062, "Duplicate entry 'pacman-Campaign-711_PRD-1671' for key 'dag_tag.PRIMARY'") [parameters: (('pacman', 'Campaign-711_PRD-1671'), ('pu', 'Campaign-711_PRD-1671'))]
What you think should happen instead?
There should be no such error if the Dag tag is updated
How to reproduce
Im using MYSQL Server version: 8.0.28 Source distribution
CREATE TABLE `dag_tag` ( `name` varchar(100) NOT NULL, `dag_id` varchar(250) CHARACTER SET utf8 COLLATE utf8_bin NOT NULL, PRIMARY KEY (`name`,`dag_id`), KEY `dag_tag_dag_id_fkey` (`dag_id`), CONSTRAINT `dag_tag_dag_id_fkey` FOREIGN KEY (`dag_id`) REFERENCES `dag` (`dag_id`) ON DELETE CASCADE ON UPDATE RESTRICT ) ENGINE=InnoDB DEFAULT CHARSET=latin1
Update the tags of a dag from upper-case to lower-case.
Operating System
centos 7
Versions of Apache Airflow Providers
No response
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
boring-cyborg commentedon May 20, 2025
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
rawwar commentedon May 20, 2025
@AshutoshSrivastavaDev , were you trying to directly update this on DB via a query?
AshutoshSrivastavaDev commentedon May 21, 2025
No I just updated the tags in the dag file.
AshutoshSrivastavaDev commentedon May 21, 2025
@rawwar
Would it be appropriate to add collation="utf8_bin" explicitly to the name column in the DagTag model to make both parts of the primary key behave consistently?
name = Column(String(TAG_MAX_LEN, collation="utf8_bin"), primary_key=True)Prevent silent primary key conflicts due to casing changes
Make the behavior more predictable
Only affect MySQL/MariaDB (PostgreSQL ignores collation at the column level)
Let me know if this sounds like a safe improvement or if there are edge cases to consider.
rawwar commentedon May 21, 2025
@AshutoshSrivastavaDev , can you try this on the latest version of Airflow 2.x? Also, can you share the DAG code, before and after?
I, too, need to look into this to answer your questions.
AshutoshSrivastavaDev commentedon May 21, 2025
we create the dags dynamically using this code
earlier the input for tags was 'PACMAN' now it is 'pacman'
AshutoshSrivastavaDev commentedon May 21, 2025
we create the dags dynamically using this code
def create_dag(self, json): dag_id = json['dag_args']['dag_id'] timetable = CronTriggerTimetable(json['dag_args']['timetable'], timezone="Asia/Kolkata") tags = json['dag_args']['tags'] retries = json['dag_args']['retries'] max_active_runs = json['dag_args']['concurrency'] max_active_tasks = MAX_ACTIVE_TASKS default_args = { 'depends_on_past': False, 'email': ['[email protected]'], 'email_on_failure': False, 'email_on_retry': False, 'retries': retries, 'retry_exponential_backoff': True, 'retry_delay': timedelta(seconds=10), 'max_retry_delay': timedelta(minutes=2), } start_date = get_date(json['dag_args'].get('start_date')) end_date = get_date(json['dag_args'].get('end_date')) self.dag = DAG(dag_id=dag_id, start_date=start_date, end_date=end_date, default_args=default_args, timetable=timetable, max_active_runs=max_active_runs, max_active_tasks=max_active_tasks, tags=tags, catchup=False, on_failure_callback=delete_slack_thread_id_and_stop_cluster, on_success_callback=delete_slack_thread_id_and_stop_cluster) return self.dagearlier the input for tags was 'PACMAN' now it is 'pacman'
AshutoshSrivastavaDev commentedon May 22, 2025
@rawwar im thinking of doing
ALTER TABLE dag_tag MODIFY name VARCHAR(100) CHARACTER SET utf8 COLLATE utf8_bin;for now, do you foresee any issues that might arise from this?
github-actions commentedon Aug 16, 2026
This issue has been automatically marked as stale because it has been open for 365 days without any activity. There has been several Airflow releases since last activity on this issue. Kindly asking to recheck the report against latest Airflow version and let us know if the issue is reproducible. The issue will be closed in next 30 days if no further activity occurs from the issue author.
github-actions commentedon Sep 16, 2026
This issue has been closed because it has not received response from the issue author.