Skip to content

Conversation

@thesuperzapper
Copy link
Member

@thesuperzapper thesuperzapper commented Feb 23, 2022

What issues does your PR fix?

Currently, the db-migrations Deployment will hang forever after printing WARNING - there are unapplied db migrations, triggering apply... when attempting to upgrade airflow from 2.2.X to something newer.

When this happens, the only solution is to restart the PgBouncer Pod.

What does your PR do?

As of Airflow 2.2.0, the upgradedb() command now creates a PG_ADVISORY_LOCK.

We run PgBouncer in pool_mode = transaction, in this mode, locks will never be released, because the "lock" and "unlock" statements are necessarily in separate transactions, and there is no guarantee that these transactions will be run on the same session by PgBouncer.

To solve this, this PR uses the server_reset_query and server_reset_query_always configs of PgBouncer, to run a SELECT pg_advisory_unlock_all() query after each transaction. This ensures any locks that were created are immediately removed. In our case, this is fine, because only one instance of db-migrations will run at a time (in most situations), so the locks arent strictly nesseesary.

Checklist

For all Pull Requests

For releasing ONLY

@thesuperzapper thesuperzapper added this to the airflow-8.6.0 milestone Feb 23, 2022
@thesuperzapper thesuperzapper added the status/ready-to-merge status - this will be merged into next release label Mar 22, 2022
@thesuperzapper thesuperzapper changed the title fix: PG_ADVISORY_LOCK are not released pgbouncer fix: PG_ADVISORY_LOCK are not released in pgbouncer Apr 1, 2022
@thesuperzapper thesuperzapper merged commit 85928b3 into airflow-helm:main Apr 1, 2022
@thesuperzapper thesuperzapper deleted the pgbouncer-locks-fix branch April 1, 2022 08:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

status/ready-to-merge status - this will be merged into next release

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

1 participant