providers-fab: Airflow 3.1.3 sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back

Apache Airflow Provider(s)

fab

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==9.16.0
apache-airflow-providers-apache-spark==5.4.0
apache-airflow-providers-celery==3.13.0
apache-airflow-providers-cncf-kubernetes==10.11.0
apache-airflow-providers-common-compat==1.10.0
apache-airflow-providers-common-io==1.6.4
apache-airflow-providers-common-messaging==2.0.0
apache-airflow-providers-common-sql==1.28.2
apache-airflow-providers-docker==4.4.4
apache-airflow-providers-elasticsearch==6.3.4
apache-airflow-providers-fab==3.0.3
apache-airflow-providers-ftp==3.13.2
apache-airflow-providers-git==0.0.9
apache-airflow-providers-google==18.1.0
apache-airflow-providers-grpc==3.8.2
apache-airflow-providers-hashicorp==4.3.3
apache-airflow-providers-http==5.4.0
apache-airflow-providers-microsoft-azure==12.8.0
apache-airflow-providers-mysql==6.3.4
apache-airflow-providers-odbc==4.10.2
apache-airflow-providers-openlineage==2.7.3
apache-airflow-providers-postgres==6.4.0
apache-airflow-providers-redis==4.3.2
apache-airflow-providers-sendgrid==4.1.4
apache-airflow-providers-sftp==5.4.1
apache-airflow-providers-slack==9.4.0
apache-airflow-providers-smtp==2.3.1
apache-airflow-providers-snowflake==6.6.0
apache-airflow-providers-ssh==4.1.5
apache-airflow-providers-standard==1.10.0

Apache Airflow version

3.1.3

Operating System

Amazon Linux 2023

Deployment

Official Apache Airflow Helm Chart

Deployment details

EKS v1.33
Airflow Chart 1.18.0
Aurora Postgres 16.9
Replicas 2 of every component
Have three roles and users (admin, group1, group 2)

What happened

I'm receiving the same errors mentioned in the following issues, despite running version 3.0.3 of the FAB provider. The login works for several days before failing with the error:

2025-12-11T23:56:28.992968Z [error    ] Exception on /login/ [GET]     [airflow.providers.fab.www.app] loc=app.py:1744
Traceback (most recent call last)
...
 raise exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. (Background on this error at: https://sqlalche.me/e/14/8s2b)

Stacktrace

airflow-api-error.log

What you think should happen instead

Ideally, the sqlAlchemy error shouldn't occur.

How to reproduce

Run FAB with in-built user authentication for several days and wait for the error to appear.

Anything else

I'm wondering if the issue may be resulted to running API replicas and hitting some irrecoverable lock state of some kind.

Are you willing to submit PR?

Yes I am willing to submit a PR!

Code of Conduct

I agree to follow this project's Code of Conduct

we're also having the same issue even updating the latest versions.

I also have the same issue, can see that this error can be reproduce by action with new login into Airflow (may by any times) then they will have this issue after along times (hours). You still keep the session with old cookies.

---> I must restart my API-Server for my team for new login. I meet this issues from fab 3.0.1 to 3.0.3.

Confirming I'm seeing the same issue at my end. Steps to reproduce: stay logged in and re-access the airflow after one day. It will usually error out with this error.

Same problem here.

Same problem with:

apache-airflow 3.1.5
apache-airflow-providers-fab 3.1.0

Problem started after:

apache-airflow 3.0.6
apache-airflow-providers-fab 2.4.1

Can someone please try these fixes:

[database]
sql_alchemy_pool_pre_ping = True

[fab]
session_lifetime_minutes = 1440

I would like to see if these config changes fix the issue. If yes, I'll create a PR to change the default value of these configs

@vincbeck

@vincbeck I've made the changes as per your suggestion, I will have my team keep an eye on it and let you know.

[database]
sql_alchemy_pool_pre_ping = True

[fab]
session_lifetime_minutes = 1440

@vincbeck I've made the changes as per your suggestion, I will have my team keep an eye on it and let you know.
[database]
sql_alchemy_pool_pre_ping = True

[fab]
session_lifetime_minutes = 1440

Thank you!

Can someone please try these fixes:

I try it

Can someone please try these fixes:
[database]
sql_alchemy_pool_pre_ping = True

[fab]
session_lifetime_minutes = 1440
I would like to see if these config changes fix the issue. If yes, I'll create a PR to change the default value of these configs

I'm already on the pre-ping settings. The new thing which is shown here is the session lifetime. Doesn't fix the issue.

  # SQLAlchemy engine options for both Airflow database and Celery result backend
  # Optimized for RDS Proxy with TCP keepalive and connection recycling
  sqlalchemy_engine_options = jsonencode({
    pool_size            = 5
    pool_recycle         = var.proxy_idle_client_timeout - 100 # 1700s (100s before proxy timeout)
    pool_pre_ping        = true                                # Test connections on checkout
    pool_reset_on_return = "rollback"                          # Auto-rollback failed transactions on connection return
  })
}

@wolfdn

@wolfdn We also had some DB errors when we integrated Azure OAuth in FAB for Airflow... do you remember such error and did we need to fix anything that might be missing here?

@trina242

I'm thinking we can call this issue closed. I don't see any reported repeats of the PendingRollBackError since 3.1.2rc2 and the 3.1.2 release. Thanks for everyone's assistance with testing and @trina242 and @vincbeck for resolving this one.

@trina242

@trina242 thanks, your advice helped.
P.S. Without clearing cookies, the issue was reproduced, still with a TaskGroup exception in the stacktrace.

@anavrotski

@anavrotski we're using OAuth with Auth0, also configured in similar way as you mentioned - https://airflow.apache.org/docs/apache-airflow-providers-fab/stable/auth-manager/webserver-authentication.html#example-using-team-based-authorization-with-github-oauth
we are too seeing the TaskGroup exception as well as the PendingRollbackError exception in airflow 3.1.6 + fab 3.1.1

We have seen it happening in 3.1.6, to overcome the issue for some time, we are restarting the pods.

Happening for me as well in 3.1.6 - helm + kubernetes install.

Still happening to me as well. airflow 3.1.6 + fab 3.1.2 - helm and k8s

@zachliu

@zachliu You need to upgrade fab provider to 3.1.2 and clear your cookies in your browser.

@wolvery & @AndersFarrFoundry Can you confirm you both upgraded fab provider manually to 3.1.2? You'll need to do this either with start-up script or custom Airflow container in your helm installation or use a newer version of Airflow with the new provider baked in already.

@Dmitrij-Salenkoff Can you please try these settings below as well with FAB 3.1.2? Are you using OAuth/SAML provider? Seems there are issues with OAuth/SAML providers which I think is a separate bug.

Can someone please try these fixes:
[database]
sql_alchemy_pool_pre_ping = True

[fab]
session_lifetime_minutes = 1440
I would like to see if these config changes fix the issue. If yes, I'll create a PR to change the default value of these configs

Resolved in airflow 3.1.7 + apache-airflow-providers-fab==3.2.0
Everyone, please upgrade your version.

I’m still getting this error with Airflow 3.1.7 + fab 3.2.0 (K8s Helm installation)

Same problem here, also Airflow 3.1.7 + FAB Provider 3.2.0 (Helm Installation).

Resolved in airflow 3.1.7 + apache-airflow-providers-fab==3.2.0
Everyone, please upgrade your version.

Sorry about that. I thought the issue with the Airflow Security tab was fully resolved, but it’s actually still happening in some case.

#61480 seems to have a fix for it. Can you guys check and test it (if possible)?

I've created a new issue as soon as this is closed
#61518

Navigation Menu

Repository navigation

providers-fab: Airflow 3.1.3 sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back #59349

Apache Airflow Provider(s)

Versions of Apache Airflow Providers

Apache Airflow version

Operating System

Deployment

Deployment details

What happened

Stacktrace

What you think should happen instead

How to reproduce

Anything else

Are you willing to submit PR?

Code of Conduct

31 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

providers-fab: Airflow 3.1.3 sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back #59349

Description

Apache Airflow Provider(s)

Versions of Apache Airflow Providers

Apache Airflow version

Operating System

Deployment

Deployment details

What happened

Stacktrace

What you think should happen instead

How to reproduce

Anything else

Are you willing to submit PR?

Code of Conduct

Activity

keshan17 commented on Dec 12, 2025

ovintuan commented on Dec 12, 2025

rmalai-amdaris commented on Dec 12, 2025

ryan-dyer-sp commented on Dec 15, 2025

alex-brunshtein commented on Dec 15, 2025

vincbeck commented on Dec 15, 2025

berglh commented on Dec 15, 2025

vincbeck commented on Dec 15, 2025

alex-brunshtein commented on Dec 16, 2025

rmalai-amdaris commented on Dec 16, 2025

31 remaining items

jscheffl commented on Jan 19, 2026

berglh commented on Jan 19, 2026

anavrotski commented on Jan 20, 2026

zachliu commented on Jan 22, 2026

wolvery commented on Jan 23, 2026

AndersFarrFoundry commented on Jan 28, 2026

Dmitrij-Salenkoff commented on Feb 2, 2026

berglh commented on Feb 2, 2026

uplsh580 commented on Feb 4, 2026

rgelsi commented on Feb 5, 2026

fbmatilla commented on Feb 5, 2026

uplsh580 commented on Feb 5, 2026

vincbeck commented on Feb 5, 2026

anavrotski commented on Feb 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions