Repository navigation
Intermittent "JWT token is not valid: Signature has expired" and PendingRollbackError after upgrade to FAB 5 #57065
Description
Apache Airflow version
3.1.0
If "Other Airflow 2/3 version" selected, which one?
No response
What happened?
After upgrading to apache-airflow-providers-fab==3.0.0, which uses Flask AppBuilder 5, we started observing intermittent authentication-related crashes that make the Airflow Api Server unavailable.
The issue appears to be triggered by an expired or invalid JWT token, after which the Flask AppBuilder authentication manager fails to recover gracefully and raises a PendingRollbackError in SQLAlchemy.
2025-10-22T07:25:37.653971Z [error ] JWT token is not valid: Signature has expired [airflow.api_fastapi.auth.managers.base_auth_manager] loc=base_auth_manager.py:107
INFO: 10.67.70.212:57914 - "GET /ui/config HTTP/1.1" 401 Unauthorized
2025-10-22T07:25:37.824145Z [error ] Exception on /login/ [GET] [airflow.providers.fab.www.app] loc=app.py:1744
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2529, in wsgi_app
response = self.full_dispatch_request()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1825, in full_dispatch_request
rv = self.handle_user_exception(e)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1821, in full_dispatch_request
rv = self.preprocess_request()
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2313, in preprocess_request
rv = self.ensure_sync(before_func)()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/security_manager.py", line 57, in before_request
g.user = get_auth_manager().get_user()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 278, in get_user
if current_user.is_anonymous and getattr(g, "user", None) is not None and not g.user.is_anonymous:
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/werkzeug/local.py", line 316, in __get__
obj = instance._get_current_object()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/werkzeug/local.py", line 520, in _get_current_object
return get_name(local()) # type: ignore
^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/flask_login/utils.py", line 25, in <lambda>
current_user = LocalProxy(lambda: _get_user())
^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/flask_login/utils.py", line 370, in _get_user
current_app.login_manager._load_user()
File "/home/airflow/.local/lib/python3.12/site-packages/flask_login/login_manager.py", line 364, in _load_user
user = self._user_callback(user_id)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1384, in load_user
user = self.get_user_by_id(int(pk))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1390, in get_user_by_id
return self.session.get(self.user_model, pk)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2853, in get
return self._get_impl(
^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2975, in _get_impl
return db_load_fn(
^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/loading.py", line 530, in load_on_pk_identity
session.execute(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1717, in execute
result = conn._execute_20(statement, params or {}, execution_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
return connection._execute_clauseelement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1808, in _execute_context
conn = self._revalidate_connection()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 650, in _revalidate_connection
self._invalid_transaction()
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 622, in _invalid_transaction
raise exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. (Background on this error at: https://sqlalche.me/e/14/8s2b)
After this occurs, the Api Server becomes unresponsive until it’s restarted.
What you think should happen instead?
The system should gracefully handle expired JWT tokens - e.g., invalidate the session, redirect the user to /login, and not leave SQLAlchemy sessions in a broken state.
How to reproduce
Unfortunately, this issue is intermittent - it’s not clear how to consistently reproduce it.
It tends to occur when:
- The user session has expired and the UI tries to refresh(/ui/config or /login/),
- Or when multiple requests hit the webserver simultaneously after an idle period.
Operating System
Debian
Versions of Apache Airflow Providers
apache-airflow==3.1.0
apache-airflow-providers-google==18.0.0
apache-airflow-providers-mysql==6.3.4
apache-airflow-providers-slack==9.3.0
apache-airflow-providers-vertica==4.1.2
apache-airflow-providers-apache-livy==4.4.3
apache-airflow-providers-ssh==4.1.4
apache-airflow-providers-common-sql==1.28.1
apache-airflow-providers-cncf-kubernetes==10.8.2
apache-airflow-providers-hashicorp==4.3.2
apache-airflow-providers-standard==1.9.0
apache-airflow-providers-postgres==6.3.0
apache-airflow-providers-apache-cassandra==3.8.2
apache-airflow-providers-fab==3.0.0
apache-airflow-providers-amazon==9.15.0
Deployment
Official Apache Airflow Helm Chart
Deployment details
airflow 3.1.0 in k8s deployed with official chart. We're also use pgbouncer for connection pooling
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Activity
ido177 commentedon Oct 22, 2025
Looks similar to #39935
eladkal commentedon Oct 22, 2025
cc @vincbeck
vincbeck commentedon Oct 22, 2025
Looking into it but I am unable to reproduce it. If some folks encounter the same issue and have "better" ways to reproduce it, I take it :)
ido177 commentedon Oct 23, 2025
Hmm, how normal is it for a bad query to trigger a rollback, which blocks database access for the entire instance, causing a crash? It seems this issue highlights another problem. Perhaps we should somehow isolate connection pools from each other so that a problem at the authentication stage doesn't crash the entire production instance?
Unfortunately, I don't know how to reproduce this issue, which is why I indicated it's intermittent.
msenmurugan commentedon Oct 24, 2025
@vincbeck We are also facing the same issue. We are using Airflow 3.1.1rc1 version. It's look like a connection pooling issue. The connection is dying during the execution of this query. This means the connection was already in a compromised state, or something external caused the connection to be severed right when Airflow tried to use it. The error occurs at cursor.execute(statement, parameters), which is the very act of sending the query to the database over an existing connection.
NOTE: Whenever we restart the API server service, it works. However, after 1 or 2 hour, again getting the same issue. This issue occurs for users that are logging in. If a user already has a opened session, it is working fine. I'm able to reproduce this issue by logging in to the Airflow via incognito mode.
vincbeck commentedon Oct 24, 2025
Pinging @potiuk here. It looks related to the errors we encountered when we migrated Fab provider to
fab5. I do not know if you have any idea. I'll try to reproduce it on my side but in case you had some ideas.potiuk commentedon Oct 25, 2025
@msenmurugan -> can you maybe take a look at the server side and see what error you could correlate with the error on the client side ? The issue in this case is that we have no idea what error it is, apparently postgres process got shut-down for some reason. It could be caused for example by too many connections opened and not released and postgress process crashing due to oom (that would be most likely hypothesis).
What you will also see is a lot of connections opened from the api-server to the database. Also it would be great to know what setup you have - whether you have pgbouncer or not - because the error might be also in the pgbouncer if you happen to have it.
The issue might also be simply that for example you have very low number of connection pool configured in pgbouncer. Generally - we would need more information about your server side and logs from the server.
potiuk commentedon Oct 25, 2025
Would love if someone could provide those logs and server-side perspective.
vincbeck commentedon Oct 30, 2025
Other similar issue: #57534
git commit -m "Fixed provider FAB - duplicated SQLAlchemy sessions ca…
Fixed provider FAB - duplicated SQLAlchemy sessions caused transactio…
Fixed provider FAB - duplicated SQLAlchemy sessions caused transactio…
Implement FabAuthManager.refresh_user for sliding JWT refresh