Repository navigation
sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back #57470
Description
Apache Airflow version
3.1.1
If "Other Airflow 2/3 version" selected, which one?
No response
What happened?
After upgrading from Airflow 3.1.0 to 3.1.1, the following error started occurring on the API server:
sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back.
In addition, when accessing the UI, an Internal Server Error appears in the browser, and in some cases, the Oops page is displayed instead.
This issue occurs very frequently, though there are occasional cases where it does not occur.
What you think should happen instead?
No response
How to reproduce
After upgrading from Airflow 3.1.0 to Airflow 3.1.1, please log in.
Operating System
ubuntu
Versions of Apache Airflow Providers
No response
Deployment
Official Apache Airflow Helm Chart
Deployment details
No response
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Reacted by Krzysztof Taborski, Shumin Z, Jibril Hartri Putra, fsm, Yuri Fontella, Seafood, SeonghwanLee and Paolo Facchinetti
Reacted by Shumin Z
Activity
arimus commentedon Oct 29, 2025
We are having the same issue after an upgrade yesterday. It's preventing even logging in to our instance
vikramkoka commentedon Oct 29, 2025
This seems very strange. Can you please share any details around this?
dougforpres commentedon Oct 29, 2025
@vikramkoka - I'm working with @arimus and can explain what we're seeing.
After the upgrade last-night things started acting weird.
It all seemed to be ok until 15:00 (server time) today. That's when I returned to the app after leaving it apparently "ok" last night.
When I click log-out I will sometimes get a 500 error with a huge stacktrace in the log indicating a transaction needs to be rolled back.
When I am able to log out, and I try to log in, sometimes I'll get a generic "Server Error" page, and other times the "Ooops" page will appear.
When I am able to log on, I'll get the home page with errors on it indicating I'm not logged on, then the page will refresh and appear to be ok.
Here is a log from the start of the 3.1.1 server yesterday.
You can see it all seemed to be ok until 15:00 today (the 29th) when a page request got a mysql timeout... then, when the user (me, as it happens) tried to log on the errors about transactions needing to roll-back started.
Apologies if this blob is too big
We got frustrated and blew away the database using the
airflow db resetcommand - it deleted the db and kicked off the migrate.Every time the migration will get to this point and freeze: (we have mysql query logging enabled)
When we eventually kill the migration, the slow query spits out an error indicating that query took "however long" to complete.
dougforpres commentedon Oct 29, 2025
PS: the missing time up until 15:00 today is due to Windows deciding to reboot my machine last night, so it was offline for that period
dougforpres commentedon Oct 30, 2025
We reinstalled everything using python 3.12 and at this point it seems to be behaving much better - we haven't retried a clean install on 3.13 yet.
rebel-wansoo commentedon Oct 30, 2025
It is obvious that if you restart the api-server the problem disappears “temporarily.” And after a few hours the problem happens again.
ktaborski commentedon Oct 31, 2025
same here:
The only solution for now is to restart api-server
ShumzZzZz commentedon Nov 2, 2025
same here, and restarting api-server pod (Im deploying using Helm) only works for a few hours, then same erro again, for me, when this happens, I cannot login, cannot access any pages under the
Securitysection on the left hand side.26 remaining items
boudey commentedon Jan 14, 2026
I am confirming the issue IS reproducible with
apache-airflow-providers-fab==3.1.0Komalpawar22 commentedon Jan 15, 2026
@here It worked for me with the fab provider version 3.0.2
s1ny1998 commentedon Jan 20, 2026
I am confirming the issue IS reproducible with apache-airflow-providers-fab==3.1.1 too
s1ny1998 commentedon Jan 20, 2026
@Komalpawar22 Did you downgrade your fab provider version?
Komalpawar22 commentedon Jan 20, 2026
@s1ny1998 Yes, using
apache-airflow-providers-fab==3.0.2now, which is working for me for now.vincbeck commentedon Jan 20, 2026
It should be fixed with the latest version 3.1.2. Can someone test it?
vincbeck commentedon Jan 20, 2026
As per #59349, it is fixed
Komalpawar22 commentedon Jan 21, 2026
I will test it.
giannisyp commentedon Jan 21, 2026
Is there a way to reproduce the error instead of waiting to reappear after few days ?
anandsummer28 commentedon Feb 16, 2026
Posting this in case it helps others hit by the same error. We saw
PendingRollbackErroron login/UI refresh/logout with Airflow 3.1.1 (API server). Below: how we reproduced it and what fixed it.Environment: Airflow 3.1.1, API server on Kubernetes (same behavior should apply to non-K8s if you can set pool recycle and terminate DB connections).
How to reproduce
Shorten the DB pool recycle so connections age quickly (replace
defaultwith your namespace if needed):kubectl set env deployment/airflow-api-server \ AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RECYCLE=30 \ -n defaultWait for the rollout to finish.
Generate load and "idle in transaction" connections by calling the auth token API in a loop (replace
YOUR_AIRFLOW_URLwith your API base URL):You’ll typically see
201for most requests.Connect to the Airflow metadata database (Postgres) and list connections:
You should see some
idleand/oridle in transactionrows from the app.Terminate both idle and idle-in-transaction backends (this leaves your current psql session alone):
Trigger the error:
You should see 500 Internal Server Error and in the API server logs:
PendingRollbackErrorand stack traces inairflow.providers.fab.www.app/security_manager/get_user_by_id. The UI can stay broken until the pod is restarted.Note: Terminating only
idle in transactionwas not enough for us; terminating bothidleandidle in transactionmade the failure reproducible.Fix that worked: FAB 3.0.2 + pool settings
apache-airflow-providers-fab==3.0.2and ensure it is used at API server (and webserver, if used) startup.AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_PRE_PING=TrueandAIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RECYCLE=1800After applying this, the same reproduction steps either no longer hit the error or, when a 500 occurred, the UI recovered on the next request instead of staying stuck.
Example for the official Airflow Helm chart (override
apiServer.args/apiServer.envand same forwebserverif you use it):The session-cleanup fixes for this issue are in apache-airflow-providers-fab 3.1.2+. If problems persist on 3.0.2, try
>=3.1.2(e.g.3.1.2or3.2.0) with the same pool settings.Summary
POOL_RECYCLE=30, hammer/auth/token, then in Postgres terminate idle and idle in transaction backends, then refresh UI or logout.apache-airflow-providers-fab==3.0.2(or>=3.1.2) at startup +PRE_PING=TrueandPOOL_RECYCLE=1800.Hope this helps anyone else hitting the same error.
mwullink commentedon Mar 5, 2026
still getting this error with airflow 3.1.7 and apache-airflow-providers-fab>=3.4.0
setting the AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_PRE_PING and AIRFLOW__DATABASE__SQL_ALCHEMY_POOL_RECYCLE does not fix it for me.
sachin-dixit-wynk commentedon Mar 6, 2026
still getting this with 3.1.7.
=================================================
INFO: 127.0.0.6:52475 - "GET / HTTP/1.1" 200 OK
INFO: 127.0.0.6:52335 - "GET / HTTP/1.1" 200 OK
INFO: 127.0.0.6:48085 - "GET /dags/dp-dev-daily-end-workflow/runs/scheduled__2026-03-05T02%3A30%3A00%2B00%3A00/tasks/dp-dev-wait-timer-end-daily-workflow HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
self.dialect.do_execute(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 952, in do_execute
cursor.execute(statement, parameters)
psycopg2.OperationalError: SSL SYSCALL error: EOF detected
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in call
await super().call(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in call
await self.middleware_stack(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive, _send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in call
await responder(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in call
await super().call(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in call
await self.app(scope, receive, self.send_with_compression)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in call
await self.app(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 182, in call
with recv_stream, send_stream, collapse_excgroups():
^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/contextlib.py", line 158, in exit
self.gen.throw(value)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 184, in call
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 49, in dispatch
new_user, current_user = await self._refresh_user(current_token)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 80, in _refresh_user
user = await resolve_user_from_token(current_token)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/core_api/security.py", line 100, in resolve_user_from_token
return await get_auth_manager().get_user_from_token(token_str)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/managers/base_auth_manager.py", line 111, in get_user_from_token
return self.deserialize_user(payload)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 375, in call
return wrapper(self._obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 358, in wrapper
v = method(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 267, in deserialize_user
return self.session.scalars(select(User).where(User.id == int(token["sub"]))).one()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/scoping.py", line 1891, in scalars
return self._proxied.scalars(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2459, in scalars
return self._execute_internal(
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
result: Result[Any] = compile_state_cls.orm_execute_statement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
result = conn.execute(
^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
return meth(
^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
return connection._execute_clauseelement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
return self._exec_single_context(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
self._handle_dbapi_exception(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2363, in _handle_dbapi_exception
raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
self.dialect.do_execute(
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 952, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) SSL SYSCALL error: EOF detected
[SQL: SELECT ab_user.id, ab_user.first_name, ab_user.last_name, ab_user.username, ab_user.password, ab_user.active, ab_user.email, ab_user.last_login, ab_user.login_count, ab_user.fail_login_count, ab_user.created_on, ab_user.changed_on, ab_user.created_by_fk, ab_user.changed_by_fk
FROM ab_user
WHERE ab_user.id = %(id_1)s]
[parameters: {'id_1': 1}]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
INFO: 127.0.0.6:38111 - "GET /api/v2/dags/dp-dev-daily-end-workflow/dagRuns/scheduled__2026-03-05T02%3A30%3A00%2B00%3A00/taskInstances/dp-dev-wait-timer-end-daily-workflow/links?map_index=-1 HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 79, in collapse_excgroups
| yield
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 183, in call
| async with anyio.create_task_group() as task_group:
| ^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 783, in aexit
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in call
| await super().call(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in call
| await self.middleware_stack(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
| raise exc
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
| await self.app(scope, receive, _send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in call
| await responder(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in call
| await super().call(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in call
| await self.app(scope, receive, self.send_with_compression)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in call
| await self.app(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 182, in call
| with recv_stream, send_stream, collapse_excgroups():
| ^^^^^^^^^^^^^^^^^^^^
| File "/usr/python/lib/python3.12/contextlib.py", line 158, in exit
| self.gen.throw(value)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
| raise exc
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 184, in call
| response = await self.dispatch_func(request, call_next)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 49, in dispatch
| new_user, current_user = await self._refresh_user(current_token)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 80, in _refresh_user
| user = await resolve_user_from_token(current_token)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/core_api/security.py", line 100, in resolve_user_from_token
| return await get_auth_manager().get_user_from_token(token_str)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/managers/base_auth_manager.py", line 111, in get_user_from_token
| return self.deserialize_user(payload)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 375, in call
| return wrapper(self._obj, *args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 358, in wrapper
| v = method(self, *args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 267, in deserialize_user
| return self.session.scalars(select(User).where(User.id == int(token["sub"]))).one()
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/scoping.py", line 1891, in scalars
| return self._proxied.scalars(
| ^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2459, in scalars
| return self._execute_internal(
| ^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
| result: Result[Any] = compile_state_cls.orm_execute_statement(
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
| result = conn.execute(
| ^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
| return meth(
| ^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
| return connection._execute_clauseelement(
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
| ret = self._execute_context(
| ^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
| conn = self._revalidate_connection()
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 673, in _revalidate_connection
| self._invalid_transaction()
| File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 663, in _invalid_transaction
| raise exc.PendingRollbackError(
| sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. Please rollback() fully before proceeding (Background on this error at: https://sqlalche.me/e/20/8s2b)
+------------------------------------
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in call
await super().call(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in call
await self.middleware_stack(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive, _send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in call
await responder(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in call
await super().call(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in call
await self.app(scope, receive, self.send_with_compression)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in call
await self.app(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 182, in call
with recv_stream, send_stream, collapse_excgroups():
^^^^^^^^^^^^^^^^^^^^
File "/usr/python/lib/python3.12/contextlib.py", line 158, in exit
self.gen.throw(value)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 184, in call
response = await self.dispatch_func(request, call_next)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 49, in dispatch
new_user, current_user = await self._refresh_user(current_token)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 80, in _refresh_user
user = await resolve_user_from_token(current_token)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/core_api/security.py", line 100, in resolve_user_from_token
return await get_auth_manager().get_user_from_token(token_str)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/managers/base_auth_manager.py", line 111, in get_user_from_token
return self.deserialize_user(payload)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 375, in call
return wrapper(self._obj, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 358, in wrapper
v = method(self, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 267, in deserialize_user
return self.session.scalars(select(User).where(User.id == int(token["sub"]))).one()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/scoping.py", line 1891, in scalars
return self._proxied.scalars(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2459, in scalars
return self._execute_internal(
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
result: Result[Any] = compile_state_cls.orm_execute_statement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
result = conn.execute(
^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
return meth(
^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
return connection._execute_clauseelement(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
ret = self._execute_context(
^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
conn = self._revalidate_connection()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 673, in _revalidate_connection
self._invalid_transaction()
File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 663, in _invalid_transaction
raise exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. Please rollback() fully before proceeding (Background on this error at: https://sqlalche.me/e/20/8s2b)
INFO: 127.0.0.6:48657 - "GET /api/v2/dags/dp-dev-daily-end-workflow/dagRuns/scheduled__2026-03-05T02%3A30%3A00%2B00%3A00/taskInstances/dp-dev-wait-timer-end-daily-workflow/logs/1?map_index=-1 HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 79, in collapse_excgroups
| yield
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 183, in call
| async with anyio.create_task_group() as task_group:
| ^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 783, in aexit
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in call
| await super().call(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in call
| await self.middleware_stack(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
| raise exc
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
| await self.app(scope, receive, _send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in call
| await responder(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in call
| await super().call(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in call
| await self.app(scope, receive, self.send_with_compression)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in call
| await self.app(scope, receive, send)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 182, in call
| with recv_stream, send_stream, collapse_excgroups():
| ^^^^^^^^^^^^^^^^^^^^
| File "/usr/python/lib/python3.12/contextlib.py", line 158, in exit
| self.gen.throw(value)
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
| raise exc
| File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 184, in call
| response = await self.dispatch_func(request, call_next)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 49, in dispatch
| new_user, current_user = await self._refresh_user(current_token)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 80, in _refresh_user
| user = await resolve_user_from_token(current_token)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/core_api/security.py", line 100, in resolve_user_from_token
| return await get_auth_manager().get_user_from_token(token_str)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/managers/base_auth_manager.py", line 111, in get_user_from_token
| return self.deserialize_user(payload)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 375, in call
| return wrapper(self._obj, *args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 358, in wrapper
| v = method(self, *args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 267, in deserialize_user
| return self.session.scalars(select(User).where(User.id == int(token["sub"]))).one()
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/scoping.py", line 1891, in scalars
| return self._proxied.scalars(
| ^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2459, in scalars
| return self._execute_internal(
| ^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
| result: Result[Any] = compile_state_cls.orm_execute_statement(
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
| result = conn.execute(
| ^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
| return meth(
| ^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
| return connection._execute_clauseelement(
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
| ret = self._execute_context(
| ^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
| conn = self._revalidate_connection()
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 673, in _revalidate_connection
| self._invalid_transaction()
| File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 663, in _invalid_transaction
| raise exc.PendingRollbackError(
| sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. Please rollback() fully before proceeding (Background on this error at: https://sqlalche.me/e/20/8s2b)
+------------------------------------
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in call
await super().call(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in call
await self.middleware_stack(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in call
raise exc
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive, _send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in call
await responder(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in call
await super().call(scope, receive, send)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in call
await self.app(scope, receive, self.send_with_compression)
File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in call
await self.app(scope, receive, send)
robertlindner commentedon Apr 9, 2026
We are also facing this issue in airflow 3.1.7 with apache-airflow-providers-fab 3.2.0
I just checked the documentation, and apparently the proposed values are the defaults since quite some time:
https://airflow.apache.org/docs/apache-airflow/2.3.4/configurations-ref.html#sql-alchemy-pool-pre-ping
https://airflow.apache.org/docs/apache-airflow/2.3.4/configurations-ref.html#sql-alchemy-pool-recycle
stefankeidel commentedon Apr 28, 2026
I have to pile on here, 3.1.7 and FAB provider 3.2.0 and we just saw this this morning
sklempna commentedon May 28, 2026
I am also still experiencing this issue. Airflow is 3.1.7, apache-airflow-providers-fab is 3.2.0