Repository navigation
Airflow 3.1.7 - Internal Server Error #61518
Description
Apache Airflow version
3.1.7
If "Other Airflow 3 version" selected, which one?
No response
What happened?
After several hours since deploy - Internal Server Error, GUI can not be opened.
wait-for-airflow-migrations
wait-for-airflow-migrations /home/airflow/.local/lib/python3.12/site-packages/airflow/providers/common/compat/security/permissions.py:30 RemovedInAirflow4Warning: The airflow.security.permissions module is deprecated; please see https://airflow.apache.org/docs/apache-airflow/stable/security/deprecated_permissions.html
wait-for-airflow-migrations 2026-02-05T16:44:38.119574Z [info [] Context impl MySQLImpl. [alembic.runtime.migration] loc=migration.py:210
wait-for-airflow-migrations 2026-02-05T16:44:38.119946Z [info [] Will assume non-transactional DDL. [alembic.runtime.migration] loc=migration.py:213
api-server | return meth(
api-server | ^^^^^
api-server | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
api-server | return connection._execute_clauseelement(
api-server | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
api-server | ret = self._execute_context(
api-server | ^^^^^^^^^^^^^^^^^^^^^^
api-server | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
api-server | conn = self._revalidate_connection()
api-server | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 673, in _revalidate_connection
api-server | self._invalid_transaction()
api-server | File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 663, in _invalid_transaction
api-server | raise exc.PendingRollbackError(
api-server | sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. Please rollback() fully before proceeding (Background on this error at: https://sqlalche.me/e/20/8s2b)
api-server +------------------------------------
api-server
api-server During handling of the above exception, another exception occurred:
api-server
api-server Traceback (most recent call last):
api-server File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 416, in run_asgi
api-server result = await app( # type: ignore[func-returns-value]
api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in __call__
api-server await super().__call__(scope, receive, send)
api-server File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
api-server await self.middleware_stack(scope, receive, send)
api-server File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
api-server raise exc
api-server File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
api-server await self.app(scope, receive, _send)
api-server File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in __call__
api-server await responder(scope, receive, send)
api-server File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in __call__
api-server await super().__call__(scope, receive, send)
api-server File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in __call__
api-server await self.app(scope, receive, self.send_with_compression)
api-server File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
api-server await self.app(scope, receive, send)
api-server File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 182, in __call__
api-server with recv_stream, send_stream, collapse_excgroups():
api-server ^^^^^^^^^^^^^^^^^^^^
api-server File "/usr/python/lib/python3.12/contextlib.py", line 158, in __exit__
api-server self.gen.throw(value)
api-server File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_utils.py", line 85, in collapse_excgroups
api-server raise exc
api-server File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/base.py", line 184, in __call__
api-server response = await self.dispatch_func(request, call_next)
api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 49, in dispatch
api-server new_user, current_user = await self._refresh_user(current_token)
api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/middlewares/refresh_token.py", line 80, in _refresh_user
api-server user = await resolve_user_from_token(current_token)
api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/core_api/security.py", line 100, in resolve_user_from_token
api-server return await get_auth_manager().get_user_from_token(token_str)
api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server File "/home/airflow/.local/lib/python3.12/site-packages/airflow/api_fastapi/auth/managers/base_auth_manager.py", line 111, in get_user_from_token
api-server return self.deserialize_user(payload)
api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 375, in __call__
api-server return wrapper(self._obj, *args, **kwargs)
api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server File "/home/airflow/.local/lib/python3.12/site-packages/cachetools/_cachedmethod.py", line 358, in wrapper
api-server v = method(self, *args, **kwargs)
api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 267, in deserialize_user
api-server return self.session.scalars(select(User).where(User.id == int(token["sub"]))).one()
api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/scoping.py", line 1891, in scalars
api-server return self._proxied.scalars(
api-server ^^^^^^^^^^^^^^^^^^^^^^
api-server File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2459, in scalars
api-server return self._execute_internal(
api-server ^^^^^^^^^^^^^^^^^^^^^^^
api-server File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
api-server result: Result[Any] = compile_state_cls.orm_execute_statement(
api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
api-server result = conn.execute(
api-server ^^^^^^^^^^^^^
api-server File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
api-server return meth(
api-server ^^^^^
api-server File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
api-server return connection._execute_clauseelement(
api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
api-server ret = self._execute_context(
api-server ^^^^^^^^^^^^^^^^^^^^^^
api-server File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
api-server conn = self._revalidate_connection()
api-server ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
api-server File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 673, in _revalidate_connection
api-server self._invalid_transaction()
api-server File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 663, in _invalid_transaction
api-server raise exc.PendingRollbackError(
api-server sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. Please rollback() fully before proceeding (Background on this error at: https://sqlalche.me/e/20/8s2b)
stream closed: EOF for airflow/airflow-api-server-f6b468f8-45gwp (wait-for-airflow-migrations)
api-server INFO: 172.25.127.162:45116 - "GET /api/v2/version HTTP/1.1" 200 OK
api-server INFO: 172.25.127.162:45132 - "GET /api/v2/version HTTP/1.1" 200 OK
What you think should happen instead?
No response
How to reproduce
Just deploy and wait ~ 16 hours.
Operating System
linux
Versions of Apache Airflow Providers
fab provider - 3.2.0
Deployment
Official Apache Airflow Helm Chart
Deployment details
Official helm chart 1.1.8
Anything else?
No response
Are you willing to submit PR?
- Yes I am willing to submit a PR!
Code of Conduct
- I agree to follow this project's Code of Conduct
Reacted by fbmatilla, pooh-duck, SeonghwanLee, Thien Chi Vi, Martin d'Allens, Krzysztof Taborski, tvt-devteam, colingrd, davidjfrickert, Moon and 2 more
Activity
abhijeets25012-tech commentedon Feb 6, 2026
i want to contribute
alex-brunshtein commentedon Feb 6, 2026
How to reproduce: don't logon and don't touch Security menu for 10 min. Try any Security or logon for error.
Works with fixed FAB provider 3.1.2. In FAB 3.2.0 error appear again.
tirkarthi commentedon Feb 8, 2026
Related #59349,#61480
abhijeets25012-tech commentedon Feb 8, 2026
Hi @anavrotski , I’m trying to reproduce/fix the PendingRollbackError in Airflow 3.1.7 + FAB 3.2.0.
It seems the problem originates in fab_auth_manager.py, but I cannot open the file in my environment. Could you advise on:
Which exact method/file to look at for the token/user session handling.
Any guidance on reproducing the issue locally or applying a workaround.
anavrotski commentedon Feb 8, 2026
Hi, I think it is better to ask code owners, not users, just reporting the issue
tvc12 commentedon Feb 9, 2026
Same the issue with Airflow 3.1.7 + FAB 3.2.0.
pranav-pw commentedon Feb 9, 2026
Earlier, we were seeing a SQLAlchemy PendingRollbackError, but the issue is now resolved.
We are using Google OAuth, and the setup works correctly with Airflow 3.1.3 deployed via the Helm chart, when apache-airflow-providers-fab==3.1.2 is used.
Based on this, I believe newer Airflow versions should also work fine. Just make sure to pin the FAB provider version to 3.1.2.
alexkuretz commentedon Feb 9, 2026
This PR appears to have fixed the issue for me: https://github.com/apache/airflow/pull/61480/changes
12 remaining items
potiuk commentedon Mar 2, 2026
@abhijeets25012-tech We are unassigning you from this issue as part of our updated assignment policy.
This is not meant to discourage your contribution — quite the opposite! You are still very welcome to work on this issue and submit a PR for it. Simply comment that you are working on it and open a PR when ready.
We found that formal assignments were not working well, as they often prevented others from contributing when the assignee was not actively working on the issue.
izhora47 commentedon Mar 7, 2026
Hi. @potiuk I see that nobody assigned to this issue, means, no fix in 3.1.8 to be expected, right? Is there a workaround at least, how to make AF working in Kubernetes and not crashing? Thanks
piosystems commentedon Mar 7, 2026
The recently released apache-airflow-providers-fab==3.4.0 seems to have resolved the issue. I have tried it and so far, so good. I presume that the next release of apache-airflow==3.1.8 will incorporate apache-airflow-providers-fab==3.4.0, so, you don't have to install it separately.
izhora47 commentedon Mar 9, 2026
3.1.8rc1 still has the same problem
uplsh580 commentedon Mar 12, 2026
@izhora47
Could you try setting up Airflow 3.1.8 and double-check the
apache-airflow-providers-fabversion as well?barunpuri commentedon Mar 13, 2026
For me, the combination of Airflow 3.1.8 and apache-airflow-providers-fab==3.4.0 appears to work without issue.
vikrams12 commentedon Mar 13, 2026
I have pulled Airflow 3.1.8 image and confirmed the version of apache-airflow-providers-fab is 3.4.0.
Hope, updating to 3.1.8 version should fix this 'PendingRollbackError'.
anavrotski commentedon Mar 14, 2026
I'm getting the next exception with Airflow 3.1.8 + fab-provider 3.4.0.
fbmatilla commentedon Mar 14, 2026
Hi @anavrotski, yours looks like a different issue. The one linked to this thread was produced by a wrong transaction status.
anavrotski commentedon Mar 14, 2026
Ok, I'm just showing that fab provider 3.4.0 still has problems, we are using 3.1.2 - it works well.
potiuk commentedon Mar 15, 2026
@anavrotski - then opening a new issue with your stack trace is a good idea. We cannot conflate different (even similar) issues into one. Feel free to open a new issue with the stack trace and circumstances you see. That will help those who want to fix things what is still needed to be fixed
potiuk commentedon Mar 15, 2026
Closing this one - but again - please open a new one with your stack trace and info
Jeoffreybauvin commentedon Mar 16, 2026
About the initial issue, I can confirm it seems to be fixed in Airflow 3.1.8.