Skip to content

Airflow 3.1.1 oauth login failure #57534

Closed

Description

@anavrotski

Apache Airflow version

3.1.1

If "Other Airflow 2/3 version" selected, which one?

No response

What happened?

History begins here.

Log in / log out works right after deploy, but after some time, it starts failing (log in is impossible).

Re-creation of the API server k8s pod helps - it starts working again.

API Server log (several attempts to log in):

INFO:     172.25.119.245:58858 - "GET /api/v2/version HTTP/1.1" 200 OK
2025-10-30T07:07:47.868081Z [error    [] Exception on /oauth-authorized/azure [GET[] [airflow.providers.fab.www.app] loc=app.py:1744
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 951, in do_execute
    cursor.execute(statement, parameters)
  File "/home/airflow/.local/lib/python3.12/site-packages/MySQLdb/cursors.py", line 179, in execute
    res = self._query(mogrified_query)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/MySQLdb/cursors.py", line 330, in _query
    db.query(q)
  File "/home/airflow/.local/lib/python3.12/site-packages/MySQLdb/connections.py", line 280, in query
    _mysql.connection.query(self, query)
MySQLdb.OperationalError: (2006, 'Server has gone away')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2529, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask_appbuilder/security/views.py", line 713, in oauth_authorized
    user = self.appbuilder.sm.auth_user_oauth(userinfo)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 2031, in auth_user_oauth
    user = self.find_user(username=username)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1426, in find_user
    return self.session.scalars(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2459, in scalars
    return self._execute_internal(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
    result: Result[Any] = compile_state_cls.orm_execute_statement(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
    result = conn.execute(
             ^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
    return meth(
           ^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 526, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
    return self._exec_single_context(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
    self._handle_dbapi_exception(
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 2355, in _handle_dbapi_exception
    raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
    self.dialect.do_execute(
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/default.py", line 951, in do_execute
    cursor.execute(statement, parameters)
  File "/home/airflow/.local/lib/python3.12/site-packages/MySQLdb/cursors.py", line 179, in execute
    res = self._query(mogrified_query)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/MySQLdb/cursors.py", line 330, in _query
    db.query(q)
  File "/home/airflow/.local/lib/python3.12/site-packages/MySQLdb/connections.py", line 280, in query
    _mysql.connection.query(self, query)
sqlalchemy.exc.OperationalError: (MySQLdb.OperationalError) (2006, 'Server has gone away')
[SQL: SELECT ab_user.id, ab_user.first_name, ab_user.last_name, ab_user.username, ab_user.password, ab_user.active, ab_user.email, ab_user.last_login, ab_user.login_count, ab_user.fail_login_count, ab_user.created_on, ab_user.changed_on, ab_user.created_by_fk, ab_user.changed_by_fk 
FROM ab_user 
WHERE lower(ab_user.username) = lower(%s)]
[parameters: ('4dac55b4-dc30-42f5-8e4f-4846ab968868',)]
(Background on this error at: https://sqlalche.me/e/20/e3q8)
INFO:     172.25.120.253:55062 - "GET /auth/oauth-authorized/azure?code=1.AQMA9KHHarFf....dacd5d2b HTTP/1.1" 500 Internal Server Error
INFO:     172.25.120.253:55062 - "GET /favicon.ico HTTP/1.1" 200 OK
INFO:     172.25.119.245:38882 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:38884 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:33248 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:33260 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:35764 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:35760 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:39466 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:39468 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:33598 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:33584 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:59828 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:59826 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:32796 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:32790 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:53710 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:53712 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:36588 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:36590 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:34038 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:34036 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:36746 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:36744 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:37786 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:37788 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:50022 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:50036 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:48980 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:48976 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.120.253:41384 - "GET / HTTP/1.1" 200 OK
INFO:     172.25.120.253:41384 - "GET /ui/config HTTP/1.1" 401 Unauthorized
INFO:     172.25.120.253:41384 - "GET /api/v2/auth/login?next=https%3A%2F%2F<our_airflow_host>%2F HTTP/1.1" 307 Temporary Redirect
INFO:     172.25.117.209:50626 - "GET /ui/auth/menus HTTP/1.1" 401 Unauthorized
INFO:     172.25.117.209:50630 - "GET /ui/config HTTP/1.1" 401 Unauthorized
INFO:     172.25.120.253:41388 - "GET /api/v2/plugins HTTP/1.1" 401 Unauthorized
INFO:     172.25.120.253:41384 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.117.209:50626 - "GET /api/v2/auth/login?next=https%3A%2F%2F<our_airflow_host>%2F HTTP/1.1" 307 Temporary Redirect
INFO:     172.25.120.253:41384 - "GET /api/v2/auth/login?next=https%3A%2F%2F<our_airflow_host>%2F HTTP/1.1" 307 Temporary Redirect
INFO:     172.25.120.253:41384 - "GET /auth/login/?next=https://<our_airflow_host> HTTP/1.1" 200 OK
INFO:     172.25.120.253:41408 - "GET /auth/static/appbuilder/css/fontawesome/solid.min.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41384 - "GET /auth/static/appbuilder/css/bootstrap.min.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41402 - "GET /auth/static/appbuilder/css/fontawesome/regular.min.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41388 - "GET /auth/static/appbuilder/css/fontawesome/fontawesome.min.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41388 - "GET /auth/static/appbuilder/css/fontawesome/brands.min.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41402 - "GET /auth/static/appbuilder/css/bootstrap-datepicker/bootstrap-datepicker3.min.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41388 - "GET /auth/static/appbuilder/css/fontawesome/v4-shims.min.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41388 - "GET /auth/static/appbuilder/css/select2/select2.min.css HTTP/1.1" 304 Not Modified
INFO:     172.25.117.209:50626 - "GET /auth/static/appbuilder/css/select2/select2-bootstrap.min.css HTTP/1.1" 304 Not Modified
INFO:     172.25.117.209:50632 - "GET /auth/static/appbuilder/css/flags/flags16.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41388 - "GET /auth/static/appbuilder/css/ab.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41402 - "GET /auth/static/dist/airflowDefaultTheme.ff5a35f322070b094aa2.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41402 - "GET /auth/static/dist/materialIcons.3e67dd6fbfcc4f3b5105.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41402 - "GET /auth/static/dist/loadingDots.2e5f555f0753107b0300.css HTTP/1.1" 304 Not Modified
INFO:     172.25.117.209:50626 - "GET /auth/static/dist/jquery-ui.min.css HTTP/1.1" 304 Not Modified
INFO:     172.25.117.209:50632 - "GET /auth/static/dist/main.3cf3be1a0c5439bb640d.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41388 - "GET /auth/static/appbuilder/js/ab_filters.js HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41402 - "GET /auth/static/dist/flash.5583a9e0cf11f2be93da.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41402 - "GET /auth/static/appbuilder/js/jquery-latest.js HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41402 - "GET /auth/static/appbuilder/js/ab_actions.js HTTP/1.1" 304 Not Modified
INFO:     172.25.117.209:50626 - "GET /auth/static/appbuilder/js/bootstrap-datepicker/bootstrap-datepicker.min.js HTTP/1.1" 304 Not Modified
INFO:     172.25.117.209:50632 - "GET /auth/static/appbuilder/js/bootstrap.min.js HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41402 - "GET /auth/static/appbuilder/js/select2/select2.min.js HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41402 - "GET /auth/static/appbuilder/js/ab.js HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41402 - "GET /auth/static/dist/jquery-ui.min.js HTTP/1.1" 304 Not Modified
INFO:     172.25.117.209:50632 - "GET /auth/static/dist/main.3cf3be1a0c5439bb640d.js HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41402 - "GET /auth/static/dist/moment.9baee5ec3d7639a10897.js HTTP/1.1" 304 Not Modified
INFO:     172.25.117.209:50632 - "GET /auth/static/appbuilder/css/webfonts/fa-brands-400.woff2 HTTP/1.1" 304 Not Modified
INFO:     172.25.117.209:50632 - "GET /auth/login/azure?next=https%3A//<our_airflow_host>/ HTTP/1.1" 302 Found
2025-10-30T07:10:14.906063Z [error    [] Exception on /oauth-authorized/azure [GET[] [airflow.providers.fab.www.app] loc=app.py:1744
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2529, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask_appbuilder/security/views.py", line 713, in oauth_authorized
    user = self.appbuilder.sm.auth_user_oauth(userinfo)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 2031, in auth_user_oauth
    user = self.find_user(username=username)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1426, in find_user
    return self.session.scalars(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2459, in scalars
    return self._execute_internal(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
    result: Result[Any] = compile_state_cls.orm_execute_statement(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
    result = conn.execute(
             ^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
    return meth(
           ^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 526, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
    conn = self._revalidate_connection()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 673, in _revalidate_connection
    self._invalid_transaction()
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 663, in _invalid_transaction
    raise exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back.  Please rollback() fully before proceeding (Background on this error at: https://sqlalche.me/e/20/8s2b)
INFO:     172.25.117.209:50632 - "GET /auth/oauth-authorized/azure?code=1.AQMA9KHH...mCp_QE&session_state=009a7329-bd24-d99e-9121-bb58dacd5d2b HTTP/1.1" 500 Internal Server Error
INFO:     172.25.117.209:50632 - "GET /favicon.ico HTTP/1.1" 200 OK
INFO:     172.25.119.245:39872 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:39870 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:59258 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:59260 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:41694 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:41680 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:48448 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:48450 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.117.209:50632 - "GET / HTTP/1.1" 200 OK
INFO:     172.25.117.209:50632 - "GET /ui/config HTTP/1.1" 401 Unauthorized
INFO:     172.25.117.209:50632 - "GET /api/v2/auth/login?next=https%3A%2F%2F<our_airflow_host>%2F HTTP/1.1" 307 Temporary Redirect
INFO:     172.25.117.209:50632 - "GET /ui/config HTTP/1.1" 401 Unauthorized
INFO:     172.25.120.253:41402 - "GET /ui/auth/menus HTTP/1.1" 401 Unauthorized
INFO:     172.25.120.253:41388 - "GET /api/v2/plugins HTTP/1.1" 401 Unauthorized
INFO:     172.25.120.253:41384 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.117.209:50632 - "GET /auth/login/?next=https://<our_airflow_host>/ HTTP/1.1" 200 OK
INFO:     172.25.120.253:41384 - "GET /api/v2/auth/login?next=https%3A%2F%2F<our_airflow_host>%2F HTTP/1.1" 307 Temporary Redirect
INFO:     172.25.120.253:41384 - "GET /api/v2/auth/login?next=https%3A%2F%2F<our_airflow_host>%2F HTTP/1.1" 307 Temporary Redirect
INFO:     172.25.120.253:41384 - "GET /auth/login/?next=https://<our_airflow_host>/ HTTP/1.1" 200 OK
INFO:     172.25.120.253:41384 - "GET /auth/static/appbuilder/css/bootstrap.min.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41402 - "GET /auth/static/appbuilder/css/fontawesome/regular.min.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41388 - "GET /auth/static/appbuilder/css/fontawesome/fontawesome.min.css HTTP/1.1" 304 Not Modified
INFO:     172.25.117.209:50632 - "GET /auth/static/appbuilder/css/fontawesome/solid.min.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41388 - "GET /auth/static/appbuilder/css/bootstrap-datepicker/bootstrap-datepicker3.min.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41402 - "GET /auth/static/appbuilder/css/fontawesome/v4-shims.min.css HTTP/1.1" 304 Not Modified
INFO:     172.25.117.209:50632 - "GET /auth/static/appbuilder/css/select2/select2.min.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41402 - "GET /auth/static/appbuilder/css/select2/select2-bootstrap.min.css HTTP/1.1" 304 Not Modified
INFO:     172.25.117.209:50632 - "GET /auth/static/appbuilder/css/ab.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41402 - "GET /auth/static/appbuilder/css/flags/flags16.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41388 - "GET /auth/static/appbuilder/css/fontawesome/brands.min.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41384 - "GET /auth/static/dist/materialIcons.3e67dd6fbfcc4f3b5105.css HTTP/1.1" 304 Not Modified
INFO:     172.25.117.209:50632 - "GET /auth/static/dist/main.3cf3be1a0c5439bb640d.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41384 - "GET /auth/static/dist/airflowDefaultTheme.ff5a35f322070b094aa2.css HTTP/1.1" 304 Not Modified
INFO:     172.25.117.209:50632 - "GET /auth/static/dist/loadingDots.2e5f555f0753107b0300.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41388 - "GET /auth/static/dist/jquery-ui.min.css HTTP/1.1" 304 Not Modified
INFO:     172.25.117.209:50626 - "GET /auth/static/appbuilder/js/bootstrap.min.js HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41388 - "GET /auth/static/dist/flash.5583a9e0cf11f2be93da.css HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41384 - "GET /auth/static/appbuilder/js/ab_actions.js HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41402 - "GET /auth/static/appbuilder/js/ab_filters.js HTTP/1.1" 304 Not Modified
INFO:     172.25.117.209:50632 - "GET /auth/static/appbuilder/js/jquery-latest.js HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41402 - "GET /auth/static/appbuilder/js/ab.js HTTP/1.1" 304 Not Modified
INFO:     172.25.117.209:50632 - "GET /auth/static/appbuilder/js/bootstrap-datepicker/bootstrap-datepicker.min.js HTTP/1.1" 304 Not Modified
INFO:     172.25.117.209:50626 - "GET /auth/static/appbuilder/js/select2/select2.min.js HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41388 - "GET /auth/static/dist/jquery-ui.min.js HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41402 - "GET /auth/static/dist/moment.9baee5ec3d7639a10897.js HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41384 - "GET /auth/static/dist/main.3cf3be1a0c5439bb640d.js HTTP/1.1" 304 Not Modified
INFO:     172.25.120.253:41388 - "GET /auth/static/appbuilder/css/webfonts/fa-brands-400.woff2 HTTP/1.1" 304 Not Modified
INFO:     172.25.119.245:56250 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:56248 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.120.253:41388 - "GET /auth/login/azure?next=https%3A//<our_airflow_host>/ HTTP/1.1" 302 Found
2025-10-30T07:11:01.580516Z [error    [] Exception on /oauth-authorized/azure [GET[] [airflow.providers.fab.www.app] loc=app.py:1744
Traceback (most recent call last):
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2529, in wsgi_app
    response = self.full_dispatch_request()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1825, in full_dispatch_request
    rv = self.handle_user_exception(e)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1823, in full_dispatch_request
    rv = self.dispatch_request()
         ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1799, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**view_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/flask_appbuilder/security/views.py", line 713, in oauth_authorized
    user = self.appbuilder.sm.auth_user_oauth(userinfo)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 2031, in auth_user_oauth
    user = self.find_user(username=username)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1426, in find_user
    return self.session.scalars(
           ^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2459, in scalars
    return self._execute_internal(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
    result: Result[Any] = compile_state_cls.orm_execute_statement(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/context.py", line 306, in orm_execute_statement
    result = conn.execute(
             ^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
    return meth(
           ^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 526, in _execute_on_connection
    return connection._execute_clauseelement(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
    ret = self._execute_context(
          ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1813, in _execute_context
    conn = self._revalidate_connection()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 673, in _revalidate_connection
    self._invalid_transaction()
  File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 663, in _invalid_transaction
    raise exc.PendingRollbackError(
sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back.  Please rollback() fully before proceeding (Background on this error at: https://sqlalche.me/e/20/8s2b)
INFO:     172.25.120.253:41388 - "GET /auth/oauth-authorized/azure?code=1.AQMA9KHHarF...E2DyWwE-IXOp4&session_state=009a7329-bd24-d99e-9121-bb58dacd5d2b HTTP/1.1" 500 Internal Server Error
INFO:     172.25.120.253:41388 - "GET /favicon.ico HTTP/1.1" 200 OK
INFO:     172.25.119.245:40900 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:40898 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:36386 - "GET /api/v2/version HTTP/1.1" 200 OK
INFO:     172.25.119.245:36388 - "GET /api/v2/version HTTP/1.1" 200 OK

What you think should happen instead?

No response

How to reproduce

Not sure when, but after several hours after the deployment of v3.1.1, it can be reproduced (but if the API server pod was restarted for some reason, the problem will not appear, need to wait).

Operating System

linux

Versions of Apache Airflow Providers

No response

Deployment

Official Apache Airflow Helm Chart

Deployment details

Chart v1.18, deployed to AWS EKS, using Azuer OAuth.

Anything else?

No response

Are you willing to submit PR?

  • Yes I am willing to submit a PR!

Code of Conduct

Activity

  1. added
    area:APIAirflow's REST/HTTP API
    priority:highHigh priority bug that should be patched quickly but does not require immediate new release
    on Oct 30, 2025
  2. vincbeck commented on Oct 30, 2025

    @vincbeck
    Contributor

    Duplicate of #57065

  3. removed
    needs-triagelabel for new issues that we didn't triage yet
    on Oct 30, 2025
  4. coenvd commented on Nov 3, 2025

    @coenvd

    We have the same issue on version 3.1.1. I thought I could work around it by refreshing with empty cache and hard reload in chrome, but this only worked once. I dont know what logs exactly to deliver in order to help you, but this is the full log that I get from the webserver when I try to log in. Please let me know what other logs I can deliver in order to help.

    INFO:     10.99.68.97:40890 - "GET /login/ HTTP/1.1" 200 OK
    INFO:     10.99.68.97:40896 - "GET /ui/config HTTP/1.1" 401 Unauthorized
    INFO:     10.99.68.175:58258 - "GET /api/v2/plugins HTTP/1.1" 401 Unauthorized
    2025-11-03T09:58:16.131127Z [error    ] Exception on /login/ [GET]     [airflow.providers.fab.www.app] loc=app.py:1744
    Traceback (most recent call last):
      File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2529, in wsgi_app
        response = self.full_dispatch_request()
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1825, in full_dispatch_request
        rv = self.handle_user_exception(e)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1821, in full_dispatch_request
        rv = self.preprocess_request()
             ^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2313, in preprocess_request
        rv = self.ensure_sync(before_func)()
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/security_manager.py", line 57, in before_request
        g.user = get_auth_manager().get_user()
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 278, in get_user
        if current_user.is_anonymous and getattr(g, "user", None) is not None and not g.user.is_anonymous:
           ^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/airflow/.local/lib/python3.12/site-packages/werkzeug/local.py", line 316, in __get__
        obj = instance._get_current_object()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/airflow/.local/lib/python3.12/site-packages/werkzeug/local.py", line 520, in _get_current_object
        return get_name(local())  # type: ignore
                        ^^^^^^^
      File "/home/airflow/.local/lib/python3.12/site-packages/flask_login/utils.py", line 25, in <lambda>
        current_user = LocalProxy(lambda: _get_user())
                                          ^^^^^^^^^^^
      File "/home/airflow/.local/lib/python3.12/site-packages/flask_login/utils.py", line 370, in _get_user
        current_app.login_manager._load_user()
      File "/home/airflow/.local/lib/python3.12/site-packages/flask_login/login_manager.py", line 364, in _load_user
        user = self._user_callback(user_id)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1384, in load_user
        user = self.get_user_by_id(int(pk))
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1390, in get_user_by_id
        return self.session.get(self.user_model, pk)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2853, in get
        return self._get_impl(
               ^^^^^^^^^^^^^^^
      File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2975, in _get_impl
        return db_load_fn(
               ^^^^^^^^^^^
      File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/loading.py", line 530, in load_on_pk_identity
        session.execute(
      File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1717, in execute
        result = conn._execute_20(statement, params or {}, execution_options)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
        return meth(self, args_10style, kwargs_10style, execution_options)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
        return connection._execute_clauseelement(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
        ret = self._execute_context(
              ^^^^^^^^^^^^^^^^^^^^^^
      File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1808, in _execute_context
        conn = self._revalidate_connection()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 650, in _revalidate_connection
        self._invalid_transaction()
      File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 622, in _invalid_transaction
        raise exc.PendingRollbackError(
    sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. (Background on this error at: https://sqlalche.me/e/14/8s2b)
    INFO:     10.99.68.97:40890 - "GET /auth/login/?next=https://jodp-airflow.internal.prod.cloud.jumbo.com/login/ HTTP/1.1" 500 Internal Server Error
    ERROR:    Exception in ASGI application
      + Exception Group Traceback (most recent call last):
      |   File "/home/airflow/.local/lib/python3.12/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
      |     result = await app(  # type: ignore[func-returns-value]
      |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in __call__
      |     await super().__call__(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
      |     await self.middleware_stack(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
      |     raise exc
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
      |     await self.app(scope, receive, _send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 29, in __call__
      |     await responder(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 130, in __call__
      |     await super().__call__(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/gzip.py", line 46, in __call__
      |     await self.app(scope, receive, self.send_with_compression)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/cors.py", line 85, in __call__
      |     await self.app(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
      |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
      |     raise exc
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
      |     await app(scope, receive, sender)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 716, in __call__
      |     await self.middleware_stack(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 736, in app
      |     await route.handle(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 462, in handle
      |     await self.app(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/fastapi/applications.py", line 1082, in __call__
      |     await super().__call__(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/applications.py", line 113, in __call__
      |     await self.middleware_stack(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 186, in __call__
      |     raise exc
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/errors.py", line 164, in __call__
      |     await self.app(scope, receive, _send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 63, in __call__
      |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
      |     raise exc
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
      |     await app(scope, receive, sender)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 716, in __call__
      |     await self.middleware_stack(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 736, in app
      |     await route.handle(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/routing.py", line 462, in handle
      |     await self.app(scope, receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/wsgi.py", line 81, in __call__
      |     await responder(receive, send)
      |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/wsgi.py", line 106, in __call__
      |     async with anyio.create_task_group() as task_group:
      |                ^^^^^^^^^^^^^^^^^^^^^^^^^
      |   File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 781, in __aexit__
      |     raise BaseExceptionGroup(
      | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
      +-+---------------- 1 ----------------
        | Traceback (most recent call last):
        |   File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2529, in wsgi_app
        |     response = self.full_dispatch_request()
        |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1825, in full_dispatch_request
        |     rv = self.handle_user_exception(e)
        |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1821, in full_dispatch_request
        |     rv = self.preprocess_request()
        |          ^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2313, in preprocess_request
        |     rv = self.ensure_sync(before_func)()
        |          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/security_manager.py", line 57, in before_request
        |     g.user = get_auth_manager().get_user()
        |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/fab_auth_manager.py", line 278, in get_user
        |     if current_user.is_anonymous and getattr(g, "user", None) is not None and not g.user.is_anonymous:
        |        ^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/werkzeug/local.py", line 316, in __get__
        |     obj = instance._get_current_object()
        |           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/werkzeug/local.py", line 520, in _get_current_object
        |     return get_name(local())  # type: ignore
        |                     ^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/flask_login/utils.py", line 25, in <lambda>
        |     current_user = LocalProxy(lambda: _get_user())
        |                                       ^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/flask_login/utils.py", line 370, in _get_user
        |     current_app.login_manager._load_user()
        |   File "/home/airflow/.local/lib/python3.12/site-packages/flask_login/login_manager.py", line 364, in _load_user
        |     user = self._user_callback(user_id)
        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1384, in load_user
        |     user = self.get_user_by_id(int(pk))
        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1390, in get_user_by_id
        |     return self.session.get(self.user_model, pk)
        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2853, in get
        |     return self._get_impl(
        |            ^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2975, in _get_impl
        |     return db_load_fn(
        |            ^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/loading.py", line 530, in load_on_pk_identity
        |     session.execute(
        |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1717, in execute
        |     result = conn._execute_20(statement, params or {}, execution_options)
        |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
        |     return meth(self, args_10style, kwargs_10style, execution_options)
        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
        |     return connection._execute_clauseelement(
        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
        |     ret = self._execute_context(
        |           ^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1808, in _execute_context
        |     conn = self._revalidate_connection()
        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 650, in _revalidate_connection
        |     self._invalid_transaction()
        |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 622, in _invalid_transaction
        |     raise exc.PendingRollbackError(
        | sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. (Background on this error at: https://sqlalche.me/e/14/8s2b)
        | 
        | During handling of the above exception, another exception occurred:
        | 
        | Traceback (most recent call last):
        |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/wsgi.py", line 109, in __call__
        |     await anyio.to_thread.run_sync(self.wsgi, environ, self.start_response)
        |   File "/home/airflow/.local/lib/python3.12/site-packages/anyio/to_thread.py", line 56, in run_sync
        |     return await get_async_backend().run_sync_in_worker_thread(
        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 2485, in run_sync_in_worker_thread
        |     return await future
        |            ^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 976, in run
        |     result = context.run(func, *args)
        |              ^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/starlette/middleware/wsgi.py", line 147, in wsgi
        |     for chunk in self.app(environ, start_response):
        |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2552, in __call__
        |     return self.wsgi_app(environ, start_response)
        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 2532, in wsgi_app
        |     response = self.handle_exception(e)
        |                ^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 1727, in handle_exception
        |     server_error = self.ensure_sync(handler)(server_error)
        |                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/www/views.py", line 82, in show_traceback
        |     return render_template("airflow/traceback.html"), 500
        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/flask/templating.py", line 147, in render_template
        |     return _render(app, template, context)
        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/flask/templating.py", line 128, in _render
        |     app.update_template_context(context)
        |   File "/home/airflow/.local/lib/python3.12/site-packages/flask/app.py", line 997, in update_template_context
        |     context.update(func())
        |                    ^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/flask_login/utils.py", line 405, in _user_context_processor
        |     return dict(current_user=_get_user())
        |                              ^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/flask_login/utils.py", line 370, in _get_user
        |     current_app.login_manager._load_user()
        |   File "/home/airflow/.local/lib/python3.12/site-packages/flask_login/login_manager.py", line 364, in _load_user
        |     user = self._user_callback(user_id)
        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1384, in load_user
        |     user = self.get_user_by_id(int(pk))
        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/fab/auth_manager/security_manager/override.py", line 1390, in get_user_by_id
        |     return self.session.get(self.user_model, pk)
        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2853, in get
        |     return self._get_impl(
        |            ^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 2975, in _get_impl
        |     return db_load_fn(
        |            ^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/loading.py", line 530, in load_on_pk_identity
        |     session.execute(
        |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 1717, in execute
        |     result = conn._execute_20(statement, params or {}, execution_options)
        |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1710, in _execute_20
        |     return meth(self, args_10style, kwargs_10style, execution_options)
        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/sql/elements.py", line 334, in _execute_on_connection
        |     return connection._execute_clauseelement(
        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1577, in _execute_clauseelement
        |     ret = self._execute_context(
        |           ^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 1808, in _execute_context
        |     conn = self._revalidate_connection()
        |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 650, in _revalidate_connection
        |     self._invalid_transaction()
        |   File "/home/airflow/.local/lib/python3.12/site-packages/sqlalchemy/engine/base.py", line 622, in _invalid_transaction
        |     raise exc.PendingRollbackError(
        | sqlalchemy.exc.PendingRollbackError: Can't reconnect until invalid transaction is rolled back. (Background on this error at: https://sqlalche.me/e/14/8s2b)
        +------------------------------------
    INFO:     10.99.72.145:54856 - "GET /api/v2/auth/login?next=https%3A%2F%2Fjodp-airflow.internal.prod.cloud.jumbo.com%2Flogin%2F HTTP/1.1" 307 Temporary Redirect
    
  5. vincbeck commented on Nov 3, 2025

    @vincbeck
    Contributor

    I'll definitely prioritize this one since it is impacting multiple users. Hopefully I can reproduce it

  6. ktaborski commented on Nov 4, 2025

    @ktaborski

    probably duplicate with:
    #57470

  7. 5 remaining items

  8. zachliu commented on Nov 12, 2025

    @zachliu
    Contributor

    looks like it's fixed by #58196
    no longer need to reboot server every 24 hours after jwt expires 🙏🙏🙏
    sadly not gonna be in 3.1.3 😿

  9. potiuk commented on Nov 12, 2025

    @potiuk
    Member

    Cool Closing. I plan to start next provider's release tomorrow.

  10. potiuk commented on Nov 12, 2025

    @potiuk
    Member

    sadly not gonna be in 3.1.3 😿

    You should be able to upgrade to newer FAB provider when it is relased.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:APIAirflow's REST/HTTP APIarea:autharea:corekind:bugThis is a clearly a bugpriority:highHigh priority bug that should be patched quickly but does not require immediate new release

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions