Skip to content

The feast server crashes when starting multiple workers with SQL registry #5784

New issue
New issue
Open
#7065
#5795
Open
The feast server crashes when starting multiple workers with SQL registry#5784
#7065
#5795
Labels
kind/bugpriority/p2

Description

@daseindev1
daseindev1
opened on Dec 17, 2025

Expected Behavior

Running the Feast feature server with multiple Gunicorn workers (e.g. feast serve -w 4) should work reliably with a SQL Registry backend, including periodic registry refresh, without raising SQLAlchemy errors.

Current Behavior

When starting the feature server with multiple workers (e.g. feast serve -w 4), Feast intermittently crashes the background registry refresh thread with:

feast-serve-1  | INFO:fastapi:Auth type: AuthManagerType.NONE
feast-serve-1  | [2025-11-24 03:58:45 +0000] [1] [INFO] Starting gunicorn 23.0.0
feast-serve-1  | [2025-11-24 03:58:45 +0000] [1] [INFO] Listening at: http://0.0.0.0:6556 (1)
feast-serve-1  | [2025-11-24 03:58:45 +0000] [1] [INFO] Using worker: uvicorn_worker.UvicornWorker
feast-serve-1  | /usr/local/lib/python3.12/site-packages/gunicorn/arbiter.py:592: DeprecationWarning: This process (pid=1) is multi-threaded, use of fork() may lead to deadlocks in the child.
feast-serve-1  |   pid = os.fork()
feast-serve-1  | [2025-11-24 03:58:45 +0000] [31] [INFO] Booting worker with pid: 31
feast-serve-1  | /usr/local/lib/python3.12/site-packages/websockets/legacy/__init__.py:6: DeprecationWarning: websockets.legacy is deprecated; see https://websockets.readthedocs.io/en/stable/howto/upgrade.html for upgrade instructions
feast-serve-1  |   warnings.warn(  # deprecated in 14.0 - 2024-11-09
feast-serve-1  | /usr/local/lib/python3.12/site-packages/uvicorn/protocols/websockets/websockets_impl.py:17: DeprecationWarning: websockets.server.WebSocketServerProtocol is deprecated
feast-serve-1  |   from websockets.server import WebSocketServerProtocol
feast-serve-1  | [2025-11-24 03:58:45 +0000] [31] [INFO] Started server process [31]
feast-serve-1  | [2025-11-24 03:58:45 +0000] [31] [INFO] Waiting for application startup.
feast-serve-1  | [2025-11-24 03:58:45 +0000] [31] [INFO] Application startup complete.
feast-serve-1  | [2025-11-24 03:58:45 +0000] [33] [INFO] Booting worker with pid: 33
feast-serve-1  | /usr/local/lib/python3.12/site-packages/websockets/legacy/__init__.py:6: DeprecationWarning: websockets.legacy is deprecated; see https://websockets.readthedocs.io/en/stable/howto/upgrade.html for upgrade instructions
feast-serve-1  |   warnings.warn(  # deprecated in 14.0 - 2024-11-09
feast-serve-1  | /usr/local/lib/python3.12/site-packages/uvicorn/protocols/websockets/websockets_impl.py:17: DeprecationWarning: websockets.server.WebSocketServerProtocol is deprecated
feast-serve-1  |   from websockets.server import WebSocketServerProtocol
feast-serve-1  | [2025-11-24 03:58:45 +0000] [33] [INFO] Started server process [33]
feast-serve-1  | [2025-11-24 03:58:45 +0000] [33] [INFO] Waiting for application startup.
feast-serve-1  | [2025-11-24 03:58:45 +0000] [33] [INFO] Application startup complete.
feast-serve-1  | [2025-11-24 03:58:45 +0000] [35] [INFO] Booting worker with pid: 35
feast-serve-1  | /usr/local/lib/python3.12/site-packages/websockets/legacy/__init__.py:6: DeprecationWarning: websockets.legacy is deprecated; see https://websockets.readthedocs.io/en/stable/howto/upgrade.html for upgrade instructions
feast-serve-1  |   warnings.warn(  # deprecated in 14.0 - 2024-11-09
feast-serve-1  | /usr/local/lib/python3.12/site-packages/uvicorn/protocols/websockets/websockets_impl.py:17: DeprecationWarning: websockets.server.WebSocketServerProtocol is deprecated
feast-serve-1  |   from websockets.server import WebSocketServerProtocol
feast-serve-1  | [2025-11-24 03:58:45 +0000] [35] [INFO] Started server process [35]
feast-serve-1  | [2025-11-24 03:58:45 +0000] [35] [INFO] Waiting for application startup.
feast-serve-1  | [2025-11-24 03:58:45 +0000] [35] [INFO] Application startup complete.
feast-serve-1  | [2025-11-24 03:58:45 +0000] [36] [INFO] Booting worker with pid: 36
feast-serve-1  | /usr/local/lib/python3.12/site-packages/websockets/legacy/__init__.py:6: DeprecationWarning: websockets.legacy is deprecated; see https://websockets.readthedocs.io/en/stable/howto/upgrade.html for upgrade instructions
feast-serve-1  |   warnings.warn(  # deprecated in 14.0 - 2024-11-09
feast-serve-1  | /usr/local/lib/python3.12/site-packages/uvicorn/protocols/websockets/websockets_impl.py:17: DeprecationWarning: websockets.server.WebSocketServerProtocol is deprecated
feast-serve-1  |   from websockets.server import WebSocketServerProtocol
feast-serve-1  | [2025-11-24 03:58:45 +0000] [36] [INFO] Started server process [36]
feast-serve-1  | [2025-11-24 03:58:45 +0000] [36] [INFO] Waiting for application startup.
feast-serve-1  | [2025-11-24 03:58:45 +0000] [36] [INFO] Application startup complete.
feast-serve-1  | Exception in thread Thread-18:
feast-serve-1  | Traceback (most recent call last):
feast-serve-1  |   File "/usr/local/lib/python3.12/threading.py", line 1075, in _bootstrap_inner
feast-serve-1  |     self.run()
feast-serve-1  |   File "/usr/local/lib/python3.12/threading.py", line 1433, in run
feast-serve-1  |     self.function(*self.args, **self.kwargs)
feast-serve-1  |   File "/usr/local/lib/python3.12/site-packages/feast/feature_server.py", line 209, in async_refresh
feast-serve-1  |     registry_proto = store.registry.proto()
feast-serve-1  |                      ^^^^^^^^^^^^^^^^^^^^^^
feast-serve-1  |   File "/usr/local/lib/python3.12/site-packages/feast/infra/registry/sql.py", line 878, in proto
feast-serve-1  |     process_project(project)
feast-serve-1  |   File "/usr/local/lib/python3.12/site-packages/feast/infra/registry/sql.py", line 861, in process_project
feast-serve-1  |     objs: List[Any] = lister(project_name, allow_cache=False)  # type: ignore
feast-serve-1  |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
feast-serve-1  |   File "/usr/local/lib/python3.12/site-packages/feast/infra/registry/caching_registry.py", line 207, in list_on_demand_feature_views
feast-serve-1  |     return self._list_on_demand_feature_views(project, tags)
feast-serve-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
feast-serve-1  |   File "/usr/local/lib/python3.12/site-packages/feast/infra/registry/sql.py", line 646, in _list_on_demand_feature_views
feast-serve-1  |     return self._list_objects(
feast-serve-1  |            ^^^^^^^^^^^^^^^^^^^
feast-serve-1  |   File "/usr/local/lib/python3.12/site-packages/feast/infra/registry/sql.py", line 1094, in _list_objects
feast-serve-1  |     rows = conn.execute(stmt).all()
feast-serve-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^
feast-serve-1  |   File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/result.py", line 1384, in all
feast-serve-1  |     return self._allrows()
feast-serve-1  |            ^^^^^^^^^^^^^^^
feast-serve-1  |   File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/result.py", line 546, in _allrows
feast-serve-1  |     make_row = self._row_getter
feast-serve-1  |                ^^^^^^^^^^^^^^^^
feast-serve-1  |   File "/usr/local/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", line 1338, in __get__
feast-serve-1  |     obj.__dict__[self.__name__] = result = self.fget(obj)
feast-serve-1  |                                            ^^^^^^^^^^^^^^
feast-serve-1  |   File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/result.py", line 471, in _row_getter
feast-serve-1  |     key_to_index = metadata._key_to_index
feast-serve-1  |                    ^^^^^^^^^^^^^^^^^^^^^^
feast-serve-1  |   File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/cursor.py", line 1366, in _key_to_index
feast-serve-1  |     self._we_dont_return_rows()
feast-serve-1  |   File "/usr/local/lib/python3.12/site-packages/sqlalchemy/engine/cursor.py", line 1346, in _we_dont_return_rows
feast-serve-1  |     raise exc.ResourceClosedError(
feast-serve-1  | sqlalchemy.exc.ResourceClosedError: This result object does not return rows. It has been closed automatically.

This happens during the periodic registry refresh (async_refresh) when it calls store.registry.proto(), which triggers SQL registry list operations (e.g. listing OnDemandFeatureViews).

Relevant code paths:

Specs:

  • OS: Red Hat 8.1
  • Python 3.12.11
  • Gunicorn 23.0.0
  • feast 0.55.1
  • SQLAlchemy 2.0.44
  • psycopg2-binary 2.9.10

Steps to reproduce

  1. Configure Feast to use the SQL Registry, e.g. (from docs):
    https://github.com/feast-dev/feast/blob/master/docs/reference/registries/sql.md
project: feature_repo
provider: local  # Provider is local because we are defining all stores manually

registry:
  registry_type: sql
  # Use a dedicated schema 'feast' to avoid naming conflicts with PostgreSQL system tables
  path: postgresql+psycopg2://postgres:postgres@postgres:5432/feast?options=-c%20search_path=feast

online_store:
  type: redis
  connection_string: "redis:6379"

offline_store:
  type: file

entity_key_serialization_version: 3

  1. Start the feature server with multiple workers:
    feast --feature-store-yaml feature_store.yaml serve --host 0.0.0.0 --port 18080 -w 4

  2. Wait for a while until multiple workers are spawned

  3. Observe sqlalchemy.exc.ResourceClosedError thrown in the background refresh thread.

  4. (Control) Starting with a single worker (-w 1) does not reproduce the error.

Possible Solution

This looks like a fork-safety / lifecycle issue: the FeatureStore/SqlRegistry (and its SQLAlchemy Engine/pool state) are created before Gunicorn forks workers, then each worker runs a background thread (threading.Timer) that uses the inherited engine/pool, leading to invalid/closed DBAPI cursor/result state.

Potential fixes:

  • Create the FeatureStore/SqlRegistry inside each worker after fork, rather than constructing it in the Gunicorn master and passing it into the app (i.e., avoid pre-fork engine creation).
  • Alternatively, dispose/recreate SQLAlchemy engines after fork (e.g. via a Gunicorn post_fork hook) so each worker has a fresh pool.
  • Documentation note / guardrail: warn against multi-worker mode with SQL Registry unless post-fork initialization is used.

Activity

  1. daseindev1
    added
    kind/bug
    priority/p2
    on Dec 17, 2025
  2. u-sheikh
    added a commit that references this issue on Dec 28, 2025

    fix: Resolve multi-worker crash with SQL Registry (feast-dev#5784)

    1081e82

  3. casaar97 commented on Oct 23, 2026

    @casaar97
    casaar97
    on Oct 23, 2026
    Contributor

    I hit the same root cause with a MySQL registry over TLS, on Feast 0.66.0, and the code path is unchanged on current master. Adding details in case they help.

    What happens. With feast serve --workers N, FeastServeApplication.__init__ calls get_app() in the Gunicorn master (sdk/python/feast/feature_server.py:1271-1281). get_app() builds VectorStoreRegistry(store) (line 599), whose constructor lists feature views and so creates the SqlRegistry and runs queries in the master (infra/registry/sql.py:351-389). The SQLAlchemy QueuePool keeps that connection idle in the master. Every forked worker inherits the pool and checks out the same DBAPI connection. Tracing the pool's connect/checkout events with the MySQL server_thread_id, the master opened server thread 13 and both workers checked out thread 13. Two processes then talk over one socket.

    What it looks like with MySQL + TLS (PyMySQL):

    pymysql.err.OperationalError: (2013, 'Lost connection to MySQL server during query ([SSL] record layer failure (_ssl.c:2590))')

    It shows up in two ways:

    • as a non-fatal sqlalchemy.pool "Exception during reset or similar" (raised from _finalize_fairy -> _reset -> do_rollback);
    • when it happens inside async_refresh() in the lifespan (feature_server.py:635, no error handling), as "Application startup failed. Exiting." The worker exits with code 3, and Gunicorn then stops the whole server ("Worker failed to boot."), even though the other workers were healthy. I confirmed this second path by injecting the same error into one worker's lifespan. Timing alone did not trigger it in 18 local runs, but it does happen in real deployments.

    Without TLS the same sharing gives other protocol errors, like the ResourceClosedError above. Re-forks from --max-requests (default 1000) inherit the master's pool again.

    Deterministic check (no Feast needed):

    import os
from sqlalchemy import create_engine, text

engine = create_engine("mysql+pymysql://user:[email protected]:3306/feast?ssl_verify_cert=false")
with engine.connect() as c:
    c.execute(text("SELECT 1"))
    parent_thread = c.connection.dbapi_connection.server_thread_id[0]

pid = os.fork()
if pid == 0:
    # engine.dispose(close=False)  # <- uncomment: the child gets its own connection
    with engine.connect() as c:
        child_thread = c.connection.dbapi_connection.server_thread_id[0]
    print("child reuses parent's connection:", child_thread == parent_thread)
    os._exit(0)
os.waitpid(pid, 0)

    With two children running queries in a loop, both fail with the 2013 errors above, in 3 of 3 runs. With engine.dispose(close=False) in each child, both pass, in 3 of 3 runs.

    Fix. SQLAlchemy's documented recipe ("Using Connection Pools with Multiprocessing or os.fork()") is engine.dispose(close=False) in the child. close=False matters: a plain dispose() in the child closes the inherited connections and sends a disconnect on the socket the parent still uses (sqlalchemy/sqlalchemy#7815). A small change would be:

    • a no-op reset_after_fork() on BaseRegistry, overridden in SqlRegistry to call dispose(close=False) on write_engine (and on read_engine when it is a different engine);
    • a Gunicorn post_fork hook, set in FeastServeApplication.load_config for every run, not only when metrics are on, that calls it.

    I see #5795 is open for this. It calls dispose() without close=False and also includes unrelated DuckDB changes. I'm happy to help there, or to send a focused PR with unit tests (a file-backed SQLite SqlRegistry + os.fork() reproduces the shared connection deterministically), whichever the maintainers prefer.

    Workaround until then. Load SQLAlchemy's pid-guard pool events before the Feast CLI, for example in a small wrapper used instead of the feast console script:

    import os
import sys

from sqlalchemy import event, exc
from sqlalchemy.pool import Pool


@event.listens_for(Pool, "connect")
def _remember_pid(dbapi_connection, connection_record):
    connection_record.info["pid"] = os.getpid()


@event.listens_for(Pool, "checkout")
def _check_pid(dbapi_connection, connection_record, connection_proxy):
    owner = connection_record.info.get("pid")
    pid = os.getpid()
    if owner is not None and owner != pid:
        # Forget the inherited connection without closing it (the parent still uses it).
        connection_record.dbapi_connection = connection_proxy.dbapi_connection = None
        raise exc.DisconnectionError(
            f"Connection record belongs to pid {owner}, attempting to check out in pid {pid}"
        )


from feast.cli.cli import cli  # noqa: E402

if __name__ == "__main__":
    sys.exit(cli())

    Run python entrypoint.py serve --workers N .... With it, every process (master and each worker) opens exactly one connection of its own, and the errors stop.

    Environment: Feast 0.66.0 (code path unchanged on master), gunicorn 25.3.0, uvicorn-worker 0.3.0, SQLAlchemy 2.0.49, PyMySQL 1.1.2, Python 3.11, MySQL 8.4 with TLS 1.3.

  4. casaar97
    added a commit that references this issue on Oct 27, 2026

    fix: Dispose SQL registry connection pools after Gunicorn fork

    b891917
  5. casaar97
    added a commit that references this issue on Nov 4, 2026

    fix: Dispose SQL registry connection pools after Gunicorn fork

    c9771ff
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bugpriority/p2

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions