feast serve: /metrics stops responding after the first gunicorn worker exits (regression in 0.67.0, #6855) #7011

Current behavior

Since 0.67.0 (#6855) the Prometheus metrics server runs in a multiprocessing.Process(daemon=True) started in the gunicorn master. Gunicorn workers are forked from the master and inherit multiprocessing's registry of child processes, which contains the metrics process. When a worker exits (--max-requests recycle, worker timeout, HUP reload), Python's atexit handler multiprocessing.util._exit_function runs in the worker, treats the metrics process as its own daemon child and calls terminate() on it (SIGTERM). Then it fails to join() it:

[INFO] Maximum request limit of 3 exceeded. Terminating process.
[INFO] Worker exiting (pid: 77539)
Exception ignored in atexit callback: <function _exit_function at 0x11b90c4a0>
Traceback (most recent call last):
  File ".../multiprocessing/util.py", line 360, in _exit_function
    p.join()
  File ".../multiprocessing/process.py", line 147, in join
    assert self._parent_pid == os.getpid(), 'can only join a child process'
AssertionError: can only join a child process
[INFO] Booting worker with pid: 77567

From then on nothing listens on port 8000: /metrics returns connection refused until feast serve is restarted, while feature serving keeps working. Workers keep writing their metric files, but nothing serves them.

Any feast serve with metrics enabled loses its metrics endpoint at the first worker restart, so with --max-requests this happens shortly after start. It happens with both multiprocessing start methods: fork (Linux) and spawn (macOS). 0.66.0 is not affected.

Expected behavior

The metrics server keeps running for the whole life of the gunicorn master, independent of worker restarts.

Steps to reproduce

1. Create an empty folder with this feature_store.yaml (local registry and SQLite online store, metrics enabled):

project: demo
registry: data/registry.db
provider: local
online_store:
    type: sqlite
    path: data/online_store.db
entity_key_serialization_version: 3
feature_server:
    metrics:
        enabled: true

2. In that folder, install Feast 0.67.0 and register the (empty) repo:

pip install "feast==0.67.0"
mkdir -p data && feast apply

3. Start the server with one worker that is restarted after 3 requests:

feast serve --workers 1 --max-requests 3 --max-requests-jitter 0 > serve.log 2>&1 &

4. Once curl -s -m 2 http://127.0.0.1:6566/health answers, check /metrics, send 5 requests (the worker restarts after the 3rd), and check /metrics again:

curl -s -m 5 -o /dev/null -w 'metrics before: %{http_code}\n' http://127.0.0.1:8000/metrics
for i in 1 2 3 4 5; do curl -s -m 5 -o /dev/null -w "request $i: %{http_code}\n" http://127.0.0.1:6566/health; done
sleep 3
curl -s -m 5 -o /dev/null -w 'metrics after: %{http_code}\n' http://127.0.0.1:8000/metrics
grep -m1 "AssertionError: can only join a child process" serve.log

Output with 0.67.0:

metrics before: 200
request 1: 200
request 2: 200
request 3: 200
request 4: 200
request 5: 200
metrics after: 000
AssertionError: can only join a child process

000 means connection refused: nothing listens on port 8000 any more, while /health keeps returning 200.

With feast==0.66.0 the same steps print metrics after: 200 and no AssertionError.

The mechanism without Feast (plain Python 3.12; prints False on Linux and macOS):

import multiprocessing as mp, os, sys, time

def serve():
    while True:
        time.sleep(1)

if __name__ == "__main__":
    p = mp.Process(target=serve, daemon=True)  # like start_metrics_server() in the gunicorn master
    p.start(); time.sleep(0.5)
    pid = os.fork()                             # like a gunicorn worker
    if pid == 0:
        sys.exit(0)                             # worker exit runs multiprocessing's atexit -> terminate(p)
    os.waitpid(pid, 0); time.sleep(0.5)
    print("metrics process alive:", p.is_alive())   # False

Possible solutions

• In each gunicorn worker (e.g. in _gunicorn_post_worker_init), remove the inherited metrics process from multiprocessing's child registry, so the worker's exit no longer terminates it. The master still owns it and stops it on shutdown.
• Start the metrics process with os.fork() or subprocess instead of multiprocessing, so forked workers don't track it, and have the master terminate it on exit.

Environment

• Feast 0.67.0 (0.66.0 not affected), gunicorn 26.2.0, prometheus-client 0.24.1, Python 3.12
• Linux (python:3.12-slim) and macOS 27