fix: Keep the metrics server running when a Gunicorn worker exits by aborgatin · Pull Request #7012 · feast-dev/feast

aborgatin · 2026-10-14T13:06:56Z

What this PR does / why we need it:

Since #6855 the metrics server runs in a multiprocessing.Process(daemon=True) started in the Gunicorn master. Every forked worker inherits multiprocessing's registry of child processes, and at worker exit multiprocessing.util._exit_function terminates every daemon process in it. So the first worker restart (--max-requests, worker timeout, reload) sends SIGTERM to the master's metrics server, and /metrics stops responding until feast serve is restarted. Every worker exit also logs AssertionError: can only join a child process. Details and steps to reproduce in #7011.

When the metrics process is started, this registers an os.register_at_fork(after_in_child=...) hook that removes it from the registry in every child forked from that process. An at-fork hook rather than a Gunicorn hook, so it also covers workers that exit before post_worker_init (e.g. a worker that fails to boot). Workers no longer terminate the metrics server; the master still terminates it on its own exit.

Tested with the steps from #7011 (--max-requests 3):

macOS, Python 3.12: /metrics returns 200 after the worker restarts (000 before the change), no AssertionError.
Linux (python:3.12-slim, 0.67.0 with this metrics.py): 0.67.0 as released: /metrics 000 after the first worker exit. With the change, 2 workers: 19 worker exits, /metrics answered all 300 checks, no AssertionError; after stopping feast serve no metrics process is left.

Which issue(s) this PR fixes:

Fixes #7011

Checks

I've made sure the tests are passing. (pytest sdk/python/tests/unit/test_metrics.py: 127 passed; ruff check and ruff format --check pass; mypy feast reports no errors in metrics.py)
My commits are signed off (git commit -s)
My PR title follows conventional commits format

Testing Strategy

Unit tests – TestMetricsProcessSurvivesForkedChildExit: start_metrics_server registers a daemon stand-in process (fork context, so it starts instantly on every platform), a forked child runs multiprocessing.util._exit_function() as a worker does at exit, then the test checks the process is still alive (before the change it is terminated with -SIGTERM). The /metrics HTTP checks are the end-to-end runs above.
Integration tests

🤖 Generated with Claude Code

codecov-commenter · 2026-10-14T13:18:38Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 88.88889% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 50.99%. Comparing base (1299e4e) to head (125f4e8).
⚠️ Report is 1 commits behind head on master.

Files with missing lines	Patch %	Lines
sdk/python/feast/metrics.py	88.88%	0 Missing and 1 partial ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #7012   +/-   ##
=======================================
  Coverage   50.99%   50.99%           
=======================================
  Files         455      455           
  Lines       56970    56979    +9     
  Branches     8313     8315    +2     
=======================================
+ Hits        29051    29059    +8     
  Misses      25935    25935           
- Partials     1984     1985    +1

Flag	Coverage Δ
go-feature-server	`30.58% <ø> (ø)`
python-unit	`52.46% <88.88%> (+<0.01%)`	⬆️

Files with missing lines	Coverage Δ
sdk/python/feast/metrics.py	`86.19% <88.88%> (+0.10%)`	⬆️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 1299e4e...125f4e8. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

patelchaitany · 2026-10-15T11:25:29Z

+            pid = os.fork()
+            if pid == 0:
+                # What a worker runs at interpreter exit
+                multiprocessing.util._exit_function()


LGTM. I reproduced the worker exit killing the metrics process and confirmed this fix keeps it alive. One small thing: the test runs the full _exit_function() inside pytest, so checking proc not in active_children() in the forked child would be safer.

Thanks for reviewing! Just applied this fix, could you please check

patelchaitany

Thanks for the update! LGTM.

Since the metrics server runs in a multiprocessing.Process started in the Gunicorn master, every forked worker inherits multiprocessing's registry of child processes. At worker exit, multiprocessing.util._exit_function terminates every daemon process in that registry, so the first worker restart (--max-requests, timeout, reload) sends SIGTERM to the master's metrics server and /metrics stops responding. It also logs "AssertionError: can only join a child process" on every worker exit. Register an at-fork hook when the metrics process is started that removes it from the registry in every forked child. Workers no longer terminate it; the master still does on its own exit. Fixes feast-dev#7011 Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Signed-off-by: Alexandr Borgatin <[email protected]>

On macOS CI the spawned metrics process can take longer than the 5 s readiness wait to start listening, so the first /metrics request was refused. Poll /metrics (up to 30 s) before forking the child. Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Signed-off-by: Alexandr Borgatin <[email protected]>

The spawned metrics server did not come up within 30 s inside the full unit suite on macOS CI. The test is about the process surviving a forked child's exit, not about serving HTTP, so register a fork-context daemon process via start_metrics_server instead and assert it is still alive after the child runs multiprocessing.util._exit_function(). Without the fix it is terminated (exitcode -SIGTERM). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Signed-off-by: Alexandr Borgatin <[email protected]>

…_exit_function Running multiprocessing.util._exit_function() in a fork of the pytest process runs every registered finalizer. Check what a worker's exit would act on instead: the metrics process must not be in the child's active_children(). Also cover _forget_metrics_process_in_child directly, and kill the stand-in process in cleanup (terminate() can wait out the join timeout under xdist). Co-Authored-By: Claude Opus 5.5 (1M context) <[email protected]> Signed-off-by: Alexandr Borgatin <[email protected]>

# [0.68.0](v0.67.0...v0.68.0) (2026-11-10) ### Bug Fixes * add watch verb on clusterroles and clusterrolebindings for feast-operator ([#7062](#7062)) ([b6bdda4](b6bdda4)) * Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([#6993](#6993)) ([f416309](f416309)) * Break event timestamp ties by created timestamp in Milvus write batches ([#7084](#7084)) ([bbaca76](bbaca76)) * Break event timestamp ties by created timestamp in the Dask offline store ([#6952](#6952)) ([eb93b08](eb93b08)) * Build the Go feature server image in two stages ([1e5e90c](1e5e90c)), closes [#7060](#7060) * Fall back to the serialized body when UDF source misses globals ([#6816](#6816)) ([f61079f](f61079f)) * Install patched libbrotli1 to address DSA-6582-1 ([#7096](#7096)) ([d458393](d458393)) * Keep description, tags and nested types of ODFV schema fields ([8ef6e75](8ef6e75)) * Keep entity columns out of StreamFeatureView proto features ([#7033](#7033)) ([d520584](d520584)) * Keep the metrics server running when a Gunicorn worker exits ([#7012](#7012)) ([68c5d54](68c5d54)), closes [#7011](#7011) * Key registry proto caches on the proto, not its id() ([#7085](#7085)) ([147adad](147adad)) * Log ODFV materialization skips instead of printing them ([#7117](#7117)) ([8fdcee7](8fdcee7)) * Pin build dependencies of source builds to the CI lock ([#7104](#7104)) ([dec2f24](dec2f24)) * Populate spec.project in registry REST feature view responses (#… ([#7037](#7037)) ([e304a84](e304a84)) * Preserve colliding source timestamps in Dask retrieval ([#7071](#7071)) ([1396c88](1396c88)) * Preserve Dask historical requests without eligible features ([#7000](#7000)) ([fff09f7](fff09f7)) * Read None elements of UUID and Decimal lists as None ([#7039](#7039)) ([03f30fe](03f30fe)) * Retire the support of Python3.10 ([#7101](#7101)) ([a56766e](a56766e)), closes [#7104](#7104) [#7089](#7089) [#7106](#7106) [#7105](#7105) * Serialize UUID and TimeUuid entity keys ([#7032](#7032)) ([0c845a1](0c845a1)) * Stop the Ray engine from hiding materialization errors ([#7026](#7026)) ([7903e2e](7903e2e)), closes [#7024](#7024) * **ui:** stop treating the monitoring baseline as the latest window ([#7056](#7056)) ([0f01802](0f01802)), closes [#7054](#7054) * Use date-only bounds for partition filters built from string timestamps ([#7116](#7116)) ([b1f2168](b1f2168)) * Validate entity column types in feast dbt import ([#7067](#7067)) ([abca80c](abca80c)), closes [#5876](#5876) * Warm up the whole DynamoDB connection pool ([#6985](#6985)) ([8ce548c](8ce548c)) ### Features * Add --gc-freeze option to feast serve ([#7089](#7089)) ([aad0c15](aad0c15)) * Add native complex types and scalar index tags to the Milvus online store ([#7120](#7120)) ([f7068d6](f7068d6)) * Add num_partitions and partition_key_isolation to the Milvus online store ([#7083](#7083)) ([68de84e](68de84e)), closes [#6940](#6940) * Add Pinecone online store integration ([#6585](#6585)) ([7a84f2a](7a84f2a)) * Add PolarisSource and PolarisProvider for Apache Polaris ([#7006](#7006)) ([386759c](386759c)) * Add row deletion to the Milvus online store ([#7082](#7082)) ([d3e2041](d3e2041)) * Add upstream FeatureView lineage support for PushSource ([#6853](#6853)) ([c9b9d72](c9b9d72)) * Optional nameless pre-computed vectors with a schema fingerprint ([#7077](#7077)) ([9420fac](9420fac)) ### Performance Improvements * Scan Lance sources lazily in the DuckDB offline store ([1299e4e](1299e4e))

# [0.68.0](feast-dev/feast@v0.67.0...v0.68.0) (2026-11-10) ### Bug Fixes * add watch verb on clusterroles and clusterrolebindings for feast-operator ([feast-dev#7062](feast-dev#7062)) ([b6bdda4](feast-dev@b6bdda4)) * Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([feast-dev#6993](feast-dev#6993)) ([f416309](feast-dev@f416309)) * Break event timestamp ties by created timestamp in Milvus write batches ([feast-dev#7084](feast-dev#7084)) ([bbaca76](feast-dev@bbaca76)) * Break event timestamp ties by created timestamp in the Dask offline store ([feast-dev#6952](feast-dev#6952)) ([eb93b08](feast-dev@eb93b08)) * Build the Go feature server image in two stages ([1e5e90c](feast-dev@1e5e90c)), closes [feast-dev#7060](feast-dev#7060) * Fall back to the serialized body when UDF source misses globals ([feast-dev#6816](feast-dev#6816)) ([f61079f](feast-dev@f61079f)) * Install patched libbrotli1 to address DSA-6582-1 ([feast-dev#7096](feast-dev#7096)) ([d458393](feast-dev@d458393)) * Keep description, tags and nested types of ODFV schema fields ([8ef6e75](feast-dev@8ef6e75)) * Keep entity columns out of StreamFeatureView proto features ([feast-dev#7033](feast-dev#7033)) ([d520584](feast-dev@d520584)) * Keep the metrics server running when a Gunicorn worker exits ([feast-dev#7012](feast-dev#7012)) ([68c5d54](feast-dev@68c5d54)), closes [feast-dev#7011](feast-dev#7011) * Key registry proto caches on the proto, not its id() ([feast-dev#7085](feast-dev#7085)) ([147adad](feast-dev@147adad)) * Log ODFV materialization skips instead of printing them ([feast-dev#7117](feast-dev#7117)) ([8fdcee7](feast-dev@8fdcee7)) * Pin build dependencies of source builds to the CI lock ([feast-dev#7104](feast-dev#7104)) ([dec2f24](feast-dev@dec2f24)) * Populate spec.project in registry REST feature view responses (#… ([feast-dev#7037](feast-dev#7037)) ([e304a84](feast-dev@e304a84)) * Preserve colliding source timestamps in Dask retrieval ([feast-dev#7071](feast-dev#7071)) ([1396c88](feast-dev@1396c88)) * Preserve Dask historical requests without eligible features ([feast-dev#7000](feast-dev#7000)) ([fff09f7](feast-dev@fff09f7)) * Read None elements of UUID and Decimal lists as None ([feast-dev#7039](feast-dev#7039)) ([03f30fe](feast-dev@03f30fe)) * Retire the support of Python3.10 ([feast-dev#7101](feast-dev#7101)) ([a56766e](feast-dev@a56766e)), closes [feast-dev#7104](feast-dev#7104) [feast-dev#7089](feast-dev#7089) [feast-dev#7106](feast-dev#7106) [feast-dev#7105](feast-dev#7105) * Serialize UUID and TimeUuid entity keys ([feast-dev#7032](feast-dev#7032)) ([0c845a1](feast-dev@0c845a1)) * Stop the Ray engine from hiding materialization errors ([feast-dev#7026](feast-dev#7026)) ([7903e2e](feast-dev@7903e2e)), closes [feast-dev#7024](feast-dev#7024) * **ui:** stop treating the monitoring baseline as the latest window ([feast-dev#7056](feast-dev#7056)) ([0f01802](feast-dev@0f01802)), closes [feast-dev#7054](feast-dev#7054) * Use date-only bounds for partition filters built from string timestamps ([feast-dev#7116](feast-dev#7116)) ([b1f2168](feast-dev@b1f2168)) * Validate entity column types in feast dbt import ([feast-dev#7067](feast-dev#7067)) ([abca80c](feast-dev@abca80c)), closes [feast-dev#5876](feast-dev#5876) * Warm up the whole DynamoDB connection pool ([feast-dev#6985](feast-dev#6985)) ([8ce548c](feast-dev@8ce548c)) ### Features * Add --gc-freeze option to feast serve ([feast-dev#7089](feast-dev#7089)) ([aad0c15](feast-dev@aad0c15)) * Add native complex types and scalar index tags to the Milvus online store ([feast-dev#7120](feast-dev#7120)) ([f7068d6](feast-dev@f7068d6)) * Add num_partitions and partition_key_isolation to the Milvus online store ([feast-dev#7083](feast-dev#7083)) ([68de84e](feast-dev@68de84e)), closes [feast-dev#6940](feast-dev#6940) * Add Pinecone online store integration ([feast-dev#6585](feast-dev#6585)) ([7a84f2a](feast-dev@7a84f2a)) * Add PolarisSource and PolarisProvider for Apache Polaris ([feast-dev#7006](feast-dev#7006)) ([386759c](feast-dev@386759c)) * Add row deletion to the Milvus online store ([feast-dev#7082](feast-dev#7082)) ([d3e2041](feast-dev@d3e2041)) * Add upstream FeatureView lineage support for PushSource ([feast-dev#6853](feast-dev#6853)) ([c9b9d72](feast-dev@c9b9d72)) * Optional nameless pre-computed vectors with a schema fingerprint ([feast-dev#7077](feast-dev#7077)) ([9420fac](feast-dev@9420fac)) ### Performance Improvements * Scan Lance sources lazily in the DuckDB offline store ([1299e4e](feast-dev@1299e4e)) Signed-off-by: Shuchu Han <[email protected]>

aborgatin requested a review from a team as a code owner October 14, 2026 13:06

patelchaitany reviewed Oct 15, 2026

View reviewed changes

patelchaitany approved these changes Oct 16, 2026

View reviewed changes

haoxu0 added the ok-to-test label Oct 17, 2026 — with ChatGPT Codex Connector

ntkathole approved these changes Oct 18, 2026

View reviewed changes

aborgatin and others added 4 commits October 18, 2026 18:57

ntkathole force-pushed the fix/metrics-process-worker-exit branch from 80dc00f to 125f4e8 Compare October 18, 2026 13:27

ntkathole merged commit 68c5d54 into feast-dev:master Oct 18, 2026
21 of 23 checks passed

patelchaitany mentioned this pull request Nov 11, 2026

chore: Sync upstream feast-dev/feast stable (0.68.0) opendatahub-io/feast#291

Merged

rhods-devops-app Bot mentioned this pull request Nov 12, 2026

Automated Sync from main to stable red-hat-data-services/feast#630

Open

Navigation Menu

Repository navigation

fix: Keep the metrics server running when a Gunicorn worker exits - #7012

fix: Keep the metrics server running when a Gunicorn worker exits#7012
ntkathole merged 4 commits into
feast-dev:masterfrom
aborgatin:fix/metrics-process-worker-exit

aborgatin commented Oct 14, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Oct 14, 2026 •

edited

Loading

Uh oh!

patelchaitany Oct 15, 2026

Uh oh!

aborgatin Oct 16, 2026

Uh oh!

patelchaitany left a comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

aborgatin commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Checks

Testing Strategy

Uh oh!

codecov-commenter commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

patelchaitany Oct 15, 2026

Choose a reason for hiding this comment

Uh oh!

aborgatin Oct 16, 2026

Choose a reason for hiding this comment

Uh oh!

patelchaitany left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

aborgatin commented Oct 14, 2026 •

edited

Loading

codecov-commenter commented Oct 14, 2026 •

edited

Loading