feat: Add --gc-freeze option to feast serve by aborgatin · Pull Request #7089 · feast-dev/feast

aborgatin · 2026-10-29T09:47:14Z

What this PR does / why we need it:

Adds an opt-in feast serve --gc-freeze flag, also available as feature_server.gc_freeze in feature_store.yaml.

Full GC collections in a feature server worker walk the registry, feature definitions and protobuf classes that live for the whole process. Each one blocks the event loop for 50–75 ms, and the requests waiting at that moment look like slow online store reads. With the flag, the server runs gc.collect() and gc.freeze() in the gunicorn master after the app is built and before the workers fork. Workers inherit the frozen state, including the ones restarted by --max-requests. On Windows, where the server runs on uvicorn without gunicorn, it freezes after get_app. Off by default, so nothing changes for current users.

Also adds a short "Garbage collection pauses" section to the performance tuning guide and the flag to the Python feature server reference.

Which issue(s) this PR fixes:

Fixes #7088

Checks

I've made sure the tests are passing.
My commits are signed off (git commit -s)
My PR title follows conventional commits format

Testing Strategy

Unit tests
Integration tests
Manual tests
Testing is not required for this change

Unit tests check the call order (collect, freeze, then the gunicorn run) for the flag and the config, and that nothing is frozen by default.

Manual test: one worker, 15 req/s for 10 min. Requests over 50 ms went from 16 of 8,991 to 0 of 8,933, p99.9 from 61 to 29 ms. Also tested on a Kubernetes deployment with 0.65.0 and --max-requests, with the same result. Details in #7088.

codecov-commenter · 2026-10-29T09:52:25Z

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 86.66667% with 2 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.30%. Comparing base (dec2f24) to head (cf86e1c).

Files with missing lines	Patch %	Lines
sdk/python/feast/feature_server.py	83.33%	2 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #7089      +/-   ##
==========================================
+ Coverage   51.28%   51.30%   +0.02%     
==========================================
  Files         458      458              
  Lines       57409    57423      +14     
  Branches     8415     8417       +2     
==========================================
+ Hits        29444    29463      +19     
+ Misses      25957    25950       -7     
- Partials     2008     2010       +2

Flag	Coverage Δ
go-feature-server	`30.58% <ø> (ø)`
python-unit	`52.78% <86.66%> (+0.02%)`	⬆️

Files with missing lines	Coverage Δ
sdk/python/feast/cli/serve.py	`69.38% <100.00%> (+0.31%)`	⬆️
sdk/python/feast/feature_store.py	`46.88% <ø> (ø)`
.../python/feast/infra/feature_servers/base_config.py	`100.00% <100.00%> (ø)`
sdk/python/feast/feature_server.py	`65.63% <83.33%> (+1.19%)`	⬆️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dec2f24...cf86e1c. Read the comment docs.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

shuchu

lgtm

shuchu · 2026-11-04T03:21:02Z

@aborgatin the PR is out-of-date, please rebase the master branch.

Full garbage collections walk the registry, feature definitions and protobuf classes that the feature server keeps for its whole life. In a worker this blocks the event loop for tens of milliseconds and shows up as slow online store reads. The new option runs gc.collect() and gc.freeze() after the app is built and before gunicorn forks the workers. Off by default; also available as feature_server.gc_freeze. Signed-off-by: Alexander Borgatin <[email protected]>

aborgatin · 2026-11-04T08:23:54Z

@aborgatin the PR is out-of-date, please rebase the master branch.

Hi @shuchu, thanks for the reminder! I've rebased onto master, with no conflicts. The one failing check (unit-test-python (3.10)) seems unrelated: sqlglotc fails to build on Python 3.10, and it fails on master too. Could you please take another look when you have a moment? Thanks!

Signed-off-by: SHUCHU HAN <[email protected]>

* fix: Retire the support of Python3.10. Signed-off-by: SHUCHU HAN <[email protected]> Signed-off-by: SHUCHU HAN <[email protected]> * fix: Update the pyproject.toml file. Signed-off-by: Shuchu Han <[email protected]> Signed-off-by: SHUCHU HAN <[email protected]> * fix: Update the pixi.lock. Signed-off-by: Shuchu Han <[email protected]> Signed-off-by: SHUCHU HAN <[email protected]> * fix: Update the pixi.lock with python3.11 Signed-off-by: Shuchu Han <[email protected]> Signed-off-by: SHUCHU HAN <[email protected]> * fix: Fix snowflake_udf unit test error. Signed-off-by: Shuchu Han <[email protected]> Signed-off-by: SHUCHU HAN <[email protected]> * fix: Pin build dependencies of source builds to the CI lock (#7104) sqlglotc has no Python 3.10 wheels on PyPI, so on 3.10 uv builds sqlglotc==30.21.0 from source. Its build pulls the newest unpinned sqlglot, whose Schema class is no longer a mypyc trait, and mypyc fails with "Multiple inheritance is not supported (except for traits)". Pass the CI lock as --build-constraints so source builds use the locked versions (sqlglot==30.21.0) instead of the newest release. Signed-off-by: Chaitany Patel <[email protected]> Signed-off-by: SHUCHU HAN <[email protected]> * feat: Add --gc-freeze option to feast serve (#7089) Signed-off-by: SHUCHU HAN <[email protected]> * ci: Pin Java 17 for HDFS registration tests (#7106) Signed-off-by: Chaitany Patel <[email protected]> Co-authored-by: Shuchu Han <[email protected]> Signed-off-by: SHUCHU HAN <[email protected]> * ci: Align Pixi version with shared lockfile (#7105) * Initial plan * ci: Align Pixi version with shared lockfile Co-authored-by: shuchu <[email protected]> --------- Co-authored-by: copilot-swe-agent[bot] <[email protected]> Co-authored-by: shuchu <[email protected]> Co-authored-by: Shuchu Han <[email protected]> Signed-off-by: SHUCHU HAN <[email protected]> * Update dependencies in pyproject.toml for ray Co-authored-by: Copilot Autofix powered by AI <[email protected]> Signed-off-by: SHUCHU HAN <[email protected]> * fix: Update the pixi.lock. Signed-off-by: Shuchu Han <[email protected]> Signed-off-by: SHUCHU HAN <[email protected]> --------- Signed-off-by: SHUCHU HAN <[email protected]> Signed-off-by: SHUCHU HAN <[email protected]> Signed-off-by: Shuchu Han <[email protected]> Signed-off-by: Chaitany Patel <[email protected]> Co-authored-by: SHUCHU HAN <[email protected]> Co-authored-by: Chaitany Patel <[email protected]> Co-authored-by: Alexandr Borgatin <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: shuchu <[email protected]> Co-authored-by: Copilot Autofix powered by AI <[email protected]>

# [0.68.0](v0.67.0...v0.68.0) (2026-11-10) ### Bug Fixes * add watch verb on clusterroles and clusterrolebindings for feast-operator ([#7062](#7062)) ([b6bdda4](b6bdda4)) * Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([#6993](#6993)) ([f416309](f416309)) * Break event timestamp ties by created timestamp in Milvus write batches ([#7084](#7084)) ([bbaca76](bbaca76)) * Break event timestamp ties by created timestamp in the Dask offline store ([#6952](#6952)) ([eb93b08](eb93b08)) * Build the Go feature server image in two stages ([1e5e90c](1e5e90c)), closes [#7060](#7060) * Fall back to the serialized body when UDF source misses globals ([#6816](#6816)) ([f61079f](f61079f)) * Install patched libbrotli1 to address DSA-6582-1 ([#7096](#7096)) ([d458393](d458393)) * Keep description, tags and nested types of ODFV schema fields ([8ef6e75](8ef6e75)) * Keep entity columns out of StreamFeatureView proto features ([#7033](#7033)) ([d520584](d520584)) * Keep the metrics server running when a Gunicorn worker exits ([#7012](#7012)) ([68c5d54](68c5d54)), closes [#7011](#7011) * Key registry proto caches on the proto, not its id() ([#7085](#7085)) ([147adad](147adad)) * Log ODFV materialization skips instead of printing them ([#7117](#7117)) ([8fdcee7](8fdcee7)) * Pin build dependencies of source builds to the CI lock ([#7104](#7104)) ([dec2f24](dec2f24)) * Populate spec.project in registry REST feature view responses (#… ([#7037](#7037)) ([e304a84](e304a84)) * Preserve colliding source timestamps in Dask retrieval ([#7071](#7071)) ([1396c88](1396c88)) * Preserve Dask historical requests without eligible features ([#7000](#7000)) ([fff09f7](fff09f7)) * Read None elements of UUID and Decimal lists as None ([#7039](#7039)) ([03f30fe](03f30fe)) * Retire the support of Python3.10 ([#7101](#7101)) ([a56766e](a56766e)), closes [#7104](#7104) [#7089](#7089) [#7106](#7106) [#7105](#7105) * Serialize UUID and TimeUuid entity keys ([#7032](#7032)) ([0c845a1](0c845a1)) * Stop the Ray engine from hiding materialization errors ([#7026](#7026)) ([7903e2e](7903e2e)), closes [#7024](#7024) * **ui:** stop treating the monitoring baseline as the latest window ([#7056](#7056)) ([0f01802](0f01802)), closes [#7054](#7054) * Use date-only bounds for partition filters built from string timestamps ([#7116](#7116)) ([b1f2168](b1f2168)) * Validate entity column types in feast dbt import ([#7067](#7067)) ([abca80c](abca80c)), closes [#5876](#5876) * Warm up the whole DynamoDB connection pool ([#6985](#6985)) ([8ce548c](8ce548c)) ### Features * Add --gc-freeze option to feast serve ([#7089](#7089)) ([aad0c15](aad0c15)) * Add native complex types and scalar index tags to the Milvus online store ([#7120](#7120)) ([f7068d6](f7068d6)) * Add num_partitions and partition_key_isolation to the Milvus online store ([#7083](#7083)) ([68de84e](68de84e)), closes [#6940](#6940) * Add Pinecone online store integration ([#6585](#6585)) ([7a84f2a](7a84f2a)) * Add PolarisSource and PolarisProvider for Apache Polaris ([#7006](#7006)) ([386759c](386759c)) * Add row deletion to the Milvus online store ([#7082](#7082)) ([d3e2041](d3e2041)) * Add upstream FeatureView lineage support for PushSource ([#6853](#6853)) ([c9b9d72](c9b9d72)) * Optional nameless pre-computed vectors with a schema fingerprint ([#7077](#7077)) ([9420fac](9420fac)) ### Performance Improvements * Scan Lance sources lazily in the DuckDB offline store ([1299e4e](1299e4e))

# [0.68.0](feast-dev/feast@v0.67.0...v0.68.0) (2026-11-10) ### Bug Fixes * add watch verb on clusterroles and clusterrolebindings for feast-operator ([feast-dev#7062](feast-dev#7062)) ([b6bdda4](feast-dev@b6bdda4)) * Apply on demand feature views in SparkRetrievalJob.to_spark_df() ([feast-dev#6993](feast-dev#6993)) ([f416309](feast-dev@f416309)) * Break event timestamp ties by created timestamp in Milvus write batches ([feast-dev#7084](feast-dev#7084)) ([bbaca76](feast-dev@bbaca76)) * Break event timestamp ties by created timestamp in the Dask offline store ([feast-dev#6952](feast-dev#6952)) ([eb93b08](feast-dev@eb93b08)) * Build the Go feature server image in two stages ([1e5e90c](feast-dev@1e5e90c)), closes [feast-dev#7060](feast-dev#7060) * Fall back to the serialized body when UDF source misses globals ([feast-dev#6816](feast-dev#6816)) ([f61079f](feast-dev@f61079f)) * Install patched libbrotli1 to address DSA-6582-1 ([feast-dev#7096](feast-dev#7096)) ([d458393](feast-dev@d458393)) * Keep description, tags and nested types of ODFV schema fields ([8ef6e75](feast-dev@8ef6e75)) * Keep entity columns out of StreamFeatureView proto features ([feast-dev#7033](feast-dev#7033)) ([d520584](feast-dev@d520584)) * Keep the metrics server running when a Gunicorn worker exits ([feast-dev#7012](feast-dev#7012)) ([68c5d54](feast-dev@68c5d54)), closes [feast-dev#7011](feast-dev#7011) * Key registry proto caches on the proto, not its id() ([feast-dev#7085](feast-dev#7085)) ([147adad](feast-dev@147adad)) * Log ODFV materialization skips instead of printing them ([feast-dev#7117](feast-dev#7117)) ([8fdcee7](feast-dev@8fdcee7)) * Pin build dependencies of source builds to the CI lock ([feast-dev#7104](feast-dev#7104)) ([dec2f24](feast-dev@dec2f24)) * Populate spec.project in registry REST feature view responses (#… ([feast-dev#7037](feast-dev#7037)) ([e304a84](feast-dev@e304a84)) * Preserve colliding source timestamps in Dask retrieval ([feast-dev#7071](feast-dev#7071)) ([1396c88](feast-dev@1396c88)) * Preserve Dask historical requests without eligible features ([feast-dev#7000](feast-dev#7000)) ([fff09f7](feast-dev@fff09f7)) * Read None elements of UUID and Decimal lists as None ([feast-dev#7039](feast-dev#7039)) ([03f30fe](feast-dev@03f30fe)) * Retire the support of Python3.10 ([feast-dev#7101](feast-dev#7101)) ([a56766e](feast-dev@a56766e)), closes [feast-dev#7104](feast-dev#7104) [feast-dev#7089](feast-dev#7089) [feast-dev#7106](feast-dev#7106) [feast-dev#7105](feast-dev#7105) * Serialize UUID and TimeUuid entity keys ([feast-dev#7032](feast-dev#7032)) ([0c845a1](feast-dev@0c845a1)) * Stop the Ray engine from hiding materialization errors ([feast-dev#7026](feast-dev#7026)) ([7903e2e](feast-dev@7903e2e)), closes [feast-dev#7024](feast-dev#7024) * **ui:** stop treating the monitoring baseline as the latest window ([feast-dev#7056](feast-dev#7056)) ([0f01802](feast-dev@0f01802)), closes [feast-dev#7054](feast-dev#7054) * Use date-only bounds for partition filters built from string timestamps ([feast-dev#7116](feast-dev#7116)) ([b1f2168](feast-dev@b1f2168)) * Validate entity column types in feast dbt import ([feast-dev#7067](feast-dev#7067)) ([abca80c](feast-dev@abca80c)), closes [feast-dev#5876](feast-dev#5876) * Warm up the whole DynamoDB connection pool ([feast-dev#6985](feast-dev#6985)) ([8ce548c](feast-dev@8ce548c)) ### Features * Add --gc-freeze option to feast serve ([feast-dev#7089](feast-dev#7089)) ([aad0c15](feast-dev@aad0c15)) * Add native complex types and scalar index tags to the Milvus online store ([feast-dev#7120](feast-dev#7120)) ([f7068d6](feast-dev@f7068d6)) * Add num_partitions and partition_key_isolation to the Milvus online store ([feast-dev#7083](feast-dev#7083)) ([68de84e](feast-dev@68de84e)), closes [feast-dev#6940](feast-dev#6940) * Add Pinecone online store integration ([feast-dev#6585](feast-dev#6585)) ([7a84f2a](feast-dev@7a84f2a)) * Add PolarisSource and PolarisProvider for Apache Polaris ([feast-dev#7006](feast-dev#7006)) ([386759c](feast-dev@386759c)) * Add row deletion to the Milvus online store ([feast-dev#7082](feast-dev#7082)) ([d3e2041](feast-dev@d3e2041)) * Add upstream FeatureView lineage support for PushSource ([feast-dev#6853](feast-dev#6853)) ([c9b9d72](feast-dev@c9b9d72)) * Optional nameless pre-computed vectors with a schema fingerprint ([feast-dev#7077](feast-dev#7077)) ([9420fac](feast-dev@9420fac)) ### Performance Improvements * Scan Lance sources lazily in the DuckDB offline store ([1299e4e](feast-dev@1299e4e)) Signed-off-by: Shuchu Han <[email protected]>

aborgatin requested a review from a team as a code owner October 29, 2026 09:47

shuchu approved these changes Nov 4, 2026

View reviewed changes

aborgatin force-pushed the feat/serve-gc-freeze branch from 8ce78b0 to bf59320 Compare November 4, 2026 07:14

aborgatin added 2 commits November 5, 2026 11:51

Merge branch 'master' into feat/serve-gc-freeze

4688541

Merge branch 'master' into feat/serve-gc-freeze

cf86e1c

shuchu merged commit aad0c15 into feast-dev:master Nov 5, 2026
28 of 30 checks passed

shuchu pushed a commit to shuchu/feast that referenced this pull request Nov 6, 2026

feat: Add --gc-freeze option to feast serve (feast-dev#7089)

a3d6b5c

Signed-off-by: SHUCHU HAN <[email protected]>

patelchaitany mentioned this pull request Nov 11, 2026

chore: Sync upstream feast-dev/feast stable (0.68.0) opendatahub-io/feast#291

Merged

rhods-devops-app Bot mentioned this pull request Nov 12, 2026

Automated Sync from main to stable red-hat-data-services/feast#630

Open

Navigation Menu

Repository navigation

feat: Add --gc-freeze option to feast serve - #7089

feat: Add --gc-freeze option to feast serve#7089
shuchu merged 3 commits into
feast-dev:masterfrom
aborgatin:feat/serve-gc-freeze

aborgatin commented Oct 29, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented Oct 29, 2026 •

edited

Loading

Uh oh!

shuchu left a comment

Uh oh!

shuchu commented Nov 4, 2026

Uh oh!

aborgatin commented Nov 4, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

aborgatin commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Checks

Testing Strategy

Uh oh!

codecov-commenter commented Oct 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

shuchu left a comment

Choose a reason for hiding this comment

Uh oh!

shuchu commented Nov 4, 2026

Uh oh!

aborgatin commented Nov 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

aborgatin commented Oct 29, 2026 •

edited

Loading

codecov-commenter commented Oct 29, 2026 •

edited

Loading

aborgatin commented Nov 4, 2026 •

edited

Loading