Feature server: long GC pauses cause latency spikes on online reads #7088

Problem

With feast serve (gunicorn + UvicornWorker), about 0.3% of /get-online-features requests take 50–100 ms, while p50 is 7 ms. The online store read timer shows the same spikes, so it looks like a slow Redis read. Redis is fine.

The cause is the Python cyclic GC. After startup a worker holds about 185k tracked objects: the registry, feature definitions, protobuf classes and modules. They live until the process exits, but every gen 2 collection walks all of them. On our setup one gen 2 collection takes 50–75 ms and frees nothing. It runs on the event loop thread, so every request waiting on the loop at that moment gets slower, including the timed Redis await.

Proposal

Add an opt-in feast serve --gc-freeze flag (and feature_server.gc_freeze in feature_store.yaml). It runs gc.collect() and gc.freeze() in the gunicorn master after the app and registry are loaded and before the workers fork. Workers inherit the frozen state, including workers restarted by --max-requests, so there is no extra cost per worker. Off by default.

This is a common fix for pre-fork servers (Instagram, Locust, vLLM freeze_gc_heap, Solara gc_freeze). Gunicorn has no built-in option (benoitc/gunicorn#1640).

I have a PR ready.

Results

Local Docker setup from the repro steps below: one worker, 15 req/s for 10 min, one entity per request, 4 feature views with 137 features.

Each case was run twice with the same result. p50 and worker memory stay the same.

Repro

1. Create a feature repo with a Redis online store, one entity and 4 feature views with about 140 Float32 features. Write rows for 20k entities.
2. Start feast serve --max-requests 100000 (one worker).
3. Send requests for one random entity at 15 req/s with random gaps (open loop) for 10 min and record the client latency.
4. Count the requests over 50 ms, then repeat with feast serve --gc-freeze.

I can share my Docker setup and load script if that helps.

Related

#7072 also saw this on the client side: gc.freeze() took p50 from 63 to 34 ms there. This issue is only about the feature server.