Skip to content

Commit f236f76

Browse files
NopeCharonntkathole
authored andcommitted
feat: Add HTTP status class label to feature server request metrics
feast_feature_server_request_total was labelled with endpoint and a status that was only ever "success" or "error", and the status came from whether the handler raised. Client errors and server errors fell into the same bucket, and handlers that returned an error response without raising (a 404 or 422 from the vector store search, a 409 when a materialization is already running) were counted as successes. The counter now carries a status_class label (2xx, 4xx, 5xx) taken from the HTTP status code the server responds with: - When the handler raises, the status code is derived the same way the REST exception handling answers: FeastError subclasses use their own http_status_code(), HTTPException uses its status_code, and anything else is a 500. - RequestMetricsContext gains a status_code attribute (default 200) that handlers set when they return a non-2xx response. The vector store search and the async /materialize and /materialize-incremental conflict paths now do so. The existing status label is kept for dashboards and alerts that use it, and is now derived from the status code as well: "success" below 400 and "error" otherwise. The metrics reference and the performance tuning guide document the new label. Fixes feast-dev#6641 Signed-off-by: NopeCharon <[email protected]>
1 parent 839e51e commit f236f76

5 files changed

Lines changed: 222 additions & 25 deletions

File tree

‎docs/how-to-guides/online-server-performance-tuning.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -829,7 +829,7 @@ This exposes a `/metrics` endpoint on port 8000. If the Prometheus Operator's `S
829829
| `feast_feature_server_transformation_duration_seconds` | ODFV execution time (requires `track_metrics=True` on the ODFV). |
830830
| `feast_feature_server_cpu_usage` | Per-worker CPU. Sustained > 80% means you need more workers or replicas. |
831831
| `feast_feature_server_memory_usage` | Per-worker memory. Growing over time may indicate a memory leak — use `maxRequests` to recycle. |
832-
| `feast_feature_server_request_total` | Request throughput and error rates by endpoint and status. |
832+
| `feast_feature_server_request_total` | Request throughput and error rates by endpoint and status. Filter on `status_class="5xx"` to alert on server errors only. |
833833
| `feast_feature_freshness_seconds` | Seconds since last materialization per feature view. Alerts you to stale data. |
834834

835835
#### Example Prometheus alert rules

‎docs/reference/feature-servers/python-feature-server.md‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -368,7 +368,7 @@ thread from starting). All categories default to `true` except
368368
|--------|------|--------|----------|-------------|
369369
| `feast_feature_server_cpu_usage` | Gauge | — | `resource` | Process CPU usage % |
370370
| `feast_feature_server_memory_usage` | Gauge | — | `resource` | Process memory usage % |
371-
| `feast_feature_server_request_total` | Counter | `endpoint`, `status` | `request` | Total requests per endpoint |
371+
| `feast_feature_server_request_total` | Counter | `endpoint`, `status`, `status_class` | `request` | Total requests per endpoint and response status (see below) |
372372
| `feast_feature_server_request_latency_seconds` | Histogram | `endpoint`, `feature_count`, `feature_view_count` | `request` | Request latency with p50/p95/p99 support (`feature_count` is bucketed, see below) |
373373
| `feast_online_features_request_total` | Counter | — | `online_features` | Total online feature retrieval requests |
374374
| `feast_online_features_entity_count` | Histogram | — | `online_features` | Entity rows per online feature request |
@@ -405,6 +405,25 @@ This produces the labels `0`, `1-5`, `6-20`, and `21+`. Note that this only
405405
affects the Prometheus label; the `feature_count` field in audit logs (see
406406
below) always reports the exact count.
407407

408+
### Request status labels
409+
410+
`feast_feature_server_request_total` carries two status labels derived from
411+
the HTTP status code the server responds with:
412+
413+
- `status_class` is the status code class: `2xx`, `4xx` or `5xx`. Use it to
414+
tell client errors (an unknown feature view or feature service, a denied
415+
permission, a materialization that is already running) apart from server
416+
errors.
417+
- `status` is `success` for status codes below 400 and `error` otherwise.
418+
419+
For example, the server-side error rate per endpoint, excluding client
420+
errors:
421+
422+
```
423+
sum by (endpoint) (rate(feast_feature_server_request_total{status_class="5xx"}[5m]))
424+
/ sum by (endpoint) (rate(feast_feature_server_request_total[5m]))
425+
```
426+
408427
### Per-ODFV transformation metrics
409428

410429
The `transformation_duration_seconds` and `write_transformation_duration_seconds`

‎sdk/python/feast/feature_server.py‎

Lines changed: 9 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -827,7 +827,7 @@ async def vector_store_search(
827827
) -> JSONResponse:
828828
with feast_metrics.track_request_latency(
829829
"/v1/vector_stores/{vector_store_id}/search"
830-
):
830+
) as metrics_ctx:
831831
try:
832832
feature_view = vs_registry.resolve(vector_store_id)
833833
assert_permissions(
@@ -854,6 +854,7 @@ async def vector_store_search(
854854
),
855855
)
856856
except FeatureViewNotFoundException:
857+
metrics_ctx.status_code = 404
857858
return JSONResponse(
858859
status_code=404,
859860
content={
@@ -864,6 +865,7 @@ async def vector_store_search(
864865
},
865866
)
866867
except ValueError as e:
868+
metrics_ctx.status_code = 422
867869
return JSONResponse(
868870
status_code=422,
869871
content={
@@ -1027,7 +1029,7 @@ async def materialize(
10271029
async_mode: bool = Query(False, alias="async"),
10281030
force: bool = Query(False),
10291031
):
1030-
with feast_metrics.track_request_latency("/materialize"):
1032+
with feast_metrics.track_request_latency("/materialize") as metrics_ctx:
10311033
fv_names = _authorize_materialize_views(
10321034
store, request.feature_views, version=request.version
10331035
)
@@ -1039,6 +1041,7 @@ async def materialize(
10391041
else:
10401042
conflict = _check_already_materializing(store, fv_names)
10411043
if conflict:
1044+
metrics_ctx.status_code = conflict.status_code
10421045
return conflict
10431046

10441047
# Reserve MATERIALIZING before 202 so concurrent requests hit 409.
@@ -1086,7 +1089,9 @@ async def materialize_incremental(
10861089
async_mode: bool = Query(False, alias="async"),
10871090
force: bool = Query(False),
10881091
):
1089-
with feast_metrics.track_request_latency("/materialize-incremental"):
1092+
with feast_metrics.track_request_latency(
1093+
"/materialize-incremental"
1094+
) as metrics_ctx:
10901095
fv_names = _authorize_materialize_views(
10911096
store, request.feature_views, version=request.version
10921097
)
@@ -1098,6 +1103,7 @@ async def materialize_incremental(
10981103
else:
10991104
conflict = _check_already_materializing(store, fv_names)
11001105
if conflict:
1106+
metrics_ctx.status_code = conflict.status_code
11011107
return conflict
11021108

11031109
_update_fv_state(store, fv_names, FeatureViewState.MATERIALIZING)

‎sdk/python/feast/metrics.py‎

Lines changed: 54 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -189,7 +189,7 @@ def build_metrics_flags(metrics_config: Optional[object] = None) -> _MetricsFlag
189189
request_count = Counter(
190190
"feast_feature_server_request_total",
191191
"Total number of requests to the Feast feature server",
192-
["endpoint", "status"],
192+
["endpoint", "status", "status_class"],
193193
)
194194
request_latency = Histogram(
195195
"feast_feature_server_request_latency_seconds",
@@ -313,13 +313,47 @@ class RequestMetricsContext:
313313
(e.g. ``/get-online-features`` where the feature count is only
314314
known after ``_get_features`` succeeds) can set the attributes
315315
on the yielded object and they will be picked up in ``finally``.
316+
317+
``status_code`` is the HTTP status code of the response. It defaults
318+
to ``200`` and is derived from the exception when the block raises,
319+
so handlers only need to set it when they *return* a non-2xx
320+
response (e.g. a 404 or 409 ``JSONResponse``).
316321
"""
317322

318-
__slots__ = ("feature_count", "feature_view_count")
323+
__slots__ = ("feature_count", "feature_view_count", "status_code")
319324

320-
def __init__(self, feature_count: str = "", feature_view_count: str = ""):
325+
def __init__(
326+
self,
327+
feature_count: str = "",
328+
feature_view_count: str = "",
329+
status_code: int = 200,
330+
):
321331
self.feature_count = feature_count
322332
self.feature_view_count = feature_view_count
333+
self.status_code = status_code
334+
335+
336+
def _status_code_for_exception(exc: Exception) -> int:
337+
"""Return the HTTP status code the feature server responds with for *exc*.
338+
339+
Mirrors the REST exception handling: ``FeastError`` subclasses carry
340+
their own status code, ``HTTPException`` is answered with its status
341+
code, and anything else becomes a 500.
342+
"""
343+
from starlette.exceptions import HTTPException
344+
345+
from feast.errors import FeastError
346+
347+
if isinstance(exc, FeastError):
348+
return exc.http_status_code()
349+
if isinstance(exc, HTTPException):
350+
return exc.status_code
351+
return 500
352+
353+
354+
def _status_class(status_code: int) -> str:
355+
"""Map an HTTP status code to its class label, e.g. ``404`` -> ``"4xx"``."""
356+
return f"{status_code // 100}xx"
323357

324358

325359
@contextmanager
@@ -328,11 +362,16 @@ def track_request_latency(
328362
):
329363
"""Context manager that records endpoint latency and increments request count.
330364
331-
Yields a :class:`RequestMetricsContext` whose ``feature_count`` and
332-
``feature_view_count`` attributes can be updated inside the block.
333-
The final values are used when recording the histogram and counter
334-
in ``finally``, so labels are accurate even when they depend on
335-
work done inside the block.
365+
Yields a :class:`RequestMetricsContext` whose ``feature_count``,
366+
``feature_view_count`` and ``status_code`` attributes can be updated
367+
inside the block. The final values are used when recording the
368+
histogram and counter in ``finally``, so labels are accurate even
369+
when they depend on work done inside the block.
370+
371+
The request counter is labelled with ``status`` (``success`` for
372+
status codes below 400, ``error`` otherwise) and ``status_class``
373+
(``2xx``, ``4xx``, ``5xx``, ...), so client errors can be told
374+
apart from server errors.
336375
337376
Gated by the ``request`` category flag.
338377
"""
@@ -342,11 +381,10 @@ def track_request_latency(
342381
return
343382

344383
start = time.monotonic()
345-
status_label = "success"
346384
try:
347385
yield ctx
348-
except Exception:
349-
status_label = "error"
386+
except Exception as exc:
387+
ctx.status_code = _status_code_for_exception(exc)
350388
raise
351389
finally:
352390
elapsed = time.monotonic() - start
@@ -355,7 +393,11 @@ def track_request_latency(
355393
feature_count=ctx.feature_count,
356394
feature_view_count=ctx.feature_view_count,
357395
).observe(elapsed)
358-
request_count.labels(endpoint=endpoint, status=status_label).inc()
396+
request_count.labels(
397+
endpoint=endpoint,
398+
status="success" if ctx.status_code < 400 else "error",
399+
status_class=_status_class(ctx.status_code),
400+
).inc()
359401

360402

361403
def track_online_features_entities(entity_count: int):

0 commit comments

Comments
 (0)