feat(datadog): emit litellm.overhead.latency as a standalone Datadog metric#28548
Conversation
[Infra] Promote internal staging to main
[Infra] Promote internal staging to main
chore(ci): promote internal staging to main
…metric Adds a new `litellm.overhead.latency` gauge metric to `DatadogMetricsLogger` (the `/api/v2/series` path). The value is sourced from `hidden_params["litellm_overhead_time_ms"]` already computed in `ResponseMetadata` and exposed in `StandardLoggingPayload`. Matches the Prometheus integration which exposes the same value via `litellm_overhead_latency_metric`. Emitted in seconds (ms ÷ 1000) for consistency with the other latency series. Co-Authored-By: Claude Sonnet 4.6 <[email protected]>
|
Shin seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
Greptile SummaryThis PR adds
Confidence Score: 4/5The change is additive and gated on the presence of a single field in hidden_params; existing metrics are unaffected and the new metric is safe to merge. The implementation is correct and well-tested. The only notable issue is that Only
|
| Filename | Overview |
|---|---|
| litellm/integrations/datadog/datadog_metrics.py | Adds litellm.overhead.latency gauge metric by reading hidden_params["litellm_overhead_time_ms"] from the StandardLoggingPayload; logic is correct but _extract_tags is called a second time unnecessarily when a simple filter on the already-computed tags list would suffice. |
| tests/test_litellm/integrations/datadog/test_datadog_metrics.py | Adds two focused unit tests for the new overhead latency metric (presence and absence paths), uses mocks only (no real network calls), and correctly updates the existing test comment. Coverage is solid. |
Reviews (1): Last reviewed commit: "feat(datadog): emit litellm.overhead.lat..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| # 3. LiteLLM Overhead Latency Metric (total - llm_api time) | ||
| hidden_params = log.get("hidden_params", {}) or {} | ||
| litellm_overhead_time_ms = hidden_params.get("litellm_overhead_time_ms") | ||
| if litellm_overhead_time_ms is not None: | ||
| overhead_tags = self._extract_tags(log) # no status_code on latency metric |
There was a problem hiding this comment.
_extract_tags a second time, but the result is simply tags (computed at line 117) minus the status_code entry. This creates both an unnecessary second traversal of the payload and a subtle maintenance risk: any future tag added in _extract_tags will be picked up by the two existing latency series but may be silently missing from the overhead series if developers forget to consider this secondary call site. Building overhead_tags by filtering the already-computed tags list is cheaper and keeps both derivations in sync.
| # 3. LiteLLM Overhead Latency Metric (total - llm_api time) | |
| hidden_params = log.get("hidden_params", {}) or {} | |
| litellm_overhead_time_ms = hidden_params.get("litellm_overhead_time_ms") | |
| if litellm_overhead_time_ms is not None: | |
| overhead_tags = self._extract_tags(log) # no status_code on latency metric | |
| # 3. LiteLLM Overhead Latency Metric (total - llm_api time) | |
| hidden_params = log.get("hidden_params", {}) or {} | |
| litellm_overhead_time_ms = hidden_params.get("litellm_overhead_time_ms") | |
| if litellm_overhead_time_ms is not None: | |
| # Reuse the already-computed tags, dropping status_code (overhead is a | |
| # pure latency signal, not tied to a request outcome). | |
| overhead_tags = [t for t in tags if not t.startswith("status_code:")] |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
ishaan-berri
merged commit e892e8f
into
BerriAI:litellm_shin_may21_2
5 participants
Summary
Closes LIT-3203 — emit LiteLLM overhead as a Datadog metric.
litellm_overhead_time_ms(= total request time − LLM-API time) was already:ResponseMetadataand stored inhidden_paramsStandardLoggingPayload.hidden_paramslitellm_overhead_latency_metricWhat was missing: a standalone time-series metric sent to the Datadog Metrics API (
/api/v2/series) — the same path that already carrieslitellm.request.total_latencyandlitellm.llm_api.latency.Changes
litellm/integrations/datadog/datadog_metrics.py_add_metrics_from_log()now readshidden_params["litellm_overhead_time_ms"]from theStandardLoggingPayloadlitellm.overhead.latency(value in seconds, consistent with other latency metrics) is appended to the queuestatus_codeis intentionally omitted (it's a latency metric, not a request-count)tests/test_litellm/integrations/datadog/test_datadog_metrics.pytest_overhead_latency_metric_emitted— verifies the metric is emitted with the correct value (250 ms → 0.25 s) and correct tagstest_overhead_latency_metric_absent_when_no_hidden_params— verifies no metric is emitted whenhidden_paramshas no overhead valuetest_add_metrics_from_logcomment to clarify why it still expects exactly 3 seriesTest plan
All 12 tests pass:
🤖 Generated with Claude Code