fix(otel): stamp http.response.status_code on all error responses by ryan-crabbe-berri · Pull Request #28405 · BerriAI/litellm

ryan-crabbe-berri · 2026-05-20T21:57:15Z

Summary

Closes the gaps that left the OTEL SERVER span without http.response.status_code so dashboards couldn't break errors down by status. This PR covers four distinct paths:

Unified 5xx — httpx.HTTPStatusError exposes the upstream status as e.response.status_code, not as a top-level .code / .status_code. get_error_information only checked the top-level attrs, so the SERVER span saw an empty error_code. Added .response.status_code as a third fallback in the same priority chain.
Admin endpoints — async_management_endpoint_{success,failure}_hook only wrote a child span and never stamped or ended the parent SERVER span. The parent stayed open and the exporter never emitted it. Both hooks now stamp the status and end() the parent — the failure hook reuses the same _record_exception_on_span helper the unified path uses, so admin and unified now produce identical error attributes.
Body-parse failures (invalid JSON) on /v1/chat/completions, /v1/messages, /v1/responses — ProxyException raised from _read_request_body bubbled out of user_api_key_auth before the SERVER span was created. The FastAPI exception handler had nothing to close, and the trace never reached the backend at all. Hoist the span creation to a new idempotent _ensure_parent_otel_span_on_request_state helper called at the top of user_api_key_auth, and wire openai_exception_handler to close the dangling span on ProxyException.
/v1/responses success + raw-exception failures — on /v1/responses the SERVER span is ended by _handle_success before async_post_call_success_hook fires, so the hook's set_response_status_code_attribute(200) silently no-op'd against an ended span. Stamp 200 + status OK at the close site via a shared _close_proxy_span_ok helper in _handle_success / _end_proxy_span_from_kwargs. On the failure side, raw exceptions without code/status_code (e.g. a bare TypeError) ended up with empty error_information.error_code — _record_exception_on_span skipped the stamp and the hook ended the span. Default to 500 in async_post_call_failure_hook.

Test plan

Unit tests

tests/test_litellm/integrations/open_telemetry/test_otel_unified_endpoints.py — 29-cell deep matrix on /v1/chat/completions 4xx/5xx (400, 401, 403, 404, 422, 429, 500, 502, 503, 504) + 9-endpoint smoke (including every httpx.HTTPStatusError 5xx case that previously failed) + success path; new test_end_proxy_span_from_kwargs_stamps_200 parametrized over chat/messages/responses; new test_async_post_call_failure_hook_defaults_to_500 for the bare-TypeError path
tests/test_litellm/integrations/open_telemetry/test_otel_passthrough_endpoints.py — 21-cell vertex deep matrix (upstream 4xx/5xx + LiteLLM auth-fail 401) + 6-provider smoke
tests/test_litellm/integrations/open_telemetry/test_otel_admin_endpoints.py — 42-cell /key/generate 4xx/5xx + success + 18-resource smoke (key/team/user/model/customer/org/budget/credentials/mcp/tag)
tests/test_litellm/integrations/open_telemetry/test_otel_exception_handler.py — closes-the-dangling-span tests, new test_openai_exception_handler_closes_span parametrized over (400, /v1/chat/completions), (400, /v1/messages), (400, /v1/responses), (429, ...), (503, ...)
All OTEL + management-helper suites green (uv run pytest tests/test_litellm/integrations/open_telemetry/ — 109 passed); no regressions
Every cell asserts the four attributes the dashboards depend on: http.response.status_code (int), url.path, http.route, and a non-zero duration

End-to-end verification against Jaeger

Stood up jaeger v2.18.0 (all-in-one) on localhost, pointed the proxy at it via OTEL_EXPORTER=otlp_http OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4318, and exercised a 36-cell live matrix against the running proxy. Used deterministic traceparent headers per scenario so each trace ID could be looked up directly via the Jaeger query API. Each assertion checks: exactly one SERVER span in the trace, http.response.status_code matches the HTTP response code (and is an int), and otel.status_code is OK for 2xx / ERROR for 4xx/5xx.

Result: 36/36 traces correct — every triggered status code lands on the SERVER span with the right value and span status.

Live status-code matrix

Status	Trigger	`/v1/chat/completions`	`/v1/messages`	`/v1/responses`
200	mock-gpt success	✓	✓	✓
400	invalid JSON body	✓	✓	✓
400	unknown model (`definitely-not-a-real-model`)	✓	✓	✓
400	`mock_response: "litellm.ContextWindowExceededError"`	✓	✓	n/a*
401	bad bearer token	✓	✓	✓
403	restricted key requesting a non-allowed model	✓	✓	✓
404	upstream `api_base` route mismatch (httpbin)	✓	✓	✓
429	`mock_response: "litellm.RateLimitError"`	✓	✓	n/a*
500	`mock_response: "litellm.InternalServerError"`	✓	✓	n/a*
500	missing required body field (raw `TypeError`)	n/a	n/a	✓
422	Pydantic body type error on `/key/generate` (`models="string"`)	—	—	— (admin: ✓)
500	`/key/generate` with invalid `duration` (handler exception)	—	—	— (admin: ✓)

* /v1/responses doesn't honor the mock_response exception-string conventions used by chat/completions and messages — it returns the string as plain content (HTTP 200). Independent of this PR; the responses-specific 500 case is exercised by the missing-input TypeError path.

502/503/504 not triggered live — the httpbin approach didn't work because the OpenAI SDK appends /chat/completions to api_base, turning https://httpbin.org/status/502 into a 404. These codes ARE covered by unit tests via make_httpx_status_error(...) in test_otel_unified_endpoints.py (chat 502/503/504), test_otel_passthrough_endpoints.py (vertex 502 upstream + 502 wrapped), and the smoke matrix (502 across 9 endpoints).

Resolves LIT-3193

httpx.HTTPStatusError exposes status under .response.status_code, not as a top-level attr, so unified-endpoint 5xx failures left the SERVER span without a status. The admin hooks only wrote a child span and never stamped or ended the parent at all, so admin 4xx/5xx (and success) responses were invisible to dashboards. Adds a fallback to .response.status_code in get_error_information, and ends the parent SERVER span in async_management_endpoint_{success,failure}_hook with the same _record_exception_on_span helper the unified path uses. Resolves LIT-3193

codecov · 2026-05-20T22:01:14Z

Codecov Report

❌ Patch coverage is 70.21277% with 28 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/auth/user_api_key_auth.py	44.00%	14 Missing ⚠️
litellm/integrations/opentelemetry.py	73.33%	8 Missing ⚠️
litellm/proxy/management_helpers/utils.py	0.00%	4 Missing ⚠️
litellm/proxy/proxy_server.py	92.85%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

greptile-apps · 2026-05-20T22:01:53Z

Greptile Summary

This PR closes two gaps that prevented the OTEL SERVER span from receiving http.response.status_code on error responses: the httpx.HTTPStatusError status-code fallback in get_error_information, and the management endpoint hooks now stamping and ending the parent span. A new _close_dangling_otel_server_span helper plus two @app.exception_handler registrations cover requests that fail before route handlers run (e.g. RequestValidationError, unexpected TypeError).

get_error_information now falls back to .response.status_code so httpx.HTTPStatusError 4xx/5xx upstream responses produce correct error_code values.
async_management_endpoint_failure_hook and async_management_endpoint_success_hook both call parent_otel_span.end(), and the failure wrapper in utils.py was refactored with an else branch so the hook fires even when http_request is absent.
New exception handlers (RequestValidationError, generic Exception) close any span stored in request.state.parent_otel_span, with the parent_otel_span now written to request.state in user_api_key_auth.

Confidence Score: 5/5

Safe to merge; the core instrumentation fixes are correct and well-tested, with only non-blocking quality gaps remaining.

The failure-path fixes (httpx fallback, management wrapper refactor, new exception handlers) are logically sound and backed by a thorough mock-only test matrix. The success-path closure for management endpoints is incomplete for endpoints that omit http_request: Request from their signatures, and the unhandled-exception response format silently changes — but neither causes wrong data, crashes, or auth bypasses on the critical request path.

litellm/proxy/management_helpers/utils.py (success-hook gating) and litellm/proxy/proxy_server.py (new unhandled-exception response format)

Important Files Changed

Filename	Overview
litellm/integrations/opentelemetry.py	Adds parent span stamping and end() to both management success and failure hooks; failure hook correctly reuses _record_exception_on_span and sets StatusCode.ERROR
litellm/litellm_core_utils/litellm_logging.py	Adds .response.status_code fallback in get_error_information to handle httpx.HTTPStatusError; logic is correct and doesn't affect non-httpx exceptions
litellm/proxy/auth/user_api_key_auth.py	Stores parent_otel_span on request.state so exception handlers can close dangling spans; only done when open_telemetry_logger is not None, so safe when OTEL is disabled
litellm/proxy/management_helpers/utils.py	Failure hook correctly moved outside the http_request guard via else fallback; success hook remains gated on http_request, so most management endpoints (without Request param) still don't close the parent span on success
litellm/proxy/proxy_server.py	Adds _close_dangling_otel_server_span helper plus two new exception handlers; otel_unhandled_exception_handler changes the error response format for truly unhandled exceptions
litellm/proxy/management_endpoints/key_management_endpoints.py	Adds @management_endpoint_wrapper to info_key_fn; endpoint lacks http_request: Request param so success-path hook won't be called from the wrapper
tests/test_litellm/integrations/open_telemetry/test_otel_admin_endpoints.py	New test file; failure matrix is thorough including httpx.HTTPStatusError cases; success test validates the hook in isolation (not the full management wrapper path)
tests/test_litellm/integrations/open_telemetry/test_otel_unified_endpoints.py	New test file; well-structured matrix for unified inference endpoint failures including all httpx.HTTPStatusError variants; mocks only, no real network calls
tests/test_litellm/integrations/open_telemetry/test_otel_passthrough_endpoints.py	New test file; covers passthrough failure paths for Vertex and 6 provider smoke endpoints; uses mock-only approach, correct
tests/test_litellm/integrations/open_telemetry/test_otel_exception_handler.py	New test file; validates _close_dangling_otel_server_span directly and tests both new exception handler functions; comprehensive edge case coverage
tests/test_litellm/integrations/open_telemetry/_helpers.py	New shared helpers; assert_server_span_attrs checks all four required OTEL attributes including span status code; make_httpx_status_error correctly builds real httpx objects
tests/test_litellm/integrations/open_telemetry/conftest.py	New shared fixtures; uses InMemorySpanExporter with real TracerProvider and SimpleSpanProcessor; server_span_factory correctly mirrors the user_api_key_auth setup

_{Reviews (3): Last reviewed commit: "fix(otel): close SERVER span on body-val..." | Re-trigger Greptile}

Pins the contract that get_error_information's response.status_code fallback is reachable from any entry point — without this, a future refactor that bypasses _record_exception_on_span in the admin hooks could regress for httpx-wrapped exceptions while the unified suite still passes.

Tighten docstrings and remove redundant section dividers/inline narration. Behavior is unchanged.

ryan-crabbe-berri · 2026-05-20T22:10:34Z

@greptileai re review

Mirror the unified failure path: stamp StatusCode.ERROR on the parent SERVER span before recording the exception, and StatusCode.OK before ending it on success. Without this, OTEL backends filtering on span status (the idiomatic primitive) miss admin-endpoint failures even though the http.response.status_code attribute is correct. Extend assert_server_span_attrs to assert span.status.status_code matches the expected outcome so the gap can't regress.

Stash the SERVER span on request.state in auth so FastAPI exception handlers can finish it for failures that occur after auth but before the route handler (e.g. /model/new TypeError, /key/generate RequestValidationError). Without this, those requests left dangling spans missing http.response.status_code. Resolves LIT-3193

ryan-crabbe-berri · 2026-05-21T00:39:57Z

@greptileai re review

veria-ai · 2026-05-21T00:48:11Z

PR overview

OpenTelemetry error response status stamping

This PR adds and closes proxy SERVER spans across error paths, management endpoints, request validation failures, and unhandled exceptions so response status codes are reflected in OTEL. I checked the changed auth initialization, exception handlers, management wrapper behavior, and error metadata extraction and did not find a security regression in the diff.

Security review

No new security issues were flagged in the latest review.
No review issues remain open on this pull request.

Risk: 2/10

Don't leak str(exc) and type(exc).__name__ to clients on uncaught exceptions. The full traceback is logged via verbose_proxy_logger and the SERVER span still gets http.response.status_code=500. Resolves LIT-3193

Closes three remaining gaps where the proxy SERVER span ended without the http.response.status_code attribute: 1. ProxyException raised from _read_request_body (e.g. invalid JSON body) bubbled out of user_api_key_auth before the SERVER span was created, so the FastAPI handler had nothing to close and the trace never reached the backend. Hoist the span creation to a new idempotent _ensure_parent_otel_span_on_request_state helper called at the top of user_api_key_auth; wire openai_exception_handler to close the dangling span. Covers /v1/chat/completions, /v1/messages, /v1/responses (shared handler). 2. /v1/responses success — _handle_success ends the proxy span before async_post_call_success_hook fires on this path, so the hook's set_response_status_code_attribute(200) silently no-op'd against an ended span. Stamp 200 + set OK status at the close site in _handle_success / _end_proxy_span_from_kwargs via a shared _close_proxy_span_ok helper, so the attribute lands regardless of which success hook runs first. 3. Failure path for exceptions without code/status_code (e.g. a bare TypeError surfacing through _handle_llm_api_exception) — empty error_information.error_code → _record_exception_on_span skips the stamp → the hook ends the span. Default to 500 in async_post_call_failure_hook so the attribute is always set. Resolves LIT-3193

…rriAI#28405) * fix(otel): stamp http.response.status_code on all error responses httpx.HTTPStatusError exposes status under .response.status_code, not as a top-level attr, so unified-endpoint 5xx failures left the SERVER span without a status. The admin hooks only wrote a child span and never stamped or ended the parent at all, so admin 4xx/5xx (and success) responses were invisible to dashboards. Adds a fallback to .response.status_code in get_error_information, and ends the parent SERVER span in async_management_endpoint_{success,failure}_hook with the same _record_exception_on_span helper the unified path uses. Resolves LIT-3193 * test(otel): exercise httpx.HTTPStatusError through admin path Pins the contract that get_error_information's response.status_code fallback is reachable from any entry point — without this, a future refactor that bypasses _record_exception_on_span in the admin hooks could regress for httpx-wrapped exceptions while the unified suite still passes. * chore(otel): trim verbose comments in LIT-3193 changes Tighten docstrings and remove redundant section dividers/inline narration. Behavior is unchanged. * fix(otel): set span.status on management hook parent SERVER span Mirror the unified failure path: stamp StatusCode.ERROR on the parent SERVER span before recording the exception, and StatusCode.OK before ending it on success. Without this, OTEL backends filtering on span status (the idiomatic primitive) miss admin-endpoint failures even though the http.response.status_code attribute is correct. Extend assert_server_span_attrs to assert span.status.status_code matches the expected outcome so the gap can't regress. * fix(otel): close SERVER span on body-validation and unhandled errors Stash the SERVER span on request.state in auth so FastAPI exception handlers can finish it for failures that occur after auth but before the route handler (e.g. /model/new TypeError, /key/generate RequestValidationError). Without this, those requests left dangling spans missing http.response.status_code. Resolves LIT-3193 * fix(otel): generic 500 body, log exception details server-side Don't leak str(exc) and type(exc).__name__ to clients on uncaught exceptions. The full traceback is logged via verbose_proxy_logger and the SERVER span still gets http.response.status_code=500. Resolves LIT-3193 * fix(otel): stamp http.response.status_code on every SERVER span path Closes three remaining gaps where the proxy SERVER span ended without the http.response.status_code attribute: 1. ProxyException raised from _read_request_body (e.g. invalid JSON body) bubbled out of user_api_key_auth before the SERVER span was created, so the FastAPI handler had nothing to close and the trace never reached the backend. Hoist the span creation to a new idempotent _ensure_parent_otel_span_on_request_state helper called at the top of user_api_key_auth; wire openai_exception_handler to close the dangling span. Covers /v1/chat/completions, /v1/messages, /v1/responses (shared handler). 2. /v1/responses success — _handle_success ends the proxy span before async_post_call_success_hook fires on this path, so the hook's set_response_status_code_attribute(200) silently no-op'd against an ended span. Stamp 200 + set OK status at the close site in _handle_success / _end_proxy_span_from_kwargs via a shared _close_proxy_span_ok helper, so the attribute lands regardless of which success hook runs first. 3. Failure path for exceptions without code/status_code (e.g. a bare TypeError surfacing through _handle_llm_api_exception) — empty error_information.error_code → _record_exception_on_span skips the stamp → the hook ends the span. Default to 500 in async_post_call_failure_hook so the attribute is always set. Resolves LIT-3193

ryan-crabbe-berri added 2 commits May 20, 2026 15:04

chore(otel): trim verbose comments in LIT-3193 changes

1a0382f

Tighten docstrings and remove redundant section dividers/inline narration. Behavior is unchanged.

greptile-apps Bot reviewed May 20, 2026

View reviewed changes

Comment thread litellm/integrations/opentelemetry.py

ryan-crabbe-berri added 2 commits May 20, 2026 16:40

veria-ai Bot reviewed May 21, 2026

View reviewed changes

Comment thread litellm/proxy/proxy_server.py Outdated

ryan-crabbe-berri added 2 commits May 20, 2026 18:03

fix(otel): generic 500 body, log exception details server-side

560bacb

Don't leak str(exc) and type(exc).__name__ to clients on uncaught exceptions. The full traceback is logged via verbose_proxy_logger and the SERVER span still gets http.response.status_code=500. Resolves LIT-3193

yassin-berriai approved these changes May 23, 2026

View reviewed changes

yuneng-berri enabled auto-merge (squash) May 23, 2026 19:17

yuneng-berri merged commit 886e91b into litellm_internal_staging May 23, 2026
102 of 116 checks passed

This was referenced May 25, 2026

fix(otel): export SERVER span on management-endpoint success without http_request #28792

Closed

fix(otel): export SERVER span on management-endpoint success without http_request #28794

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(otel): stamp http.response.status_code on all error responses#28405

fix(otel): stamp http.response.status_code on all error responses#28405
yuneng-berri merged 7 commits into
litellm_internal_stagingfrom
litellm_otel_http_status_all_errors

ryan-crabbe-berri commented May 20, 2026 •

edited

Loading

Uh oh!

codecov Bot commented May 20, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented May 20, 2026 •

edited

Loading

Important Files Changed

Uh oh!

ryan-crabbe-berri commented May 20, 2026

Uh oh!

Uh oh!

ryan-crabbe-berri commented May 21, 2026

Uh oh!

veria-ai Bot commented May 21, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

ryan-crabbe-berri commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Test plan

Unit tests

End-to-end verification against Jaeger

Live status-code matrix

Uh oh!

codecov Bot commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

greptile-apps Bot commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

ryan-crabbe-berri commented May 20, 2026

Uh oh!

Uh oh!

ryan-crabbe-berri commented May 21, 2026

Uh oh!

veria-ai Bot commented May 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR overview

OpenTelemetry error response status stamping

Security review

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ryan-crabbe-berri commented May 20, 2026 •

edited

Loading

codecov Bot commented May 20, 2026 •

edited

Loading

greptile-apps Bot commented May 20, 2026 •

edited

Loading

veria-ai Bot commented May 21, 2026 •

edited

Loading