fix(logging): backport upstream PR #28476 — recalc cost after router retry#74
Merged
Merged
Conversation
…router retry Backports BerriAI#28476 (merged 2026-05-22) to ship/v1.83.10. ## Root cause When Router retries an LLM call, every attempt shares one `Logging` instance. `failure_handler` sets `model_call_details["response_cost"] = 0` on each failed attempt (introduced by upstream BerriAI#4604). The success path in `_process_hidden_params_and_response_cost` then short-circuits via a preserve branch (introduced by BerriAI#21844) that treats any non-None existing cost as authoritative — including the stale 0 from the prior failed attempt — so SpendLog records $0 despite a successful response with real usage. ## Fix Tighten the preserve branch with a walrus-bound `!= 0` guard so a stale zero from failure_handler falls through to recalculation. Legitimate zero costs (cache_hit, pass-through handlers writing `result._hidden_params["response_cost"] = 0`) are still preserved by the earlier two branches. ```python elif ( existing_cost := self.model_call_details.get("response_cost") ) is not None and existing_cost != 0: pass # preserve pass-through cost ``` ## Scope of backport Cherry-picked from upstream merge commit b557492. Kept the `litellm_logging.py` fix + the 3 new regression tests in `tests/test_litellm/litellm_core_utils/test_litellm_logging.py`. Dropped the `tests/local_testing/test_amazing_vertex_completion.py` hunk — that's an upstream-CI Google Maps Platform flake skip, unrelated to this fix and not run in our CI. ## Tier C — universal bug fix in `litellm/` core. Already merged upstream (BerriAI#28476), so this is purely a backport for v1.83.10. ## Verification ``` $ pytest tests/test_litellm/litellm_core_utils/test_litellm_logging.py \ -k "process_hidden_params or async_success_handler_preserves" -v ... 4 passed in 0.72s ``` E2E reproduction (Key1 503 → fallback Key2 200) was previously confirmed against this exact mechanism — spend went from 0 to 0.000968 with the fix applied.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tier classification
litellm/coreIf Tier C or D, did you try upstream first?
Relevant issues
Fixes the production-observed bug where Router-retried calls record $0 spend in SpendLog despite a successful response with real usage.
Root cause
When Router retries an LLM call, every attempt shares one
Logginginstance.failure_handlerwritesmodel_call_details["response_cost"] = 0(upstream [Bug/Feature]: Langfuse should be sentinput_cost,output_cost, andtotal_costBerriAI/litellm#4604)._process_hidden_params_and_response_costruns.model_call_details["response_cost"]is not None (it's 0), assumes a pass-through handler already calculated the cost, andpasses — skipping recalculation.Fix
Tighten the preserve branch with a walrus-bound
!= 0guard so a stale zero from failure_handler falls through to recalc. Legitimate zero costs (cache_hit, pass-through handlers writing 0 toresult._hidden_params) are still preserved by the earlier two branches.Scope of backport
Cherry-picked from upstream merge commit
b55749248d. Kept:litellm/litellm_core_utils/litellm_logging.py— the 4-line fixtests/test_litellm/litellm_core_utils/test_litellm_logging.py— 3 new regression tests (140 lines)Dropped the
tests/local_testing/test_amazing_vertex_completion.pyhunk in the upstream commit — that's an upstream-CI Google Maps Platform flake skip, unrelated to this fix and not run in our CI.Pre-Submission checklist
tests/test_litellm/litellm_core_utils/test_litellm_logging.py(3 new regression tests)--target-version py310clean on changed filesScreenshots / Proof of Fix
E2E reproduction (Key1 503 → fallback Key2 200) was previously confirmed against this exact mechanism —
spendwent from0to0.000968with the fix applied (real Gemini call through proxy with fallback routing).New regression tests explicitly cover all three cases:
test_process_hidden_params_recalculates_cost_after_failure_handler_zeroresponse_cost=0via failure_handler, success attempt has usagetest_process_hidden_params_preserves_zero_cost_in_hidden_paramsresponse_cost=0toresult._hidden_paramstest_process_hidden_params_uses_hidden_params_cost_after_failure_handler_zeromodel_call_detailsAND a real value inhidden_paramsConflict resolutions
litellm/litellm_core_utils/litellm_logging.pytests/test_litellm/litellm_core_utils/test_litellm_logging.pytests/local_testing/test_amazing_vertex_completion.py