Cherry-pick #29311, #29343, #29358, #27913, and #29447 onto stable/1.85.x#29458
Cherry-pick #29311, #29343, #29358, #27913, and #29447 onto stable/1.85.x#29458yuneng-berri wants to merge 5 commits into
Conversation
…9343) * refactor(proxy/auth): normalize Bearer prefix in safe-hash helper UserAPIKeyAuth._safe_hash_litellm_api_key now strips a leading "Bearer "/"bearer " prefix before its existing sk-/JWT classification, so the helper produces the same hashed output regardless of whether the caller stripped the Authorization header prefix or passed the header value through unchanged. * refactor(proxy/auth): make Bearer-prefix strip case-insensitive Per RFC 7235 the HTTP authorization scheme token is case-insensitive. Replace the two-prefix loop with a single case-insensitive check so the helper normalizes "Bearer ", "bearer ", "BEARER ", and any mixed-case variant before classifying the remainder as sk- or JWT. The contract test gains coverage of "BEARER " and "BeArEr ". * test(mcp): align auth-handler test expectations with safe-hash helper The two MCP auth tests asserted that UserAPIKeyAuth(api_key="Bearer ...") retained the raw header bytes on the api_key field. _safe_hash_litellm_api_key now normalizes that input — stripping the Bearer prefix and hashing the resulting sk- key — so the expectations move to the normalized form: the bare token in the parametrize case, and hash_token("sk-...") in the backward-compat assertion. This matches what the real auth flow produces (the builder strips Bearer and the DB stores the hashed token), so the mocks now line up with production rather than with the un-normalized validator output.
…eroing counter (#29358) * fix(reset_budget): write only {spend, budget_reset_at} and stop pre-zeroing counter ResetBudgetJob's batched update_data path shipped the full key/user/team model on each reset. Prisma rejects object_permission_id and budget_limits on the update input type, so any row carrying those fields detonated the entire batch -- spend never reset, budget_reset_at never advanced. After v1.84.0 started populating object_permission_id on UI-created keys, this fires routinely. _reset_budget_common also zeroed the cross-pod spend counter before the DB write, so failed resets left enforcement reading 0 from the counter while the DB still held the over-budget spend, admitting requests past the cap until the counter naturally re-saturated from new reservations. Switch the write to per-row narrow updates ({spend, budget_reset_at}) via db.batch_, and move the counter invalidation out of _reset_budget_common so it only fires after the DB write commits. On DB-write failure the counter is left untouched, enforcement continues to block, and the next scheduler tick can retry without leaving a bypass window. Fixes #27730. * fix(reset_budget): address Greptile review on #29358 - Strengthen the bypass-half regression test: replace the for-loop over call_args_list (vacuously true when empty) with assert_not_called(), so the test would actually flag a re-introduction of counter-zeroing via any code path. - Add the same explanatory docstring on _write_user_reset_updates and _write_team_reset_updates that _write_key_reset_updates already has, so all three helpers point future maintainers at #27730. * test(reset_budget): update test_proxy_budget_reset for new batch-write path Same shape as the previous test_reset_budget_job.py update: keys/users/teams now write through prisma.db.batch_().<table>.update, not update_data, so the tests need a batcher mock and updated assertions. Adds: - _wire_batcher_for_test helper that returns a list which accumulates per-row batch updates captured from prisma_client.db.batch_(). - _attrify helper that wraps dict fixtures so getattr(item, "token") works alongside the dict item-access the fake_reset_* mocks rely on. The new narrow-write helpers use getattr to pull out the row's id, and would silently skip plain dicts otherwise. - Updates 3 partial_failure tests to assert against the batch-call list (rows by id, payload contains only {spend, budget_reset_at}) instead of update_data.assert_awaited_once + data_list inspection. - Updates test_reset_budget_continues_other_categories_on_failure: only budget + enduser still flow through update_data; key/user/team go through the batch path now. - Wires the batcher mock into 3 service_logger_*_success tests so commit() is actually awaitable and the success hook fires. These tests were silently passing locally only because the editable install in .venv pointed at the main repo, not the worktree — running pytest with PYTHONPATH overridden to the worktree (matching CI) reproduces the failures.
Greptile SummaryThis cherry-pick backports three bug fixes from v1.84.4 onto the
Confidence Score: 4/5Safe to merge; all three backported fixes are well-bounded and come with corresponding tests. The dedup-flag-on-error trade-off and the batch-commit all-or-nothing behaviour are observable but not regressions relative to the 1.85.x baseline. Three self-contained fixes with good test coverage, clean conflict resolution, and no new dependencies. The only open questions are cosmetic (returned value for unknown-format Bearer tokens) or operational (batch-failure observability) rather than correctness issues. litellm/proxy/common_utils/reset_budget_job.py (batch commit failure path, all-or-nothing rollback) and litellm/litellm_core_utils/litellm_logging.py (has_dispatched_final_stream_success set-then-forget on handler error)
|
| Filename | Overview |
|---|---|
| litellm/litellm_core_utils/litellm_logging.py | Adds dispatch_success_handlers, _is_assembled_stream_success, and _is_sync_litellm_request to centralize success-logging dispatch and add a dedup guard for final assembled streams; also modifies async_success_handler to bypass should_run_logging for assembled-stream responses |
| litellm/proxy/_types.py | _safe_hash_litellm_api_key now strips "Bearer " prefix (case-insensitive) before classifying sk- or JWT keys, normalizing hashed output regardless of caller stripping behaviour; well-covered by updated tests |
| litellm/proxy/common_utils/reset_budget_job.py | Adds write_key/user/team_reset_updates helpers that write only {spend, budget_reset_at} via Prisma batch(); removes pre-zeroing of Redis counter from _reset_budget_common; counter invalidation now happens in callers after commit, preventing a bypass window on DB write failure |
| litellm/proxy/pass_through_endpoints/streaming_handler.py | Replaces direct async_success_handler + executor.submit pair with dispatch_success_handlers(prefer_async_handlers=True); resolves duplicate trace emission for Claude Code pass-through streaming |
| litellm/proxy/pass_through_endpoints/success_handler.py | _handle_logging migrated from calling both success_handler and async_success_handler separately to dispatch_success_handlers(prefer_async_handlers=True); removes duplicate thread_pool_executor submission |
| litellm/proxy/common_request_processing.py | Deferred-stream guardrail callback and orphaned-stream path both updated to use dispatch_success_handlers; removes duplicate executor.submit calls and simplifies error handling into a single exception block |
| tests/proxy_unit_tests/test_proxy_reject_logging.py | Adds _register_proxy_test_logger to wire test logger into all callback lists; changes embedding test model from text-embedding-ada-002 to fake-model — embedding model change warrants a brief note |
Comments Outside Diff (2)
-
litellm/proxy/common_utils/reset_budget_job.py, line 340-349 (link)Batch commit failure abandons all rows — no partial-success path
_write_key_reset_updates(and the mirrored user/team variants) commits all per-row updates in a single Prisma transaction. Ifbatcher.commit()raises for any reason (e.g., a transient DB error unrelated to field types), every key in the batch is left un-reset — budgets stay enforced correctly, but none of the_invalidate_spend_countercalls that follow theawaitever run. The outerexceptblock logs the failure, so it is visible, but operators have no signal distinguishing "Prisma DataError on a specific row" from "transient network hiccup that affected the whole batch." Adding a brief exception log inside_write_*_reset_updatesbefore re-raising would aid triage. -
tests/proxy_unit_tests/test_proxy_reject_logging.py, line 1069-1072 (link)Embedding model changed from
text-embedding-ada-002tofake-model— why?The parametrized embedding test case previously used
"text-embedding-ada-002". Changing it to"fake-model"is presumably needed because the test router is not configured with anadadeployment and the validation now catches model mismatches more strictly. Could you confirm: is this purely a test-infrastructure fix, or is it masking a regression where embedding requests with real model names are now rejected differently?Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time! Is the model name change from "text-embedding-ada-002" to "fake-model" purely to match the mock router's configuration, or does it reflect a change in how the proxy validates model names on this path?
Reviews (1): Last reviewed commit: "fix(reset_budget): write only {spend, bu..." | Re-trigger Greptile
| print_verbose( | ||
| "Logging Details LiteLLM-Async Success Call, cache_hit={}".format(cache_hit) | ||
| ) | ||
| if not self.should_run_logging( | ||
| if not self._is_assembled_stream_success( | ||
| result |
There was a problem hiding this comment.
Assembled-stream bypass makes
has_dispatched_final_stream_success flag permanent on error
dispatch_success_handlers sets has_dispatched_final_stream_success = True before awaiting async_success_handler. If async_success_handler raises, the flag is already committed to model_call_details, so any subsequent retry call to dispatch_success_handlers will return early — silently swallowing the final stream log rather than retrying. This is an acceptable trade-off for most cases, but worth documenting: once the flag is set, the success log is considered dispatched even on failure. A comment noting this intentional "set-then-forget" pattern would clarify the design intent for future readers.
| normalized = api_key | ||
| if normalized[:7].lower() == "bearer ": | ||
| normalized = normalized[7:] | ||
| if normalized.startswith("sk-"): | ||
| return hash_token(normalized) | ||
| from litellm.proxy.auth.handle_jwt import JWTHandler | ||
|
|
||
| if JWTHandler.is_jwt(token=api_key): | ||
| return f"hashed-jwt-{hash_token(token=api_key)}" | ||
| return api_key | ||
| if JWTHandler.is_jwt(token=normalized): | ||
| return f"hashed-jwt-{hash_token(token=normalized)}" | ||
| return normalized |
There was a problem hiding this comment.
Unknown-format keys now return the stripped value, not the original
For keys that do not match sk- or JWT patterns but carry a Bearer prefix (e.g., "Bearer opaque-api-key"), the function previously returned "Bearer opaque-api-key" unchanged; it now returns "opaque-api-key". Any downstream consumer that stored or compared the old value will see a different string. In logging/audit contexts this is cosmetic, but any cache key or equality check built on the returned value would silently stop matching after the upgrade.
…der body (#27913) * fix(rate-limit): stop v3 limiter from leaking internal stash to provider body PR #27001 (atomic TPM rate limit) introduced a reservation flow that writes four LiteLLM-internal keys onto the request data dict: _litellm_rate_limit_descriptors _litellm_tpm_reserved_tokens _litellm_tpm_reserved_model _litellm_tpm_reserved_scopes _litellm_tpm_reservation_released These keys are forwarded as request body params to the upstream provider, which rejects them as unknown fields: OpenAI -> 400 'Unknown parameter: _litellm_rate_limit_descriptors' (mapped by litellm to RateLimitError / 429, hiding the bug behind a misleading 'throttling_error' code) Anthropic -> 400 '_litellm_rate_limit_descriptors: Extra inputs are not permitted' Net effect: every chat completion against any real provider fails the moment a virtual key has any tpm_limit / rpm_limit set — i.e. v3-enforced key-level TPM/RPM limits are broken end-to-end. The v3 RPM/TPM check itself still runs (raises 429 on over-limit), but the success path poisons the upstream body. Reproduced on litellm_internal_staging HEAD (410ce76) against gpt-4o-mini and claude-haiku-4-5 with a 1-RPM/1-TPM key — first request fails with the provider's unknown-field error. Fix: the stash is metadata only. - Add RATE_LIMIT_DESCRIPTORS_KEY constant and a _LITELLM_STASH_KEYS registry so we have a single source of truth for stash keys. - New helper _stash_value_in_metadata_channels writes to data['metadata'] / data['litellm_metadata'] without touching the top level. - _stash_reservation_in_data and the descriptor stash now route through that helper. _mark_reservation_released stops writing top-level. - _lookup_stashed_value also checks kwargs['metadata'] / kwargs['litellm_metadata'] (raw request_data shape) in addition to kwargs['litellm_params']['metadata'] (completion kwargs shape). - async_post_call_failure_hook now reads descriptors via the unified metadata lookup instead of request_data.get(top-level). - Defense in depth: async_pre_call_hook strips any stash key that somehow surfaced at the top level (stale cache, future refactor, test fixture) before returning. Tests: - New regression test asserts no _litellm_* stash key is present at the top level of data after async_pre_call_hook, and that the metadata channel still carries the reservation + descriptors so success / failure reconciliation works. - Existing test_tpm_concurrent.py tests that asserted top-level presence are updated to read from data['metadata'] — the location is an implementation detail; the spec is that post-call callbacks can resolve the stash. Verified end-to-end against OpenAI gpt-4o-mini and Anthropic claude-haiku-4-5 via /v1/chat/completions on a low-rpm key: - With limits not exceeded: HTTP 200, valid completion response, no leaked fields in body. - With RPM exceeded: HTTP 429 from v3 enforcement ('Rate limit exceeded ... Limit type: requests'). - With TPM exceeded: HTTP 429 from v3 enforcement ('Rate limit exceeded ... Limit type: tokens'). Full v3 hook test suite passes (171 tests). Co-authored-by: Mateo Wang <[email protected]> * chore(rate-limit): use RATE_LIMIT_DESCRIPTORS_KEY constant in test, trim noisy comments Address greptile P2: test fixture now uses the imported constant. Drop comments that re-explain what well-named identifiers already convey. * fix(rate-limit): reject caller-supplied stash values to prevent TPM-refund abuse Strip _LITELLM_STASH_KEYS from data top-level and both metadata channels at the start of async_pre_call_hook. Without this, an authenticated caller can inject _litellm_rate_limit_descriptors plus _litellm_tpm_reserved_tokens in body metadata, trigger a proxy-side rejection, and cause async_post_call_failure_hook to refund TPM counters against attacker-named scopes (e.g. another tenant's api_key). --------- Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: Mateo Wang <[email protected]>
…quest body (#29447) * fix: stop use_chat_completions_api flag from leaking into provider request body use_chat_completions_api is a LiteLLM control flag that forces the /responses -> /chat/completions bridge. It was missing from all_litellm_params, so get_non_default_completion_params treated it as a model-specific param and forwarded it to the upstream provider. A model-level "use_chat_completions_api: true" in the proxy config therefore reached the chat-completions path and was rejected by strict providers (OpenAI/Anthropic) with HTTP 400 for an unknown body field. Register it as a known internal param so it is stripped on every path (completion, the responses bridge that calls litellm.completion, and filter_out_litellm_params). Adds a regression test driving litellm.completion() with a mocked OpenAI client that asserts the flag never reaches the request body. * test: clarify extra_body assertion in use_chat_completions_api leak test Replace the misleading 'not in ... or {}' precedence idiom with an explicit parenthesized guard that also handles extra_body being None.
|
Superseded by #29460 — the original branch name didn't match the |
Relevant issues
Backport of three PRs that landed in v1.84.4 (via #29352 and #29363), plus two more (#27913 and #29447) that don't apply to the 1.84.x line but do apply to 1.85.x. All five carry into the next 1.85 patch.
Linear ticket
N/A
Pre-Submission checklist
make test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewCI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Screenshots / Proof of Fix
Five cherry-picks from
litellm_internal_stagingontostable/1.85.x:fcdf0231d0581c30f1e887b0e4748594a043efb2a06ec43b367d1bd9d9f40e2510dee27a462a4220acbbfe9cae65b6e04da6One conflict during the #29311 cherry-pick in
litellm/proxy/pass_through_endpoints/streaming_handler.py:stable/1.85.xcarried the olderawait litellm_logging_obj.async_success_handler(...)call that the PR rewrites. Resolution keeps the surrounding endpoint-dispatchif/elif/elifchain (untouched by the PR) and replaces only the trailingasync_success_handlercall withdispatch_success_handlersplus the explanatory comment the PR adds.diff v1.84.4:litellm/proxy/pass_through_endpoints/streaming_handler.py HEAD:litellm/proxy/pass_through_endpoints/streaming_handler.pyreturns empty for the patched section, confirming the resolution matches v1.84.4 exactly.The remaining four cherry-picks applied with auto-merges only, no manual conflict resolution.
Type
🐛 Bug Fix
Changes
Three fixes already shipped in v1.84.4 plus two that only apply to the 1.85 line.
#29089 / #29311 removes a duplicate Claude Code trace emission caused by the pass-through streaming handler invoking the success-handler path twice. #29343 normalizes a leading
Bearerprefix (case-insensitive, per RFC 7235) before classifying an API key as sk- or JWT inUserAPIKeyAuth._safe_hash_litellm_api_key, so the helper produces consistent hashed output regardless of whether the caller stripped the prefix. #29358 switchesResetBudgetJob's batched key/user/team update to a per-row narrow{spend, budget_reset_at}write and removes the pre-zeroing of the cross-pod spend counter, so a single bad row no longer detonates the batch and a failed reset no longer leaves a budget-bypass window.#27913 stops the v3 rate-limiter from leaking its internal TPM/RPM-reservation stash (
_litellm_rate_limit_descriptors,_litellm_tpm_reserved_*) onto the provider request body. #29447 stops theuse_chat_completions_apicontrol flag from leaking onto the provider request body via the OpenAI bridge. Both surface as upstream HTTP 400s for unknown fields; both were absent from v1.84.x (different code paths) so the v1.84.4 cherry-pick set didn't include them.First cherry-pick PR against the new
stable/1.85.xline branch (created from v1.85.2 earlier today).