Skip to content

Cherry-pick #29311, #29343, #29358, #27913, and #29447 onto stable/1.85.x#29458

Closed
yuneng-berri wants to merge 5 commits into
stable/1.85.xfrom
worktree-cherrypick-1-85-x
Closed

Cherry-pick #29311, #29343, #29358, #27913, and #29447 onto stable/1.85.x#29458
yuneng-berri wants to merge 5 commits into
stable/1.85.xfrom
worktree-cherrypick-1-85-x

Conversation

@yuneng-berri

@yuneng-berri yuneng-berri commented Jun 1, 2026

Copy link
Copy Markdown
Collaborator

Relevant issues

Backport of three PRs that landed in v1.84.4 (via #29352 and #29363), plus two more (#27913 and #29447) that don't apply to the 1.84.x line but do apply to 1.85.x. All five carry into the next 1.85 patch.

Linear ticket

N/A

Pre-Submission checklist

  • I have added meaningful tests (carried over from the source PRs)
  • My PR passes all unit tests on make test-unit
  • My PR's scope is as isolated as possible; it only solves 1 specific problem
  • I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

CI (LiteLLM team)

  • Branch creation CI run
    Link:

  • CI run for the last commit
    Link:

  • Merge / cherry-pick CI run
    Links:

Screenshots / Proof of Fix

Five cherry-picks from litellm_internal_staging onto stable/1.85.x:

Commit on this branch PR Source on staging Also in v1.84.4?
fcdf0231d0 #29089 / #29311 (fix: duplicate claude code traces) 581c30f1e8 yes
87b0e47485 #29343 (refactor(proxy/auth): normalize Bearer prefix in safe-hash helper) 94a043efb2 yes
a06ec43b36 #29358 (fix(reset_budget): write only {spend, budget_reset_at} and stop pre-zeroing counter) 7d1bd9d9f4 yes
0e2510dee2 #27913 (fix(rate-limit): stop v3 limiter from leaking internal stash to provider body) 7a462a4220 no, doesn't apply to 1.84.x
acbbfe9cae #29447 (fix: stop use_chat_completions_api flag from leaking into provider request body) 65b6e04da6 no, doesn't apply to 1.84.x

One conflict during the #29311 cherry-pick in litellm/proxy/pass_through_endpoints/streaming_handler.py: stable/1.85.x carried the older await litellm_logging_obj.async_success_handler(...) call that the PR rewrites. Resolution keeps the surrounding endpoint-dispatch if/elif/elif chain (untouched by the PR) and replaces only the trailing async_success_handler call with dispatch_success_handlers plus the explanatory comment the PR adds. diff v1.84.4:litellm/proxy/pass_through_endpoints/streaming_handler.py HEAD:litellm/proxy/pass_through_endpoints/streaming_handler.py returns empty for the patched section, confirming the resolution matches v1.84.4 exactly.

The remaining four cherry-picks applied with auto-merges only, no manual conflict resolution.

Type

🐛 Bug Fix

Changes

Three fixes already shipped in v1.84.4 plus two that only apply to the 1.85 line.

#29089 / #29311 removes a duplicate Claude Code trace emission caused by the pass-through streaming handler invoking the success-handler path twice. #29343 normalizes a leading Bearer prefix (case-insensitive, per RFC 7235) before classifying an API key as sk- or JWT in UserAPIKeyAuth._safe_hash_litellm_api_key, so the helper produces consistent hashed output regardless of whether the caller stripped the prefix. #29358 switches ResetBudgetJob's batched key/user/team update to a per-row narrow {spend, budget_reset_at} write and removes the pre-zeroing of the cross-pod spend counter, so a single bad row no longer detonates the batch and a failed reset no longer leaves a budget-bypass window.

#27913 stops the v3 rate-limiter from leaking its internal TPM/RPM-reservation stash (_litellm_rate_limit_descriptors, _litellm_tpm_reserved_*) onto the provider request body. #29447 stops the use_chat_completions_api control flag from leaking onto the provider request body via the OpenAI bridge. Both surface as upstream HTTP 400s for unknown fields; both were absent from v1.84.x (different code paths) so the v1.84.4 cherry-pick set didn't include them.

First cherry-pick PR against the new stable/1.85.x line branch (created from v1.85.2 earlier today).

mateo-berri and others added 3 commits June 1, 2026 16:22
…9343)

* refactor(proxy/auth): normalize Bearer prefix in safe-hash helper

UserAPIKeyAuth._safe_hash_litellm_api_key now strips a leading
"Bearer "/"bearer " prefix before its existing sk-/JWT classification, so
the helper produces the same hashed output regardless of whether the
caller stripped the Authorization header prefix or passed the header
value through unchanged.

* refactor(proxy/auth): make Bearer-prefix strip case-insensitive

Per RFC 7235 the HTTP authorization scheme token is case-insensitive.
Replace the two-prefix loop with a single case-insensitive check so the
helper normalizes "Bearer ", "bearer ", "BEARER ", and any mixed-case
variant before classifying the remainder as sk- or JWT. The contract
test gains coverage of "BEARER " and "BeArEr ".

* test(mcp): align auth-handler test expectations with safe-hash helper

The two MCP auth tests asserted that UserAPIKeyAuth(api_key="Bearer ...")
retained the raw header bytes on the api_key field. _safe_hash_litellm_api_key
now normalizes that input — stripping the Bearer prefix and hashing the
resulting sk- key — so the expectations move to the normalized form:
the bare token in the parametrize case, and hash_token("sk-...") in the
backward-compat assertion. This matches what the real auth flow produces
(the builder strips Bearer and the DB stores the hashed token), so the
mocks now line up with production rather than with the un-normalized
validator output.
…eroing counter (#29358)

* fix(reset_budget): write only {spend, budget_reset_at} and stop pre-zeroing counter

ResetBudgetJob's batched update_data path shipped the full key/user/team
model on each reset. Prisma rejects object_permission_id and budget_limits
on the update input type, so any row carrying those fields detonated the
entire batch -- spend never reset, budget_reset_at never advanced. After
v1.84.0 started populating object_permission_id on UI-created keys, this
fires routinely.

_reset_budget_common also zeroed the cross-pod spend counter before the
DB write, so failed resets left enforcement reading 0 from the counter
while the DB still held the over-budget spend, admitting requests past
the cap until the counter naturally re-saturated from new reservations.

Switch the write to per-row narrow updates ({spend, budget_reset_at})
via db.batch_, and move the counter invalidation out of
_reset_budget_common so it only fires after the DB write commits. On
DB-write failure the counter is left untouched, enforcement continues
to block, and the next scheduler tick can retry without leaving a
bypass window.

Fixes #27730.

* fix(reset_budget): address Greptile review on #29358

- Strengthen the bypass-half regression test: replace the for-loop over
  call_args_list (vacuously true when empty) with assert_not_called(),
  so the test would actually flag a re-introduction of counter-zeroing
  via any code path.
- Add the same explanatory docstring on _write_user_reset_updates and
  _write_team_reset_updates that _write_key_reset_updates already has,
  so all three helpers point future maintainers at #27730.

* test(reset_budget): update test_proxy_budget_reset for new batch-write path

Same shape as the previous test_reset_budget_job.py update: keys/users/teams
now write through prisma.db.batch_().<table>.update, not update_data, so the
tests need a batcher mock and updated assertions. Adds:

- _wire_batcher_for_test helper that returns a list which accumulates per-row
  batch updates captured from prisma_client.db.batch_().
- _attrify helper that wraps dict fixtures so getattr(item, "token") works
  alongside the dict item-access the fake_reset_* mocks rely on. The new
  narrow-write helpers use getattr to pull out the row's id, and would
  silently skip plain dicts otherwise.
- Updates 3 partial_failure tests to assert against the batch-call list
  (rows by id, payload contains only {spend, budget_reset_at}) instead of
  update_data.assert_awaited_once + data_list inspection.
- Updates test_reset_budget_continues_other_categories_on_failure: only
  budget + enduser still flow through update_data; key/user/team go through
  the batch path now.
- Wires the batcher mock into 3 service_logger_*_success tests so commit()
  is actually awaitable and the success hook fires.

These tests were silently passing locally only because the editable install
in .venv pointed at the main repo, not the worktree — running pytest with
PYTHONPATH overridden to the worktree (matching CI) reproduces the failures.
@yuneng-berri yuneng-berri requested a review from a team June 1, 2026 23:24
@greptile-apps

greptile-apps Bot commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This cherry-pick backports three bug fixes from v1.84.4 onto the stable/1.85.x branch: duplicate Claude Code trace elimination, Bearer-prefix normalization in the auth-key hashing helper, and a budget-reset job narrowed to write only {spend, budget_reset_at} per row (avoiding the Prisma DataError that aborted entire batches on rows with object_permission_id/budget_limits). All three changes come with their original test coverage carried over and adapted for the 1.85.x baseline.

Confidence Score: 4/5

Safe to merge; all three backported fixes are well-bounded and come with corresponding tests. The dedup-flag-on-error trade-off and the batch-commit all-or-nothing behaviour are observable but not regressions relative to the 1.85.x baseline.

Three self-contained fixes with good test coverage, clean conflict resolution, and no new dependencies. The only open questions are cosmetic (returned value for unknown-format Bearer tokens) or operational (batch-failure observability) rather than correctness issues.

litellm/proxy/common_utils/reset_budget_job.py (batch commit failure path, all-or-nothing rollback) and litellm/litellm_core_utils/litellm_logging.py (has_dispatched_final_stream_success set-then-forget on handler error)

Important Files Changed

Filename Overview
litellm/litellm_core_utils/litellm_logging.py Adds dispatch_success_handlers, _is_assembled_stream_success, and _is_sync_litellm_request to centralize success-logging dispatch and add a dedup guard for final assembled streams; also modifies async_success_handler to bypass should_run_logging for assembled-stream responses
litellm/proxy/_types.py _safe_hash_litellm_api_key now strips "Bearer " prefix (case-insensitive) before classifying sk- or JWT keys, normalizing hashed output regardless of caller stripping behaviour; well-covered by updated tests
litellm/proxy/common_utils/reset_budget_job.py Adds write_key/user/team_reset_updates helpers that write only {spend, budget_reset_at} via Prisma batch(); removes pre-zeroing of Redis counter from _reset_budget_common; counter invalidation now happens in callers after commit, preventing a bypass window on DB write failure
litellm/proxy/pass_through_endpoints/streaming_handler.py Replaces direct async_success_handler + executor.submit pair with dispatch_success_handlers(prefer_async_handlers=True); resolves duplicate trace emission for Claude Code pass-through streaming
litellm/proxy/pass_through_endpoints/success_handler.py _handle_logging migrated from calling both success_handler and async_success_handler separately to dispatch_success_handlers(prefer_async_handlers=True); removes duplicate thread_pool_executor submission
litellm/proxy/common_request_processing.py Deferred-stream guardrail callback and orphaned-stream path both updated to use dispatch_success_handlers; removes duplicate executor.submit calls and simplifies error handling into a single exception block
tests/proxy_unit_tests/test_proxy_reject_logging.py Adds _register_proxy_test_logger to wire test logger into all callback lists; changes embedding test model from text-embedding-ada-002 to fake-model — embedding model change warrants a brief note

Comments Outside Diff (2)

  1. litellm/proxy/common_utils/reset_budget_job.py, line 340-349 (link)

    P2 Batch commit failure abandons all rows — no partial-success path

    _write_key_reset_updates (and the mirrored user/team variants) commits all per-row updates in a single Prisma transaction. If batcher.commit() raises for any reason (e.g., a transient DB error unrelated to field types), every key in the batch is left un-reset — budgets stay enforced correctly, but none of the _invalidate_spend_counter calls that follow the await ever run. The outer except block logs the failure, so it is visible, but operators have no signal distinguishing "Prisma DataError on a specific row" from "transient network hiccup that affected the whole batch." Adding a brief exception log inside _write_*_reset_updates before re-raising would aid triage.

  2. tests/proxy_unit_tests/test_proxy_reject_logging.py, line 1069-1072 (link)

    P2 Embedding model changed from text-embedding-ada-002 to fake-model — why?

    The parametrized embedding test case previously used "text-embedding-ada-002". Changing it to "fake-model" is presumably needed because the test router is not configured with an ada deployment and the validation now catches model mismatches more strictly. Could you confirm: is this purely a test-infrastructure fix, or is it masking a regression where embedding requests with real model names are now rejected differently?

    Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time! Is the model name change from "text-embedding-ada-002" to "fake-model" purely to match the mock router's configuration, or does it reflect a change in how the proxy validates model names on this path?

Reviews (1): Last reviewed commit: "fix(reset_budget): write only {spend, bu..." | Re-trigger Greptile

Comment on lines 2562 to +2566
print_verbose(
"Logging Details LiteLLM-Async Success Call, cache_hit={}".format(cache_hit)
)
if not self.should_run_logging(
if not self._is_assembled_stream_success(
result

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Assembled-stream bypass makes has_dispatched_final_stream_success flag permanent on error

dispatch_success_handlers sets has_dispatched_final_stream_success = True before awaiting async_success_handler. If async_success_handler raises, the flag is already committed to model_call_details, so any subsequent retry call to dispatch_success_handlers will return early — silently swallowing the final stream log rather than retrying. This is an acceptable trade-off for most cases, but worth documenting: once the flag is set, the success log is considered dispatched even on failure. A comment noting this intentional "set-then-forget" pattern would clarify the design intent for future readers.

Comment thread litellm/proxy/_types.py
Comment on lines +2716 to +2725
normalized = api_key
if normalized[:7].lower() == "bearer ":
normalized = normalized[7:]
if normalized.startswith("sk-"):
return hash_token(normalized)
from litellm.proxy.auth.handle_jwt import JWTHandler

if JWTHandler.is_jwt(token=api_key):
return f"hashed-jwt-{hash_token(token=api_key)}"
return api_key
if JWTHandler.is_jwt(token=normalized):
return f"hashed-jwt-{hash_token(token=normalized)}"
return normalized

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Unknown-format keys now return the stripped value, not the original

For keys that do not match sk- or JWT patterns but carry a Bearer prefix (e.g., "Bearer opaque-api-key"), the function previously returned "Bearer opaque-api-key" unchanged; it now returns "opaque-api-key". Any downstream consumer that stored or compared the old value will see a different string. In logging/audit contexts this is cosmetic, but any cache key or equality check built on the returned value would silently stop matching after the upgrade.

mateo-berri and others added 2 commits June 1, 2026 16:44
…der body (#27913)

* fix(rate-limit): stop v3 limiter from leaking internal stash to provider body

PR #27001 (atomic TPM rate limit) introduced a reservation flow that
writes four LiteLLM-internal keys onto the request data dict:

  _litellm_rate_limit_descriptors
  _litellm_tpm_reserved_tokens
  _litellm_tpm_reserved_model
  _litellm_tpm_reserved_scopes
  _litellm_tpm_reservation_released

These keys are forwarded as request body params to the upstream provider,
which rejects them as unknown fields:

  OpenAI    -> 400 'Unknown parameter: _litellm_rate_limit_descriptors'
              (mapped by litellm to RateLimitError / 429, hiding the bug
               behind a misleading 'throttling_error' code)
  Anthropic -> 400 '_litellm_rate_limit_descriptors: Extra inputs are
               not permitted'

Net effect: every chat completion against any real provider fails the
moment a virtual key has any tpm_limit / rpm_limit set — i.e. v3-enforced
key-level TPM/RPM limits are broken end-to-end. The v3 RPM/TPM check
itself still runs (raises 429 on over-limit), but the success path
poisons the upstream body.

Reproduced on litellm_internal_staging HEAD (410ce76) against
gpt-4o-mini and claude-haiku-4-5 with a 1-RPM/1-TPM key — first request
fails with the provider's unknown-field error.

Fix: the stash is metadata only.

  - Add RATE_LIMIT_DESCRIPTORS_KEY constant and a _LITELLM_STASH_KEYS
    registry so we have a single source of truth for stash keys.
  - New helper _stash_value_in_metadata_channels writes to
    data['metadata'] / data['litellm_metadata'] without touching the
    top level.
  - _stash_reservation_in_data and the descriptor stash now route
    through that helper. _mark_reservation_released stops writing
    top-level.
  - _lookup_stashed_value also checks kwargs['metadata'] /
    kwargs['litellm_metadata'] (raw request_data shape) in addition to
    kwargs['litellm_params']['metadata'] (completion kwargs shape).
  - async_post_call_failure_hook now reads descriptors via the unified
    metadata lookup instead of request_data.get(top-level).
  - Defense in depth: async_pre_call_hook strips any stash key that
    somehow surfaced at the top level (stale cache, future refactor,
    test fixture) before returning.

Tests:
  - New regression test asserts no _litellm_* stash key is present at
    the top level of data after async_pre_call_hook, and that the
    metadata channel still carries the reservation + descriptors so
    success / failure reconciliation works.
  - Existing test_tpm_concurrent.py tests that asserted top-level
    presence are updated to read from data['metadata'] — the location
    is an implementation detail; the spec is that post-call callbacks
    can resolve the stash.

Verified end-to-end against OpenAI gpt-4o-mini and Anthropic
claude-haiku-4-5 via /v1/chat/completions on a low-rpm key:

  - With limits not exceeded: HTTP 200, valid completion response,
    no leaked fields in body.
  - With RPM exceeded: HTTP 429 from v3 enforcement
    ('Rate limit exceeded ... Limit type: requests').
  - With TPM exceeded: HTTP 429 from v3 enforcement
    ('Rate limit exceeded ... Limit type: tokens').

Full v3 hook test suite passes (171 tests).

Co-authored-by: Mateo Wang <[email protected]>

* chore(rate-limit): use RATE_LIMIT_DESCRIPTORS_KEY constant in test, trim noisy comments

Address greptile P2: test fixture now uses the imported constant.
Drop comments that re-explain what well-named identifiers already convey.

* fix(rate-limit): reject caller-supplied stash values to prevent TPM-refund abuse

Strip _LITELLM_STASH_KEYS from data top-level and both metadata channels at
the start of async_pre_call_hook. Without this, an authenticated caller can
inject _litellm_rate_limit_descriptors plus _litellm_tpm_reserved_tokens in
body metadata, trigger a proxy-side rejection, and cause
async_post_call_failure_hook to refund TPM counters against attacker-named
scopes (e.g. another tenant's api_key).

---------

Co-authored-by: Cursor Agent <[email protected]>
Co-authored-by: Mateo Wang <[email protected]>
…quest body (#29447)

* fix: stop use_chat_completions_api flag from leaking into provider request body

use_chat_completions_api is a LiteLLM control flag that forces the
/responses -> /chat/completions bridge. It was missing from
all_litellm_params, so get_non_default_completion_params treated it as a
model-specific param and forwarded it to the upstream provider. A
model-level "use_chat_completions_api: true" in the proxy config therefore
reached the chat-completions path and was rejected by strict providers
(OpenAI/Anthropic) with HTTP 400 for an unknown body field.

Register it as a known internal param so it is stripped on every path
(completion, the responses bridge that calls litellm.completion, and
filter_out_litellm_params).

Adds a regression test driving litellm.completion() with a mocked OpenAI
client that asserts the flag never reaches the request body.

* test: clarify extra_body assertion in use_chat_completions_api leak test

Replace the misleading 'not in ... or {}' precedence idiom with an explicit
parenthesized guard that also handles extra_body being None.
@yuneng-berri yuneng-berri changed the title Cherry-pick #29311, #29343, and #29358 onto stable/1.85.x Cherry-pick #29311, #29343, #29358, #27913, and #29447 onto stable/1.85.x Jun 1, 2026
@yuneng-berri yuneng-berri deleted the worktree-cherrypick-1-85-x branch June 2, 2026 00:15
@yuneng-berri

Copy link
Copy Markdown
Collaborator Author

Superseded by #29460 — the original branch name didn't match the litellm_* prefix needed for CI to run. Same five cherry-picks, same body, new branch litellm_cherrypick_1_85_x.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants