…olSpec (#29814)

* feat(bedrock): forward strict and additionalProperties to Converse toolSpec

Bedrock Converse supports strict in toolSpec since 2026-02, but
_bedrock_tools_pt only whitelisted type/properties/required/name/description,
so strict: true was silently dropped and Claude-on-Bedrock ignored enum
constraints that GPT and direct-Anthropic honored. Forward strict from the
OpenAI function and additionalProperties from the schema (Bedrock requires
the latter alongside strict), passing each only when present.

https://claude.ai/code/session_01WQjWd8NfUB3vxERwudbHkv

* fix(bedrock): only forward strict tool schemas to Claude on Converse

Nova, Llama and GPT-OSS on Bedrock reject the strict field
(BedrockException 'This model doesn't support the strict field'), and the
GPT-OSS request-body test asserts strict/additionalProperties are stripped.
Forwarding them to every model broke the llm_translation suite, so gate the
forwarding on the anthropic base model since only Claude honours strict
tool schemas on Bedrock.

…per-user env vars (#29856)

* fix(mcp): flag missing per-user env vars on the card for every accessible server

The dashboard MCP card grid lists servers via the registry-backed manager
(get_all_mcp_servers_unfiltered for admins in view_all mode, the allowed-context
aggregation otherwise), but the per-user env-var status endpoint that drives the
red "user fields missing" highlight resolved servers through the much narrower
get_all_mcp_servers_for_user, which only returns servers explicitly granted on
the calling key. An admin's dashboard session key carries no per-server MCP
grant, so the status feed came back empty and the card never turned red even
when the logged-in user had not filled in their required variables.

Both surfaces now share a single _resolve_accessible_mcp_servers helper, so the
status feed is computed over exactly the cards the user sees. The helper returns
servers unredacted; the status endpoint needs the raw env_vars and still only
ever reports is_set booleans, never the stored secret values.

* test(mcp): drop dead get_all_mcp_servers_for_user patch from view_all regression test

The bulk status endpoint resolves servers through _resolve_accessible_mcp_servers
now, so the old get_all_mcp_servers_for_user patch in the admin view_all
regression test is never hit. Removing it keeps the test honest about which code
path it exercises.

* feat(ui): add budget duration to edit team member form

Editing a team member created a member budget with no duration, so the
budget never reset. This threads a budget reset period through the edit
flow end to end and reuses the shared duration dropdown so the options
stay in sync with the rest of the UI.

Resolves LIT-2651

* fix(proxy): validate member budget_duration and persist clears

Reject budget_duration values that can't be parsed, are non-positive, or overflow date math before any write, so a bad value can't be persisted and later crash the budget reset job.

Clearing the budget duration in the edit-member form now sends null and clears the column end to end, so the dropdown's clear control reflects a real change instead of being a no-op

* chore(ui): regenerate schema.d.ts for member budget_duration

Adds budget_duration to TeamMemberUpdateRequest/Response in the generated dashboard types so the Check UI API Types Sync gate passes

The Workflow Runs page rendered its table at roughly a quarter of the
available width. Its root container is a flex child of the dashboard
content row but set only padding, min-height and background, so with no
width it shrank to the table's natural content size. Sibling pages
(logs, memory) fill the area with a full-width root; mirror that by
setting width 100% on the container.

Fixes LIT-3636

…odel, and llm_provider fields (#27687)

* feat(exceptions): add RateLimitErrorCategory + headers/detail fields on RateLimitError

LiteLLM previously surfaced rate-limit conditions through several unrelated
error classes (RateLimitError, FastAPI HTTPException(429), BaseLLMException).
This commit adds the data model needed to consolidate them under a single
class:

* RateLimitErrorCategory enum exposing four categorical values
  (vendor_rate_limit, vendor_batch_rate_limit, litellm_rate_limit,
  litellm_batch_rate_limit) so callers can switch on the rate-limit source.
* New optional fields on RateLimitError:
  - category (defaults to vendor_rate_limit, preserving today's behavior for
    every existing call site in exception_mapping_utils);
  - headers (preserves retry-after / rate_limit_type / reset_at across the
    proxy boundary instead of dropping them on the floor);
  - detail (mirrors FastAPI HTTPException.detail so the same instance can be
    serialized through both paths).

litellm.RateLimitErrorCategory is re-exported at the package root to match
the existing exception-export pattern.

LIT-2968

Co-authored-by: Mateo Wang <[email protected]>

* feat(proxy): add ProxyRateLimitError unifying RateLimitError + HTTPException

Adds a single proxy-side error class that subclasses BOTH
litellm.exceptions.RateLimitError AND fastapi.HTTPException via cooperative
multiple inheritance.

Why both bases:
* Subclassing RateLimitError lets user code catch every rate-limit source
  with one 'except RateLimitError' and switch on the new .category field.
* Subclassing HTTPException keeps every existing FastAPI plumbing path (the
  isinstance(e, HTTPException) branches in proxy_server.py route handlers,
  FastAPI's own dispatcher, and tests asserting pytest.raises(HTTPException))
  working without modification, and preserves retry-after / rate_limit_type /
  reset_at headers on the wire.

The class declaration order is (HTTPException, RateLimitError) so the MRO
puts HTTPException's no-super-call __init__ ahead of openai's cooperative
__init__ chain — preventing openai.APIError.super().__init__(message) from
landing in HTTPException.__init__(status_code=message).

LIT-2968

Co-authored-by: Mateo Wang <[email protected]>

* refactor(proxy/hooks): raise ProxyRateLimitError from budget + iteration limiters

Replaces three bare HTTPException(status_code=429, ...) call sites with
ProxyRateLimitError, which is both a RateLimitError (catchable by category)
and an HTTPException (preserves existing FastAPI serialization). Drops the
now-unused HTTPException import in the iteration / per-session limiters.

LIT-2968

Co-authored-by: Mateo Wang <[email protected]>

* refactor(proxy/hooks): raise ProxyRateLimitError from parallel-request limiters

Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3
parallel-request limiters (key/team/user/model/customer rate limits) with
ProxyRateLimitError. Updates the raise_rate_limit_error helper's return type
annotation accordingly.

LIT-2968

Co-authored-by: Mateo Wang <[email protected]>

* refactor(proxy/hooks): raise ProxyRateLimitError from dynamic rate limiters

Replaces HTTPException(status_code=429, ...) call sites in the v1 and v3
dynamic rate limiters (project-level TPM/RPM allocation, model-saturation
checks, priority-based limits, fail-closed guards) with ProxyRateLimitError.
The v3 limiter still imports HTTPException for an unrelated bare 'except
HTTPException:' branch.

LIT-2968

Co-authored-by: Mateo Wang <[email protected]>

* refactor(proxy/hooks): raise ProxyRateLimitError from batch rate limiter

Replaces HTTPException(status_code=429, ...) in batch_rate_limiter._raise_rate_limit_error
with ProxyRateLimitError tagged as RateLimitErrorCategory.LITELLM_BATCH_RATE_LIMIT
so users can distinguish batch-level throttling (which counts requests/tokens
across an uploaded batch input file before submission) from the generic
key/team/user RPM/TPM limiter.

The HTTPException import is retained because the same module raises
HTTPException for unrelated 403/IO error paths.

LIT-2968

Co-authored-by: Mateo Wang <[email protected]>

* test(rate-limit): pin down unified rate-limit error contract

Adds a dedicated test module covering the new RateLimitErrorCategory enum,
RateLimitError.category default + override behavior, ProxyRateLimitError's
dual nature (RateLimitError + HTTPException), and a parametrized regression
guard that asserts every proxy hook module imports the unified class.

The regression guard catches the failure mode the refactor is designed to
prevent: someone re-introducing a bare HTTPException(status_code=429, ...)
in one of the hook modules instead of going through ProxyRateLimitError.

LIT-2968

Co-authored-by: Mateo Wang <[email protected]>

* feat(logging): expose rate-limit category via StandardLoggingPayload

Adds an optional 'error_rate_limit_category' field to
StandardLoggingPayloadErrorInformation, populated from the unified
RateLimitError.category attribute (introduced in the previous commits on
this branch).

Why: the .category attribute is reachable off the raw exception today via
getattr(e, 'category', None), but the structured contract that downstream
custom callbacks / loggers / spend log writers consume is the
StandardLoggingPayload. Without this field, a user building custom
rate-limit metrics on top of callback data has to special-case the raw
exception object — which defeats the purpose of the StandardLoggingPayload
abstraction.

The field is None for non-rate-limit exceptions (so consumers can read it
unconditionally without isinstance checks) and is one of the
RateLimitErrorCategory string values otherwise.

LIT-2968

Co-authored-by: Mateo Wang <[email protected]>

* test(rate-limit): assert StandardLoggingPayload carries the category

Five tests covering: vendor default, explicit litellm_rate_limit and
litellm_batch_rate_limit values, None for non-rate-limit exceptions, and
None when no exception is provided. Pins down the contract that custom
callbacks can read 'error_information.error_rate_limit_category' off the
StandardLoggingPayload to drive custom rate-limit metrics without ever
reaching for the raw exception.

LIT-2968

Co-authored-by: Mateo Wang <[email protected]>

* fix(types): silence mypy [misc] on intentional dual-base attr overlap

mypy emits two [misc] errors on the ProxyRateLimitError class line because
its two bases declare overlapping attributes with related-but-not-identical
annotations:

* status_code: int on starlette HTTPException vs. Literal[429] on openai's
  RateLimitError (every openai status-error subclass narrows it the same
  way and silences pyright with the same convention).
* headers: Mapping[str, str] | None on HTTPException vs. our Optional[
  Dict[str, str]] (the proxy hooks always carry a stringified dict).

Both narrowings are intentional and enforced at construction time. Add a
type: ignore[misc] with an inline explanation rather than relax the
annotations on the parent or change the wire-format guarantees.

LIT-2968

Co-authored-by: Mateo Wang <[email protected]>

* test(rate-limit): add direct hook-invocation tests to lift patch coverage

Adds six end-to-end tests that drive each refactored hook past its
limit and assert the unified ProxyRateLimitError is raised with the
correct category and dual-base shape. Complements the
import-shape-only parametrized guard above by actually executing the
new 'raise ProxyRateLimitError(...)' lines so codecov's patch coverage
sees them as hit.

Hooks covered (one test each):
* parallel_request_limiter v1 — direct call to raise_rate_limit_error()
* parallel_request_limiter v3 — direct call to _handle_rate_limit_error
  with a fabricated OVER_LIMIT response
* max_iterations_limiter — full async_pre_call_hook with mocked agent
  registry, second call exceeds budget=1
* max_budget_limiter — async_pre_call_hook with mocked get_current_spend
* dynamic_rate_limiter v1 — async_pre_call_hook with mocked
  check_available_usage forcing available_tpm == 0
* batch_rate_limiter — direct _raise_rate_limit_error call, asserts
  category is the batch-specific LITELLM_BATCH_RATE_LIMIT (not the
  generic LITELLM_RATE_LIMIT)

LIT-2968

Co-authored-by: Mateo Wang <[email protected]>

* fix: guard rate_limit_category extraction with isinstance check

* test(rate-limit): cover remaining hook raise sites for codecov

Adds five more direct hook-invocation tests so every PR-touched line
in the proxy hooks is exercised by tests in tests/test_litellm/, which
codecov measures:

* parallel_request_limiter v1 — check_key_in_limits inline raise
  (the second raise site, separate from the raise_rate_limit_error
  helper covered earlier)
* dynamic_rate_limiter v1 — RPM raise branch (TPM branch was already
  covered)
* dynamic_rate_limiter v3 — parametrized over all three raise sites:
  model_saturation_check, priority_model, and the fail-closed
  fallback for an unrecognized descriptor_key
* max_budget_per_session_limiter — full async_pre_call_hook with a
  mocked agent registry and over-budget cached spend

All 42 tests in test_rate_limit_error_unification.py now pass and
together exercise every changed import + raise line across the eight
refactored proxy hooks.

LIT-2968

Co-authored-by: Mateo Wang <[email protected]>

* fix: use computed error_message in ProxyRateLimitError detail

* fix(parallel-request-limiter): drop None from detail; annotate raise_rate_limit_error as NoReturn

The v1 ' raise_rate_limit_error' helper built an unused 'error_message'
variable and then assembled the actual ' detail' via an f-string that
interpolated 'additional_details' verbatim — producing
'Max parallel request limit reached None' when invoked without
arguments (flagged by code review).

Fix the helper to:
- use the constructed 'error_message' as the detail
- annotate the helper as NoReturn since it always raises
- drop the redundant 'raise'/'return' at the two call sites

Add two regression tests covering both the with- and without-
additional_details paths.

LIT-2968

Co-authored-by: Mateo Wang <[email protected]>

* fix(proxy/hooks): drop literal 'None' from raise_rate_limit_error detail

The v1 parallel_request_limiter's raise_rate_limit_error helper has a
long-standing bug: it computes a None-guarded 'error_message' string but
then ignores it and emits an f-string that interpolates the raw
'additional_details' arg. Callers that pass no argument get
'Max parallel request limit reached None' as the user-facing detail.

This commit:
* wires error_message into the detail kwarg so the None-guard actually
  applies and operators see a clean message;
* changes the return-type annotation from ProxyRateLimitError to NoReturn
  (the function always raises) so type-checkers know callers after this
  invocation are unreachable.

Greptile P1 + P2 review feedback on PR #27687.

LIT-2968

Co-authored-by: Mateo Wang <[email protected]>

* fix(types): demote TypedDict floating string to a # comment

A string literal placed after a field declaration in a TypedDict body is
not a per-field docstring — it's an orphaned string expression Python
discards. Tools like mypy / pyright that inspect TypedDict fields won't
surface that text either.

Move the documentation for error_rate_limit_category to a real comment
so the intent is visible to readers and type-checker tooling without
the misleading docstring framing.

Greptile P2 review feedback on PR #27687.

LIT-2968

Co-authored-by: Mateo Wang <[email protected]>

* security(exceptions): do not auto-copy vendor response headers to e.headers

A vendor 429 response can set arbitrary headers (Set-Cookie, CORS
overrides, …). Previously, when RateLimitError was constructed with only
a 'response=' (no explicit 'headers=' kwarg), self.headers fell back to
a copy of response.headers. If a downstream proxy serializer ever
forwarded e.headers to the client, a malicious upstream could inject
browser-interpreted headers for the proxy origin.

Drop the fallback. Only headers passed explicitly via the headers= kwarg
make it onto self.headers (proxy hooks pass retry-after etc. — they
control what's surfaced). Vendor response headers stay reachable on
e.response.headers for callers that explicitly want them.

Today's proxy_server.py route handlers don't actually forward e.headers
on the wire (they construct ProxyException without passing headers), so
no current behavior changes — this is a defensive narrowing so the
fallback can never be turned into a vector when someone wires
e.headers through later.

Veria-AI security review feedback on PR #27687.

LIT-2968

Co-authored-by: Mateo Wang <[email protected]>

* test(rate-limit): regression guards for review-pass fixes

Pins down the three review-pass fixes:

* test_parallel_request_limiter_v1_helper_no_additional_details — calls
  raise_rate_limit_error() with no args and asserts the detail does NOT
  contain the literal string 'None'. Pre-fix, callers got 'Max parallel
  request limit reached None'.
* test_rate_limit_error_does_not_auto_copy_response_headers — passes a
  vendor httpx.Response with a Set-Cookie header to RateLimitError
  WITHOUT an explicit headers= kwarg, asserts self.headers stays None
  (no leak), then re-checks that an explicit headers= kwarg DOES
  populate self.headers. Vendor headers remain reachable on
  e.response.headers for callers that explicitly want them.
* The existing v1-helper test now also asserts the additional_details
  string makes it through to the detail.

LIT-2968

Co-authored-by: Mateo Wang <[email protected]>

* feat(rate-limit): add orthogonal RateLimitType (requests/tokens/concurrent_requests/budget/max_iterations)

trho's last ask in the LIT-2968 thread: distinguish rate-limit failures by
the dimension that was exceeded, not just by who rate-limited (vendor vs.
litellm). Adds:

- RateLimitType str-enum exposed at `litellm.RateLimitType` with values
  requests / tokens / concurrent_requests / budget / max_iterations.
- `rate_limit_type` kwarg on litellm.RateLimitError + ProxyRateLimitError;
  None default so existing callers (vendor-429 path in exception_mapping_utils)
  remain a no-op.
- StandardLoggingPayloadErrorInformation.error_rate_limit_type so custom
  callbacks can split rate-limit failures by cause without parsing free-text
  error messages. Mirror to error_rate_limit_category extraction in
  get_error_information(); single isinstance(RateLimitError) check covers both.
- map_v3_rate_limit_type() helper to collapse the v3 limiter's internal labels
  ("requests", "tokens", "max_parallel_requests") onto the public enum so
  the v3 limiter and dynamic_rate_limiter_v3 share one mapping. Defensive
  None on unknown values rather than silently picking a wrong dimension.

Co-authored-by: Mateo Wang <[email protected]>

* feat(proxy/hooks): wire rate_limit_type onto every limiter raise site

Each refactored proxy hook now populates rate_limit_type with the dimension
that actually tripped the limit, so downstream consumers (custom callbacks,
prometheus exporters via the StandardLoggingPayload) can split key/team/user
rate-limit failures by cause:

- parallel_request_limiter (v1): detect dimension from current vs. limit in
  the post-cache branch (concurrent_requests > tokens > requests, matches the
  boolean condition order). Base case (current is None, one limit set to 0)
  picks the most-specific zero. raise_rate_limit_error() helper accepts an
  explicit rate_limit_type kwarg with CONCURRENT_REQUESTS default (matches
  every existing internal call site, including the global-limit branch).
- parallel_request_limiter (v3): forward status["rate_limit_type"] through
  map_v3_rate_limit_type() so "max_parallel_requests" → CONCURRENT_REQUESTS
  for the public field while the raw v3 jargon stays on the HTTP header for
  wire-format backward compat.
- dynamic_rate_limiter (v1): TPM-zero → TOKENS, RPM-zero → REQUESTS. Pass
  data["model"] through so callbacks see the model that hit the limit
  (addresses the secondary "provider missing" complaint in the original
  Slack thread, partially — the model is what dashboards typically split on).
- dynamic_rate_limiter (v3): forward status["rate_limit_type"] via
  map_v3_rate_limit_type() at every raise site (model_saturation_check,
  priority_model, fail-closed unknown-descriptor guard). Also pass model.
- batch_rate_limiter: limit_type is hard-typed "requests"|"tokens" — map
  directly without going through the helper's None branch.
- max_budget_limiter, max_budget_per_session_limiter: BUDGET.
- max_iterations_limiter: MAX_ITERATIONS.

Co-authored-by: Mateo Wang <[email protected]>

* test(rate-limit): cover RateLimitType enum, hook wiring, and StandardLoggingPayload propagation

27 new tests across five new test classes:

- TestRateLimitType: enum exposed at litellm.RateLimitType, all five values
  defined, RateLimitError default is None (vendor 429 path makes no claim
  about which dimension), accepts both string and enum forms with
  str-coercion guarantee for downstream JSON serializers.
- TestProxyRateLimitErrorType: ProxyRateLimitError default is None, accepts
  string or enum, doesn't break existing callers that pass nothing.
- TestMapV3RateLimitType: pins each v3-internal → public-enum mapping
  (tokens, requests, max_parallel_requests → concurrent_requests, unknown
  → None) so a future v3 refactor can't silently swap dimensions.
- TestStandardLoggingPayloadCarriesType: the new error_rate_limit_type
  field reaches the structured payload for both ProxyRateLimitError and
  plain RateLimitError, is None when unspecified, and is None for
  non-rate-limit exceptions (symmetric with error_rate_limit_category).
- TestProxyHooksWireTypeCorrectly: drives the actual raise sites in the
  v1 parallel_request_limiter helper, the v3 _handle_rate_limit_error
  (both "tokens" and "max_parallel_requests" paths), and the batch
  limiter (both tokens and requests paths) — coverage tools see the new
  rate_limit_type= kwargs as exercised, not just the import shape.

Co-authored-by: Mateo Wang <[email protected]>

* test(rate-limit): cover _coerce_message branches and v1 dimension detection

Drives the patch coverage on the new orthogonal RateLimitType wiring up
to (or close to) 100% on the touched files.

ProxyRateLimitError._coerce_message — was 22% covered, now 100%:
* nested {error: {message}} dict
* nested {message: {message}} dict (alt key)
* dict without 'error'/'message' keys → JSON dump fallback
* non-JSON-serializable dict value → str() fallback
* non-string non-mapping detail (int) → str() coercion

v1 parallel_request_limiter dimension detection — was 0% covered, now
exercised across 6 parametrized cases:
* check_key_in_limits else-branch: current at concurrent / TPM / RPM cap
  → asserts rate_limit_type is concurrent_requests / tokens / requests.
* check_key_in_limits base case (current is None): max_parallel_requests
  / tpm_limit / rpm_limit set to 0 → asserts the most-specific zero
  attribution wins per the helper's order.

LIT-2968

Co-authored-by: Mateo Wang <[email protected]>

* feat(proxy/hooks): add ProxyHTTPRateLimitError + provider resolver

Introduces a small helper layer used by every proxy-side rate-limit
hook so that the 429 they raise carries a populated llm_provider /
model — instead of an empty exception.llm_provider that downstream
loggers (Prometheus failure metric, observability callbacks) read as
'no provider attribution'.

ProxyHTTPRateLimitError inherits from both fastapi.HTTPException
(so the proxy server still renders it as a 429) and
litellm.exceptions.RateLimitError (so isinstance checks and
PrometheusLogger._get_exception_class_name pick up llm_provider).
We deliberately don't call RateLimitError.__init__ — it constructs
an httpx.Response we don't need and would just add failure surface;
attribute parity is what downstream consumers care about.

resolve_llm_provider_for_rate_limit() wraps litellm.get_llm_provider
defensively. Internal limiter hooks fire from async_pre_call_hook —
well before get_llm_provider runs anywhere else in the request
lifecycle — so we have to call it ourselves at raise time. If the
model is missing or unparseable (alias, router-only model) we fall
back to llm_provider='litellm_proxy' rather than letting a second
exception leak out and break the request path.

Co-authored-by: Mateo Wang <[email protected]>

* fix(proxy/hooks): populate llm_provider on parallel-request 429s

Both v1 and v3 parallel-request limiters fired bare HTTPException(429)
from inside async_pre_call_hook. The downstream Prometheus failure
metric reads exception.llm_provider via _get_exception_class_name —
the empty value showed up as exception_class='HTTPException' and
left model_id='None' on the time series.

Threads requested_model through every raise site in:

* parallel_request_limiter.py:
  - check_key_in_limits (the per-key/per-model/per-user/per-team/
    per-customer over-limit path)
  - raise_rate_limit_error (zero-limit + global_max_parallel_requests
    paths) — now takes an optional requested_model kwarg
* parallel_request_limiter_v3.py:
  - _handle_rate_limit_error (the OVER_LIMIT translator), called
    from both the should_rate_limit pre-check and the TPM
    reservation path

Resolved via resolve_llm_provider_for_rate_limit so unknown / missing
models silently fall back to llm_provider='litellm_proxy' instead of
breaking the request path with a second exception.

Co-authored-by: Mateo Wang <[email protected]>

* fix(proxy/hooks): populate llm_provider on dynamic-rate-limit 429s

Same plumbing change as the parallel limiters, applied to both
dynamic_rate_limiter (v1) and dynamic_rate_limiter_v3:

* v1: TPM-zero and RPM-zero paths in async_pre_call_hook now resolve
  data['model'] -> (model, llm_provider) once and pass it into both
  raises.
* v3: All three raise sites in _check_rate_limits — the
  model_saturation_check enforced raise, the priority_model
  enforced raise, and the fail-closed unknown-descriptor branch —
  now attribute the 429 to the actual provider.

Falls back to llm_provider='litellm_proxy' when the model can't be
resolved.

Co-authored-by: Mateo Wang <[email protected]>

* fix(proxy/hooks): populate llm_provider on batch-rate-limit 429s

batch_rate_limiter._raise_rate_limit_error now takes a
requested_model kwarg threaded from data['model'] in
_check_and_increment_batch_counters. The batch-creation 429 is what
gets raised when the input file's tokens/requests count would push
the per-key TPM/RPM window over its limit.

Co-authored-by: Mateo Wang <[email protected]>

* fix(proxy/hooks): populate llm_provider on budget/iterations 429s

Final batch of internal raise sites — the user/session-budget and
max-iterations hooks. Same pattern: resolve data['model'] once at
raise time, attach to ProxyHTTPRateLimitError so Prometheus and
observability callbacks can attribute the 429.

Hooks updated:
* max_budget_limiter (per-user max_budget exceeded)
* max_iterations_limiter (per-session agent iteration cap)
* max_budget_per_session_limiter (per-session dollar cap)

All three fall back to llm_provider='litellm_proxy' when data['model']
is missing or unparseable. Drops the now-unused HTTPException import
from each module.

Co-authored-by: Mateo Wang <[email protected]>

* test(proxy/hooks): pin provider field on internal rate-limit 429s

Regression coverage for the 'provider field missing' bug across every
proxy-side rate-limit hook + the helper layer:

* ProxyHTTPRateLimitError class shape (HTTPException + RateLimitError,
  dict-detail stringification, None-provider normalization).
* resolve_llm_provider_for_rate_limit happy paths
  (gpt-4o-mini, anthropic/..., bedrock/...) plus all three fallback
  branches (None, '', unknown name) plus a 'get_llm_provider raises'
  case that asserts we swallow the secondary exception.
* For each limiter (parallel v1/v3, dynamic v1/v3, batch,
  max_budget, max_iterations, max_budget_per_session): assert the
  raised exception is a RateLimitError carrying the resolved
  model + llm_provider, and a sibling test that asserts the
  fallback path returns 'litellm_proxy' without leaking a second
  exception.
* Two PrometheusLogger._get_exception_class_name pins so the
  Prometheus failure metric label flips from 'HTTPException' to
  'Openai.ProxyHTTPRateLimitError' (or 'Litellm_proxy.*' on
  fallback) — that's what dashboards consume.

Co-authored-by: Mateo Wang <[email protected]>

* perf(proxy/hooks): defer provider resolution to over-limit branches

* fix: use error_message in raise_rate_limit_error to avoid literal 'None' in detail

* Consolidate rate_limiter_utils imports in dynamic_rate_limiter

* fix(proxy): set num_retries/max_retries on ProxyHTTPRateLimitError

ProxyHTTPRateLimitError inherits from RateLimitError but did not call
RateLimitError.__init__, so num_retries/max_retries were never set.
When Starlette's HTTPException lacks __str__, MRO falls through to
RateLimitError.__str__, which unconditionally reads these attributes
and raises AttributeError during logging/traceback formatting.
Initialize them to None defensively.

* fix(mypy): silence base-class status_code conflict on ProxyHTTPRateLimitError

HTTPException declares 'status_code: int' while openai.RateLimitError
(via APIStatusError) declares 'status_code: Literal[429] = 429'. Mypy
flags the multi-base override as [misc] in CI lint. The runtime semantics
are fine (we set self.status_code in __init__), so silence the
class-level annotation conflict with a targeted ignore.

Co-authored-by: Mateo Wang <[email protected]>

* fix: annotate batch limiter _raise_rate_limit_error as NoReturn

* feat(prometheus): rate-limit category/type labels + exception_class back-compat (follow-up to #27687) (#27706)

* feat(prometheus): add rate_limit_category and rate_limit_type labels

Adds two new labels to litellm_proxy_failed_requests_metric so dashboards
can split 429s by rate-limit source (vendor vs. litellm-internal) and by
the dimension that was exceeded (requests/tokens/concurrent_requests/
budget/max_iterations) without parsing free-text error messages.

Closes the Prometheus side of LIT-2718. The unified RateLimitError.category
and .rate_limit_type fields landed in PR #27687 but were only surfaced on
StandardLoggingPayload (custom-callback channel); this exposes them on
the metric label set as well.

Both labels are populated only when the underlying exception is a
litellm.RateLimitError; non-rate-limit failures keep them empty.

Co-authored-by: Mateo Wang <[email protected]>

* feat(prometheus): populate rate-limit labels + preserve exception_class back-compat

Two coupled changes in the Prometheus integration:

1. async_post_call_failure_hook now extracts the new RateLimitError
   .category / .rate_limit_type fields (added in PR #27687) via a
   _extract_rate_limit_labels helper and forwards them through
   UserAPIKeyLabelValues onto litellm_proxy_failed_requests_metric.
   Empty for non-rate-limit failures.

2. _get_exception_class_name special-cases ProxyRateLimitError and
   keeps emitting 'HTTPException' for the exception_class label.
   Without this shim, ProxyRateLimitError (which multi-inherits from
   HTTPException + RateLimitError) would silently flip the label
   from 'HTTPException' (the historical value for proxy-side 429s)
   to 'ProxyRateLimitError', breaking existing dashboards / alerts
   that key off exception_class='HTTPException'. Distinguishing
   vendor vs. litellm 429s is now the job of the new
   rate_limit_category label.

Co-authored-by: Mateo Wang <[email protected]>

* test(prometheus): cover rate-limit labels and exception_class back-compat

Adds 19 tests across:
- enum / label-list registration
- _extract_rate_limit_labels for vendor RateLimitError, ProxyRateLimitError,
  non-rate-limit and None inputs (incl. parametrized over every
  RateLimitErrorCategory x RateLimitType combo)
- _get_exception_class_name back-compat: ProxyRateLimitError keeps the
  legacy 'HTTPException' string while vendor RateLimitError keeps the
  historical 'Provider.ClassName' format
- end-to-end through async_post_call_failure_hook with both
  ProxyRateLimitError and vendor RateLimitError, asserting both new
  labels populate and exception_class stays back-compat

Co-authored-by: Mateo Wang <[email protected]>

* fix(prometheus): tolerate missing fastapi in lazy ProxyRateLimitError import

Address greptile feedback:
- async_post_call_failure_hook docstring: drop the stale labelnames listing
  and reference PrometheusMetricLabels.litellm_proxy_failed_requests_metric
  as the source of truth so the doc cannot drift from the actual labelset.
- _get_exception_class_name: guard the lazy ProxyRateLimitError import with
  ImportError so router-side fallback callsites don't blow up in non-proxy
  installs that don't have fastapi (a transitive dep of
  proxy.common_utils.proxy_rate_limit_error). Behavior is unchanged when
  fastapi is available.

Also fix the existing enterprise callback test that asserted the old
labelset on litellm_proxy_failed_requests_metric — it now expects the new
rate_limit_category / rate_limit_type labels populated for vendor 429s.

---------

Co-authored-by: Cursor Agent <[email protected]>
Co-authored-by: Mateo Wang <[email protected]>

* fix(bugbot): simplify rate-limit label coercion + guard None detail

- prometheus.py _extract_rate_limit_labels: RateLimitError.__init__ already
  normalizes category/rate_limit_type to plain str, so the getattr(.value)
  + isinstance dance was dead code. Reduce to str(value) if not None.
- proxy_rate_limit_error.py _coerce_message: short-circuit None to ''
  instead of falling through to str(None) = 'None', which produced the
  literal message 'litellm.RateLimitError: None'.

* fix(rate-limit): surface unified category/type fields on BudgetExceededError

The most common budget cap (virtual-key max_budget enforcement in
auth_checks.py) raises litellm.BudgetExceededError, a bare Exception
subclass that bypassed the unified rate-limit error class introduced
by PR #27687. Custom callbacks reading
StandardLoggingPayload.error_information saw category=None and
rate_limit_type=None for these 429s, missing the most common budget
case (team / org / end-user budgets all hit the same code path).

Surface the fields off BudgetExceededError as plain attributes:
- category = RateLimitErrorCategory.LITELLM_RATE_LIMIT
- rate_limit_type = RateLimitType.BUDGET
- llm_provider = "" (or caller-supplied)

Switch get_error_information and _extract_rate_limit_labels from
isinstance(RateLimitError) gating to duck-typed attribute reads,
guarded by membership in the rate-limit enums so unrelated third-party
exceptions exposing a .category attribute can't leak garbage values
into the payload.

This is strictly additive: BudgetExceededError keeps its bare-Exception
base class, so `except BudgetExceededError:` handlers keep firing and
`except RateLimitError:` does not start catching budget errors.

* fix(rate-limit): validate enum membership at duck-typed read sites + enrich BudgetExceededError llm_provider

Two follow-ups uncovered during the second QA pass on PR #27687:

1. Guard third-party `.category` / `.rate_limit_type` attribute leakage.
   The duck-typed read in `get_error_information` and
   `_extract_rate_limit_labels` would forward any string attribute named
   `category` / `rate_limit_type` on an unrelated third-party exception
   into the StandardLoggingPayload and Prometheus labels — silently
   mislabeling custom-callback payloads and blowing out Prometheus label
   cardinality. Add `validate_rate_limit_category` /
   `validate_rate_limit_type` helpers that gate on the documented enum
   value sets; non-matching values are dropped to None.

2. Enrich BudgetExceededError.llm_provider from request_data.
   Budget checks live in tenant-scoped helpers (key / team / org / tag /
   end-user / project) that don't see the request model, so the
   BudgetExceededError they raise carried llm_provider="" — leaving
   custom-metrics consumers without provider attribution for the most
   common 429 case. Resolve it once at the central
   UserAPIKeyAuthExceptionHandler seam, before post_call_failure_hook
   fires, so the StandardLoggingPayload the callback sees has the same
   provider attribution as RPM/TPM 429s.

Regression tests pin both: 4 leakage tests + 4 enrichment tests. The
leakage tests would fail under the pre-validation version of either read
site; the enrichment tests would fail if the handler skipped the
resolver call.

* fix(rate-limit): resolve router model_name aliases to real provider (#27914)

* fix(rate-limit): resolve router model_name aliases to real provider

For nearly every real LiteLLM proxy deployment the request model is a
router model_name alias (e.g. 'tpm-locked' -> litellm_params.model:
openai/gpt-4o-mini), and 'litellm.get_llm_provider' doesn't know about
router aliases — it raises 'LLMProviderNotProvidedError'. The resolver
then fell through to the defensive 'litellm_proxy' fallback, so the
'llm_provider' field this PR adds was effectively always
'litellm_proxy' in the field, defeating its purpose for the most common
proxy configuration.

Add a router-alias fallback step: when 'get_llm_provider' raises, scan
the active 'llm_router.model_list' for a deployment whose 'model_name'
matches the request model and resolve from its 'litellm_params.model'
instead. If multiple deployments share the same alias (load-balancing
case) the first one wins — every deployment under one alias should
agree on provider in any sensible config, and 'first' is deterministic
so the Prometheus label stays stable.

Defensive throughout: an uninitialized router, a malformed deployment,
a 'litellm_params.model' that itself fails 'get_llm_provider' — every
branch falls through to the existing 'litellm_proxy' fallback rather
than letting a secondary exception escape and mask the rate-limit
error we're trying to surface.

Tests:
  - test_router_alias_resolves_to_underlying_provider: alias
    'tpm-locked' -> 'openai/gpt-4o-mini' produces provider='openai',
    model='gpt-4o-mini'.
  - test_router_alias_with_multiple_deployments_uses_first.
  - test_router_alias_unknown_falls_back.
  - test_router_alias_with_malformed_deployment_falls_back.
  - Existing fallback test updated to also stub
    'litellm.proxy.proxy_server.llm_router' so it exercises the
    full 'no resolution anywhere' path.

Co-authored-by: Mateo Wang <[email protected]>

* fix(rate-limit): harden router alias resolver + test isolation

- Wrap _resolve_provider_from_router_alias loop in top-level try/except so
  a non-iterable model_list / unexpected deployment shape can't escape and
  mask the 429 with a 500.
- Type-check litellm_params before .get() to handle non-dict truthy values.
- Patch llm_router=None in the parametrized fallback test so a router left
  by another test in the session can't redirect the unknown-model path.

---------

Co-authored-by: Cursor Agent <[email protected]>
Co-authored-by: Mateo Wang <[email protected]>

* fix(bugbot): preserve "BudgetExceededError" Prometheus label

Adding llm_provider to BudgetExceededError (so callbacks get provider
attribution from StandardLoggingPayload) made the provider-prefix step in
_get_exception_class_name silently flip the label from "BudgetExceededError"
to e.g. "Openai.BudgetExceededError", breaking dashboards keyed on the
historical value.

Short-circuit BudgetExceededError in _get_exception_class_name the same way
ProxyRateLimitError already is. Provider/category attribution still lands on
the new rate_limit_category / rate_limit_type labels.

* test: fix invalid 'rpm' rate_limit_type in v3 limiter test mocks

The v3 rate limiter only emits 'requests', 'tokens', or
'max_parallel_requests'. Using 'rpm' caused map_v3_rate_limit_type to
return None, leaving the expected RateLimitType.REQUESTS untested.

Co-authored-by: Yassin Kortam <[email protected]>

* fix(bugbot): hoist provider resolver + opt-in prom rate-limit labels

- dynamic_rate_limiter.py: hoist resolve_llm_provider_for_rate_limit
  above the TPM/RPM if/elif so the lookup runs once per request, matching
  the pattern in dynamic_rate_limiter_v3.py.
- prometheus.py: gate the new rate_limit_category / rate_limit_type
  labels on litellm_proxy_failed_requests_metric behind
  litellm.prometheus_emit_rate_limit_labels (default False). Mirrors the
  existing prometheus_emit_stream_label opt-in. Preserves the metric's
  pre-unification label set so existing dashboards / recording rules
  keep matching after upgrade; operators can enable the new labels once
  downstream consumers include them.
- Tests updated: default-off back-compat case, opt-in path enables the
  flag before asserting label presence.

* fix: stabilize prometheus label sets and drop redundant model normalization

- Cache PrometheusLogger.get_labels_for_metric per metric_name so that
  the label set used to construct counters at __init__ time stays in
  sync with the label set used at increment time, even if module-level
  toggles like prometheus_emit_rate_limit_labels or
  prometheus_emit_stream_label are flipped at runtime. Without this,
  toggling these flags after the logger was created would cause
  ValueError from prometheus_client because the runtime labels would
  not match the counter's declared labelnames.
- Drop redundant 'model or ""' guard in ProxyRateLimitError.__init__
  where model is already normalized one step earlier.

Co-authored-by: Yassin Kortam <[email protected]>

* perf(dynamic_rate_limiter): only resolve provider when rate limit hit

Co-authored-by: Yassin Kortam <[email protected]>

* test(prometheus): clear cached metric labels after toggling rate-limit flag

The PrometheusLogger caches each metric's label set at construction
time so that labels used at counter.labels(...) time stay consistent
with the labels the metric was registered with. The enterprise
async_post_call_failure_hook test toggles
litellm.prometheus_emit_rate_limit_labels = True AFTER the fixture
has already built the logger, so without invalidating the cache the
rate_limit_category / rate_limit_type labels never reach the mocked
counter and the assert_called_once_with check fails.

Co-authored-by: Yassin Kortam <[email protected]>

* test: fix CI failures from prom label cache + flaky time-window assertion

PrometheusLogger.get_labels_for_metric now caches the per-metric label
set at first read so the labels passed to counter.labels(...) stay in
lock step with the labels the counter was registered with. This broke
two existing test patterns:

- test_prometheus_labels.py: tests bind the real method onto a
  MagicMock, but MagicMock auto-creates a Mock for _cached_metric_labels
  whose .get(...) returns a truthy Mock — treated as a populated cache
  and returned as the label set, producing empty filtered labels and
  KeyError on labels["requested_model"] / ["route"]. Seed real {}
  containers for _cached_metric_labels and label_filters before binding.

- test_prometheus_logging_callbacks.py::test_set_team_budget_metrics_with_custom_labels:
  the fixture builds the logger before the test monkeypatches
  litellm.custom_prometheus_metadata_labels, so the cached label set
  never picks up the new metadata labels. Clear the cache after the
  monkeypatch (same pattern already used for the rate-limit toggle in
  test_async_post_call_failure_hook).

UI: view_logs/index.test.tsx "Last Minute" window assertion is off by
one at the minute boundary. start_date is floored to the minute, so the
dropped sub-minute fraction can push the truncated-seconds diff up to
(minMinutes+1)*60 exactly when the click lands near a minute rollover.
Switch the upper bound to toBeLessThanOrEqual.

* feat(otel-v2): surface rate_limit_category + rate_limit_type on failed LLM-call spans

PR #28909 introduced the typed v2 OTel engine that builds spans from
StandardLoggingPayload, with SpanError carrying error_type + message and
the genai mapper stamping error.type onto every failed LLM-call span.
This PR's earlier commits added error_rate_limit_category and
error_rate_limit_type to the same StandardLoggingPayload.error_information
the v2 engine reads — but neither field reached a span attribute, so v2
OTel traces stayed opaque about *why* a 429 fired (vendor vs litellm,
RPM vs TPM vs concurrent vs budget vs max_iterations) even after the
custom-callback and prometheus surfaces gained that decomposition.

Three coupled changes:

1. semconv.py: add LiteLLM.ERROR_RATE_LIMIT_CATEGORY /
   LiteLLM.ERROR_RATE_LIMIT_TYPE under the litellm.* vendor namespace
   (no GenAI semconv equivalent exists for who-rate-limited /
   which-dimension).

2. payloads.py: extend SpanError with rate_limit_category +
   rate_limit_type, populated by _parse_error() from the same
   error_information.error_rate_limit_* fields the custom-callback
   channel and prometheus rate_limit_category / rate_limit_type labels
   read. Single source of truth across all three observability surfaces.

3. mappers/genai.py: stamp the two attributes on the LLM-call span when
   present. drop_none guarantees they stay absent (not 'None') for
   non-rate-limit failures so trace consumers can read them
   unconditionally.

Three regression tests in test_otel_v2_emitter.py pin: a vendor /
litellm-internal RateLimitError lands category=litellm_rate_limit +
rate_limit_type=requests on the span; a BudgetExceededError lands
rate_limit_type=budget; a non-rate-limit failure (BadRequestError)
keeps the rate_limit_* attributes absent. Mutation-tested against
reverting either the SpanError extension or the _parse_error read site
— both new tests fail under either mutation.

Co-authored-by: Mateo Wang <[email protected]>

* test: align prometheus user-budget + logs quick-select tests with merged code

The merge into this branch left two test patterns out of step with the code
they exercise.

test_set_user_budget_metrics_includes_user_email_and_alias_labels_when_opted_in
flipped litellm.prometheus_user_budget_label_include_email_alias after the
fixture had already built the PrometheusLogger. get_labels_for_metric now
snapshots each metric's label set at construction time, so the runtime flip
no longer reached the cached labels. Enable the flag before constructing the
logger, matching how the proxy applies config at startup.

view_logs/index.test.tsx referenced uiSpendLogsCall and moment without
importing them, and the merged index.tsx now fetches through
useLogFilterLogic (the hook the file stubs out) rather than calling
uiSpendLogsCall directly. Add the imports and restore the real hook for the
Quick Select window assertions so the call is actually observed.

* refactor(otel/v2): drop rate-limit decomposition from the LLM-call span

Proxy-side rate limits (litellm_rate_limit, budget, max_iterations) are
rejected at the gate before any upstream call, so async_post_call_failure_hook
tags the synthetic failure log with LITELLM_LOGGING_NO_UPSTREAM_LLM_CALL and the
v2 OTel logger never opens an LLM-call span for them; the
litellm.error.rate_limit_category / litellm.error.rate_limit_type attributes
were dead for exactly the cases they were meant to surface. The only failure
that does open an LLM-call span carrying a RateLimitError is a vendor 429, where
rate_limit_type is always None and the category just restates
error.type=RateLimitError.

The decomposition still reaches downstream consumers through
StandardLoggingPayload.error_information.error_rate_limit_* and the prometheus
rate_limit_category / rate_limit_type labels, both unchanged.

Removes the SpanError fields, the _parse_error reads, the genai mapper
attributes, the semconv keys, and the three span tests that asserted a scenario
that never reaches the mapper in production.

* fix(batch_rate_limiter): map max_parallel_requests to concurrent_requests

* refactor(prometheus): drop transitive fastapi import from _get_exception_class_name

Read the legacy exception_class label from a prometheus_exception_class_name
marker on ProxyRateLimitError instead of importing the proxy module, keeping
the integrations layer free of a transitive fastapi dependency.

* chore(ui): sync schema.d.ts with unified rate-limit error spec

The ProxyRateLimitError docstring flows into the proxy OpenAPI spec's 429
response description, so the generated dashboard types were out of sync.
Regenerated via npm run gen:api (Check UI API Types Sync).

---------

Co-authored-by: Cursor Agent <[email protected]>
Co-authored-by: Mateo Wang <[email protected]>
Co-authored-by: Yassin Kortam <[email protected]>

…ed (#29872)

The Guardrails page hardcoded defaultActiveKey="submitted", so admins
landed on the "Submitted Guardrails" tab (the last of their four tabs)
instead of the primary view. The original intent was for non-admins,
whose only tab is Submitted Guardrails, to default there; admins should
open on their first tab.

Make the default role-aware: admins default to the first tab (Guardrail
Garden), non-admins keep Submitted Guardrails.

… subclass (#29869)

…I types (#29885)

The dashboard calls UI-internal proxy routes that the public /openapi.json hides with include_in_schema=False, so they never reached schema.d.ts and could not be typed. The type generator now force-includes those routes when it dumps the spec for openapi-typescript; this mutates a throwaway interpreter only, so the spec the proxy actually serves is unchanged.

Regenerates schema.d.ts so 86 internal route families (for example /v2/model/info, /global/spend/*, /config/*, /v2/login, /sso/*) are now typed, with no public route removed. This unblocks migrating the dashboard's data fetching onto the typed $api client.

Branch CI note: schema.d.ts is generated; CI regenerates and diffs it via the same gen:api script.

* feat(proxy): publish /v2/model/info in Swagger OpenAPI spec

Expose the v2 model info endpoint in /docs by removing include_in_schema=False
and documenting query parameters used by the admin UI and proxy CLI consumers.

Co-authored-by: Cursor <[email protected]>

* chore(ui): regenerate schema.d.ts for /v2/model/info OpenAPI docs

Co-authored-by: Cursor <[email protected]>

---------

Co-authored-by: Cursor <[email protected]>

Consolidate the three hand-synced copies of the migrated-pages map
(LEGACY_REDIRECTS in app/page.tsx, MIGRATED_PAGES in the dashboard layout,
and MIGRATED_PAGES in leftnav) into one shared module,
src/utils/migratedPages.ts, which also owns the migratedHref helper. Delete
the unused, incomplete Sidebar2 prototype.

No runtime behavior change: the map is still empty and Sidebar2 had no
importers, so this is pure deduplication ahead of the per-page App Router
migration. Follow-up work will unify the remaining base-URL builders
(layout's withBase and page.tsx's redirect) onto migratedHref.

)

The provider logo base path was relative ("../ui/assets/logos/"). With
trailingSlash enabled, the public model hub is served at /ui/model_hub_table/,
so the browser resolved the base to /ui/ui/assets/logos/ (a doubled /ui/), which
404s every icon. The authenticated hub renders inside the single-level /ui/ SPA
route where the relative path resolves correctly, so only the public hub broke.
Make the base root-absolute so it resolves at any route depth.

The create guardrail modal used antd's default maskClosable, so clicking
outside it dismissed the modal and reset every field the user had entered.
Setting maskClosable={false} keeps the modal open; it now closes only via
the explicit close button or Cancel, matching the other form modals in the
dashboard

…9870)

The key edit page showed the default key type (no allowed_routes restriction)
as "Default", while the key creation form already labels the same value
"Full Access". Align the edit page to the create form so the two surfaces agree
on both the label and its description.

…#29953)

* fix(ui): unify migrated-route URLs and cut the API Reference page over to path routing

Route all migrated-page navigation through one /ui-prefixed, serverRootPath-aware
builder in migratedPages.ts (migratedHref/legacyPageHref/legacyKeyForPathname),
replacing the three divergent base-URL constructions that lived in the dashboard
layout's withBase, leftnav, and the page.tsx redirect. The previous migratedHref
read NEXT_PUBLIC_BASE_URL, which no build sets, so it produced URLs without the
/ui prefix the app is served under; every other internal link hardcodes /ui and
this now matches that convention.

Remove the sidebar's own pushState navigation so the parent (legacy root page or
dashboard layout) is the single owner of navigation, fixing the double-navigate
that fired when moving between a path route and a legacy ?page= route.

Cut API Reference over to its path route: add api_ref -> api-reference to
MIGRATED_PAGES and delete its arm from the legacy switch. Visiting
/ui/?page=api_ref redirects to /ui/api-reference, the sidebar links to and
highlights it, and navigating away returns to the legacy switch.

* fix(ui): address review on migrated-page routing

Keep the legacy hyphenated ?page=api-reference form working by mapping it to the
api-reference route alongside api_ref; the old switch matched both, so a bookmark
using the hyphen would otherwise fall through to the Usage default.

Add legacyKeyForPathname coverage: a migrated path (with and without trailing
slash) resolves to the api_ref sidebar key rather than the alias, a non-migrated
path returns null, and a non-root serverRootPath prefix is stripped before matching.

* fix(ui): populate serverRootPath from getUiConfig so migrated nav links keep the root path

getUiConfig updated proxyBaseUrl but never called updateServerRootPath, so the
module-level serverRootPath stayed at its "/" default. Under a custom
server_root_path the unified migratedHref/legacyPageHref builders then dropped
the prefix and the sidebar produced /ui/api-reference (404) instead of
/<root>/ui/api-reference. Adds the missing updateServerRootPath call plus a
regression test asserting getUiConfig sets serverRootPath and that migratedHref
carries the prefix

…the Tools page (#29867)

* fix(ui): let non-creator users OAuth into OBO-mode MCP servers from the Tools page

* fix(ui): clear OBO Tools-tab one-shot on navigate-back and gate on credential-status errors

* feat(bedrock_mantle): add SigV4/IAM auth to Responses API route (fixes #29665) (#29788)

* feat(responses): add default no-op sign_request to BaseResponsesAPIConfig

* feat(responses): call sign_request after body is final, send signed bytes when signed

* feat(bedrock_mantle): add SigV4 sign_request via composed BaseAWSLLM (bearer path)

* test(bedrock_mantle): cover SigV4 access-key, AssumeRole, body bytes, region/auth consistency

* feat(bedrock_mantle): defer auth to sign_request; validate_environment no longer requires bearer

* docs(bedrock_mantle): document SigV4 + Bearer auth on Responses route

* test(responses): cover fake-stream signing order and mantle bearer arg/env precedence

* fix(bedrock_mantle): wrap all botocore credential errors with both-paths guidance

* fix(bedrock_mantle): catch specific credential errors, not all BotoCoreError, so STS transport failures are not masked

* fix(bedrock_mantle): sign the compact Responses route too, not just create

* fix(github-copilot): route per-model on /v1/responses based on model info (#29747)

* feat(focus): add GCS destination for FOCUS export (#29751)

* test: add failing tests for FocusGCSDestination

* feat: add FocusGCSDestination reusing GCSBucketBase auth

* feat: register FocusGCSDestination in factory; export from __init__

* fix(focus): preserve GCS_PATH_SERVICE_ACCOUNT when service_account_json not in config

* style: apply Black formatting to gcs_destination and tests

* style: apply Black formatting to factory.py

* fix(bedrock): omit empty additionalModelRequestFields and system from Converse API payload (#29565)

Amazon Nova Pro (and other strict Bedrock models) return 400 Malformed input
request when additionalModelRequestFields: {} or system: [] are present in the
payload. Both fields are optional in CommonRequestObject (total=False) and must
be omitted rather than sent as empty structures.

Co-authored-by: shin-berri <[email protected]>
Co-authored-by: yuneng-jiang <[email protected]>
Co-authored-by: Claude Sonnet 4.6 <[email protected]>

* fix(proxy): recognize *.cognitiveservices.azure.com as OpenAI-compatible in pass-through cost tracking (#29730)

* fix(proxy): recognize *.cognitiveservices.azure.com as OpenAI-compatible

Azure OpenAI resources created via the newer "Azure AI Foundry" /
Cognitive Services pathway live on `*.cognitiveservices.azure.com`
subdomains, not the older `openai.azure.com`. Both are valid Azure
OpenAI surfaces in production today.

The OpenAI pass-through cost-tracking handler hard-codes only the older
hostname in five places (four `is_openai_*_route` methods on
OpenAIPassthroughLoggingHandler, plus is_openai_route on
PassThroughEndpointLogging). As a result, calls from newer Azure
deployments are silently classified as "not an OpenAI route", the
dispatch into the cost-tracking handler is skipped, and tokens/cost
never get extracted into LiteLLM_SpendLogs — the row gets written with
prompt_tokens=0, completion_tokens=0, spend=0, model='unknown'.

Reproduced 2026-06-04 against a real Azure OpenAI deployment on
`*.cognitiveservices.azure.com` proxied through LiteLLM v1.88.0.

Fix: factor the hostname check into a single helper
`_is_openai_compatible_host` listing all three recognized surfaces
(api.openai.com, openai.azure.com, cognitiveservices.azure.com), and
have all five call sites delegate to it. Purely additive — never
weakens recognition for the originally-supported hostnames.

Adds a test
`test_is_openai_route_recognizes_cognitiveservices_azure_com` that
exercises all four `is_openai_*_route` static methods against
`*.cognitiveservices.azure.com` URLs (positive cases per route + a
small cross-route negative to confirm route-specific path matching
still works on the new hostname).

Out of scope for this PR (separate followup):
  - `openai_passthrough_handler` calls chat/completions
    `transform_response` on Responses API payloads (`output:` not
    `choices:`), which throws inside the dispatch and drops the
    SpendLogs row entirely. Recognized + tracked separately.

* ci: trigger fresh run

Empty commit to re-run checks. The previous auth-and-jwt failure was
a transient HuggingFace Hub 429 rate-limit hitting tokenizer downloads
in tests/proxy_unit_tests/test_custom_tokenizer_bug.py — unrelated to
this PR's scope (hostname recognition in pass-through cost tracking).
No code change.

---------

Co-authored-by: shin-berri <[email protected]>
Co-authored-by: yuneng-jiang <[email protected]>

* fix(responses): preserve forced-function tool_choice name in Responses to Chat transform (#29812)

The Responses API forces a specific function with a top-level name
({"type": "function", "name": "X"}), but _transform_tool_choice only handled the
nested Chat Completions shape and fell through to returning "required" for the flat
form, silently dropping the function name and degrading a forced function call to
force-any-tool. Map the flat Responses shape to the nested Chat shape, keeping the
"required" fallback when no name is present.

* Preserve x-anthropic-billing-header system blocks for first-party Anthropic (#29584)

* Preserve x-anthropic-billing-header system blocks for first-party Anthropic

PR #20951 strips system blocks beginning with "x-anthropic-billing-header:" for
every Anthropic target. That block is how the first-party Anthropic API recognizes
Claude Code subscription (OAuth) traffic, so dropping it makes requests that carry
only that block, such as the auto-mode tool-safety classifier, fail with a
misleading 429 rate_limit_error; normal turns still work because they also carry
the "You are Claude Code" identity block.

Gate the strip behind should_strip_billing_metadata(), defaulting to False on the
first-party AnthropicConfig and AnthropicMessagesConfig so the block is kept, and
overridden to True on the providers that reach these transforms and reject the
block (Bedrock platform, Vertex, Azure for the chat path; Minimax, Azure, DeepSeek
for the messages path). Behavior for those providers is unchanged.

* Strip billing header on Bedrock invoke and Vertex messages pass-through

Two more subclasses reach the gated strip but inherited keep-by-default.
AmazonAnthropicClaudeConfig (Bedrock invoke) calls AnthropicConfig.transform_request,
which calls translate_system_message, and VertexAIPartnerModelsAnthropicMessagesConfig
(Vertex messages pass-through) calls super().transform_anthropic_messages_request.
Override should_strip_billing_metadata() to True on both.

Add a parametrized test asserting the flag for every first-party base (False) and
provider subclass (True), covering all overrides, plus a translate_system_message
regression test for the Bedrock invoke path.

* fix(cache): log hashed cache keys (#29890)

* fix(ui): save routing groups as list (#29889)

* Revert "fix(ui): save routing groups as list (#29889)" (#29928)

This reverts commit 9b1f78f.

* feat(parasail): add Parasail as a JSON-configured OpenAI-compatible provider (#29842)

* feat(parasail): add Parasail as a JSON-configured OpenAI-compatible provider

Registers parasail in the openai_like JSON provider loader with both
/v1/chat/completions and /v1/responses support. Parasail's Responses API
rejects store:true and any request that omits store, so the loader gains a
force_store_false special_handling flag; the parasail entry sets it and
the generated Responses config overrides store=false on every call. This
keeps callers from hitting "State storage not supported" and matches what
Parasail's docs require.

Adds the PARASAIL enum value, listing under openai_compatible_providers,
provider documentation at docs/my-website/docs/providers/parasail.md, and
a focused unit test file under tests/test_litellm/llms/parasail/ that
covers JSON registration, chat URL construction, Responses URL
construction with PARASAIL_API_BASE override, and the force_store_false
regression in both the caller-sent-store=true and caller-omitted cases.

* fix(parasail): register in provider_endpoints_support, drop in-repo docs

Greptile review feedback. The provider doc belongs in the litellm-docs
repo, not this one's docs/my-website tree; removing it here. Adds the
parasail entry to provider_endpoints_support.json so the
check_provider_folders_documented.py CI check passes (chat_completions
and responses true; others false).

* fix: normalize Anthropic passthrough server tool usage (#29827)

* test(anthropic): cover server_tool_use dict cost tracking

* fix: normalize Anthropic server tool usage

(cherry picked from commit 982f726)

* fix: keep server tool usage subscriptable

(cherry picked from commit 70280b9)

---------

Co-authored-by: Genmin <[email protected]>

* fix(proxy): fix typo generic_role_mappoings -> generic_role_mappings in ui_sso.py (#29753)

Co-authored-by: shin-berri <[email protected]>
Co-authored-by: yuneng-jiang <[email protected]>

* feat(proxy): add disable_budget_reservation general setting (#27639) (#29493)

* feat(proxy): add disable_budget_reservation general setting (#27639)

* feat(proxy): register disable_budget_reservation in ConfigGeneralSettings (#27639)

* docs(proxy): document disable_budget_reservation concurrency tradeoff (#27639)

* ci: re-trigger flaky docker build (prisma generate ECONNRESET)

* fix(proxy): warn and document budget enforcement tradeoff when disable_budget_reservation is set (#27639)

* feat(gemini_tts): adding support to Gemini TTS languageCode parameters (#29623)

* Adding support to Gemini TTS Language Code parameters

* Mapping Gemini TTS languageCode param in Docstring

* Use snake_case for language_code input keyMapping Gemini TTS languageCode param in Docstring

* Restoring files modified under enterprise/litellm_enterprise due to lint/formatting checks

---------

Co-authored-by: João Garrido <[email protected]>

* feat(guardrails): capture user and model metadata in CrowdStrike AIDR (#29517)

* fix(proxy): require OpenAI path segment for shared Azure Cognitive Services domains

Address Greptile review: the `*.cognitiveservices.azure.com` /
`*.openai.azure.com` domains are shared by every Azure Cognitive Service
(Speech, Vision, Language, ...), so a hostname-only substring match
misclassified non-OpenAI Azure traffic as OpenAI routes.

- Replace the substring host test with suffix matching (rejects look-alike
  domains like cognitiveservices.azure.com.attacker.example).
- Add `_is_openai_compatible_url` that requires an OpenAI-style path marker
  (`/openai/` or `/v1/`) on the shared Azure domains, and use it in
  PassThroughEndpointLogging.is_openai_route (previously hostname-only).
- Add negative tests for Azure Speech/Vision paths and look-alike domains.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* fix: support Responses input in Redis semantic cache (#29581)

* fix: support responses input in redis semantic cache

* test: cover redis semantic prompt extraction

* test: handle blank redis semantic text fallbacks

* chore: remove async cache dead statement

* test: cover redis semantic cache miss paths

* fix: filter sensitive cache lookup kwargs

* chore: rerun ci after huggingface rate limit

* chore(ui): regenerate dashboard API types (npm run gen:api)

Sync src/lib/http/schema.d.ts with the proxy OpenAPI spec: adds the
disable_budget_reservation general-settings field and picks up the
RateLimitError docstring reindent. Fixes the gen:api CI drift check.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

* test(bedrock): assert empty additionalModelRequestFields is omitted

The Converse transformer now drops an empty additionalModelRequestFields
block instead of sending it as `{}`. Update test_bedrock_top_k_param so
models without top_k support (llama3) assert the key is absent rather than
equal to an empty dict.

Co-Authored-By: Claude Opus 4.8 <[email protected]>

---------

Co-authored-by: Kent <[email protected]>
Co-authored-by: codgician <[email protected]>
Co-authored-by: Praveen Ghuge <[email protected]>
Co-authored-by: Roi <[email protected]>
Co-authored-by: shin-berri <[email protected]>
Co-authored-by: yuneng-jiang <[email protected]>
Co-authored-by: Claude Sonnet 4.6 <[email protected]>
Co-authored-by: Liam Scott <[email protected]>
Co-authored-by: abhay23-AI <[email protected]>
Co-authored-by: Ceder Dens <[email protected]>
Co-authored-by: 冯基魁 <[email protected]>
Co-authored-by: Kai Huang <[email protected]>
Co-authored-by: rinto <[email protected]>
Co-authored-by: Genmin <[email protected]>
Co-authored-by: Arnav Bhilwariya <[email protected]>
Co-authored-by: Armaan Sandhu <[email protected]>
Co-authored-by: João Garrido <[email protected]>
Co-authored-by: João Garrido <[email protected]>
Co-authored-by: Kenan Yildirim <[email protected]>
Co-authored-by: Dávid Balatoni <[email protected]>

* feat(galileo): add health check support for UI callback test

Register galileo in /health/services so the proxy UI callback connection test works.

Co-authored-by: Cursor <[email protected]>

* feat(galileo): verify API key via /current_user health check

Call Galileo's current_user endpoint so the UI callback test validates credentials against the provider.

Co-authored-by: Cursor <[email protected]>

* chore(ui): regenerate schema.d.ts for galileo health service

Co-authored-by: Cursor <[email protected]>

* fix(galileo): return IntegrationHealthCheckStatus from async_health_check

Fixes mypy assignment error in health_services_endpoint where response was
narrowed to IntegrationHealthCheckStatus from earlier branches.

Co-authored-by: Cursor <[email protected]>

* Fix Galileo logging to match Langfuse across all endpoint types.

Stop skipping ingest when output is empty and log embeddings with a placeholder so embedding, speech, and other non-text responses are recorded like Langfuse.

Co-authored-by: Cursor <[email protected]>

* fix(galileo): remove unreachable health-check guard and None output sentinel

The use_v2_api flag is derived from bool(api_key), so the inner
GALILEO_API_KEY check inside the v2 branch could never run; collapse the
credential validation into the username/password path with a combined
message. _serialize_galileo_output now returns an empty string for None,
so _get_galileo_input_output_content always yields a str and the
post-call None coalescing guard is no longer needed.

* test(galileo): cover async_health_check failure paths and empty model response

Add regression tests for the Galileo health check unhealthy branches
(missing project id, missing base url, missing credentials, auth
failure, and request exception) and for logging a model response with
no choices, which now queues an empty output instead of being skipped.

---------

Co-authored-by: Cursor <[email protected]>
Co-authored-by: mateo-berri <[email protected]>

…deleted (#29875)

* fix(model-management): allow deleting a BYOK model after its team is deleted

A team BYOK model (model_info.team_id set) became undeletable once its team
was deleted: POST /model/delete ran can_user_make_model_call, which looked the
team up and raised 400 "Team id=... does not exist in db" before the delete
could run, so the model lingered on the Models + Endpoints page with no way to
remove it.

Drop the team-existence prerequisite from the delete path. When the model's
team still exists the normal auth check runs unchanged; when it is gone a proxy
admin may delete the orphan and any other caller gets a 403. The check is
fail-closed, so a missing or errored team lookup can only block the delete or
require an admin, never grant a non-admin access. Add/update/health keep their
team-existence validation.

* refactor(model-management): drop redundant team lookup on model delete

Move the orphaned-team handling into can_user_make_model_call behind an
allow_missing_team flag instead of pre-checking team existence in delete_model.
The endpoint no longer issues its own litellm_teamtable lookup, so deleting a
model whose team still exists hits the team table once instead of twice. The
auth behavior is unchanged: a proxy admin can delete a model whose team was
deleted, any other caller gets a 403, and add/update/health keep the strict
"team must exist" validation.

…28913)

* fix(jwt-auth): defer to single-team DB fallback on claim mismatch

Extends the single-team DB fallback introduced in #26418 to two more
cases where it previously could not run:

* `find_and_validate_specific_team_id`: when `team_id_jwt_field` is
  configured and a claim value is present in the token but the team
  does not exist in the LiteLLM DB (HTTPException 404 from
  `get_team_object`), return `(None, None)` instead of raising — the
  auth_builder fallback then attributes the request to the user's
  single DB team. Only HTTPException is caught; other errors (e.g.
  "No DB Connected") still propagate.

* `find_team_with_model_access`: when none of the `team_ids_jwt_field`
  groups resolve to a real LiteLLM team, return `(None, None)` instead
  of raising 403 so the same fallback path runs. If at least one group
  DID resolve to a team but none granted the requested model, the
  original 403 is preserved (legitimate access denial — not a claim
  mismatch). Tracked via the new `any_claim_team_resolved` flag.

The strict `is_required_team_id` raise and `enforce_team_based_model_access`
raise remain unchanged. Unit tests cover both new soft-fail paths and
guard each preserved path (strict required, enforce_team_based, the
preserved 403, and the non-HTTPException propagation).

Co-authored-by: Cursor <[email protected]>

* fix(jwt-auth): narrow HTTPException catch to 404 (greptile review)

Address Greptile review comments on #28913:

* `find_and_validate_specific_team_id`: re-raise HTTPException when
  `status_code != 404`, pinning the catch to the "team doesn't exist
  in db" path documented for `get_team_object`. A future change that
  introduces a different status code (e.g. 403 for a blocked team)
  will now propagate instead of silently falling through to the
  single-team DB fallback.

* Add `test_find_and_validate_specific_team_id_non_404_http_exception_propagates`
  parametrised over 400 / 403 / 500 to lock in the contract.

Co-authored-by: Cursor <[email protected]>

* fix(jwt-auth): gate claim-mismatch fallback behind opt-in flag

The unresolved-team-claim fallback added in the previous commit
weakened the strict claim-based authorization contract by default —
an authenticated user whose JWT carries a stale or invalid team
claim could still consume their single DB team's models/quota via
the fallback.

Gate both soft-fail paths in `find_and_validate_specific_team_id`
and `find_team_with_model_access` behind a new opt-in flag
`team_claim_fallback` on `LiteLLM_JWTAuth` (default False).

Default-off preserves the pre-existing strict behavior. Operators
who intentionally treat IdP team claims as advisory (e.g. machine
tokens whose group claims live in a separate namespace from
LiteLLM team_ids) opt in via config.

Adds two regression tests guarding the default-off behavior.

Co-authored-by: Cursor <[email protected]>

---------

Co-authored-by: Cursor <[email protected]>

#29525)

On /team/update for a standalone (no-org) team, _check_user_team_limits()
compared the request max_budget against the caller's personal max_budget
whenever max_budget was present in the payload. A team admin whose personal
budget is lower than the team's budget could not edit any field (tpm_limit,
team name, etc.) because the UI re-sends the unchanged max_budget on every
update, tripping the personal-budget check.

Pass the team's current max_budget into _check_user_team_limits() and skip the
personal-budget comparison when the incoming value is unchanged or lower than
the team's current budget. Only genuine increases above the team's current
budget are still validated against the caller's personal limit, so no
over-relaxation. Proxy admins and the org-scoped path are unaffected.

Adds two regression tests for the standalone update path (unchanged budget +
tpm_limit change, and lowering the budget), both for a caller whose personal
budget is below the team budget.

Co-authored-by: Cursor <[email protected]>

…9697)

glm-5p1 supports native tools on Fireworks; explicit false flags caused
drop_params to strip tools and tool_choice before the provider request.

Co-authored-by: Cursor <[email protected]>

…cks (#29899)

* fix(vertex): propagate Vertex AI metadata in streaming success callbacks

Streaming calls assembled via stream_chunk_builder were missing
vertex_ai_grounding_metadata and vertex_ai_url_context_metadata in
standard_logging_object.response. Merge metadata from chunks into the
assembled response and mirror non-streaming hidden_params on Gemini chunks.

Co-authored-by: Cursor <[email protected]>

* refactor(vertex): move streaming metadata merge into provider config hook

Address review feedback by delegating assembled-stream metadata propagation
to VertexGeminiConfig via BaseConfig.apply_assembled_streaming_response_metadata,
and only write chunk hidden_params when metadata is non-empty.

Co-authored-by: Cursor <[email protected]>

* fix(redaction): scrub Vertex provider metadata when message logging is off

Clear vertex_ai_grounding_metadata and related fields from standard
logging responses and assembled streaming ModelResponse objects so
turn_off_message_logging cannot leak prompt-derived web search queries.

Co-authored-by: Cursor <[email protected]>

* Use assembled model for streaming metadata hook

* Fix Vertex metadata redaction bypass in logging callbacks.

Scrub Vertex provider fields from litellm_params.metadata.hidden_params during perform_redaction so streaming success_handler merges do not leak prompt-derived metadata when message logging is disabled.

Co-authored-by: Cursor <[email protected]>

* Fix Vertex streaming metadata from hidden params

* fix(vertex): mirror vertex_ai_safety_results on assembled streaming responses

The non-streaming transform_response stores safety data under
vertex_ai_safety_results, but the streaming path only wrote
vertex_ai_safety_ratings. Assembled streaming responses therefore never
carried vertex_ai_safety_results, so any consumer reading that field saw
a silent difference between streaming and non-streaming calls.

Set vertex_ai_safety_results alongside vertex_ai_safety_ratings in the
shared stream metadata setter and add it to the assembled metadata field
list so it propagates through stream_chunk_builder.

* fix(streaming): log provider streaming metadata hook failures instead of swallowing them

* refactor(vertex): share single Vertex metadata field tuple across redaction and streaming

* refactor(vertex): move Vertex metadata redaction helpers into llms/vertex_ai

---------

Co-authored-by: Cursor <[email protected]>
Co-authored-by: mateo-berri <[email protected]>

Allow internal users to fetch their backend-scoped project list so the key creation project dropdown can populate for selected teams.

Raise the PyJWT floor in pyproject (>=2.13.0,<3.0) and re-resolve uv.lock so
the proxy installs 2.13.0 instead of 2.12.0. Bump the ws transitive-version
override in the dashboard from 8.19.0 to 8.20.1 and regenerate package-lock;
jsdom and openai both dedupe onto the single 8.20.1 copy.

Both are routine dependency maintenance bumps to keep pinned versions current.

…leted (#29977)

A team's BYOK models (rows in LiteLLM_ProxyModelTable with model_info.team_id set)
were left orphaned when the team was deleted; they lingered in the database and kept
showing on the Models + Endpoints page. delete_team now removes them via a new
delete_team_models helper that deletes the rows in one transaction and syncs the
in-memory router only after that transaction commits, run before the team rows are
deleted so a mid-flight failure never leaves the team gone with its models orphaned

…8184)

* feat(vantage): include organization metadata in FOCUS Tags export

Join LiteLLM_OrganizationTable when building Vantage/FOCUS export rows so
organization_id and organization_alias appear in Tags for org-level filtering.

Co-authored-by: Cursor <[email protected]>

* test(focus): include api_requests in organization Tags tests

FocusTransformer now requires api_requests after staging merge; add the
column to test fixtures so integrations CI can run the Tags assertions.

Co-authored-by: Cursor <[email protected]>

---------

Co-authored-by: Cursor <[email protected]>

…gs (#29991)

Capture user_id and extra_info from metadata or litellm_metadata. The single-bag read dropped identity whenever a request carried a present litellm_metadata field (null or a user-supplied dict), since /chat/completions routes the authenticated identity into metadata while the guardrail read litellm_metadata first

* feat(proxy): add configurable response headers middleware

Adds a small ASGI middleware that sets standard response headers
(X-Frame-Options, Content-Security-Policy frame-ancestors, X-Content-Type-Options)
on proxy and UI responses. Strict-Transport-Security is optional and gated
behind LITELLM_ENABLE_HSTS for HTTPS deployments. Values use setdefault so a
route that sets its own header is preserved.

* feat(proxy/ui): make login page credentials hint configurable

build_ui_login_form accepts a hide_default_credentials_hint parameter and
google_login reads LITELLM_HIDE_DEFAULT_CREDENTIALS_HINT (or general_settings)
so the legacy login page behaves consistently with the new UI. Also collapses
a duplicated branch and removes an unused variable and module-level constant.

* fix(proxy/ui): apply credentials hint flag on /fallback/login

The /fallback/login handler still rendered the default-credentials hint
regardless of LITELLM_HIDE_DEFAULT_CREDENTIALS_HINT. Collapse its duplicate
branch and forward the flag, matching google_login, so all login surfaces
behave consistently. Adds regression tests for /fallback/login and makes the
ui_sso test helper restore os.environ so env vars do not leak across tests.

)

Switch the zizmor check to fail on any finding at medium severity or
above (advanced-security off, min-severity medium, annotations on) so it
can be promoted to a required check, and pin the engine to zizmor 1.24.1
through zizmor-action v0.5.6 for deterministic runs.

Clear the findings that were outstanding so the check passes: correct
mismatched action pin version comments, scope the proxy endpoint
workflow's id-token and pull-requests permissions to the jobs that use
them, and mark the server-root-path docker build as non-publishing while
dropping its shared gha build cache.

…tion streaming passthrough (#30800)

* fix(proxy): use e.request_data for logging_obj in ModifyResponseException streaming passthrough

When a guardrail blocks a streaming request pre-call by raising
ModifyResponseException (or RejectedRequestError), chat_completion streams the
violation message back as a 200 by building a CustomStreamWrapper. It read the
logging object from the outer request body (`data.get("litellm_logging_obj")`),
but that dict never carries litellm_logging_obj -- it diverges from the
processor's data at function_setup, and only the processor copy (exposed as
e.request_data, already bound to `_data` here) gets the logging object
attached. CustomStreamWrapper.__init__ then dereferences
`logging_obj.model_call_details` on None and 500s the request with
"AttributeError: 'NoneType' object has no attribute 'model_call_details'".

Read logging_obj from `_data` (= e.request_data) in both streaming
passthrough handlers so the refusal streams correctly. The non-streaming and
the anthropic/responses passthrough paths were unaffected.

Adds a regression test asserting the wrapper receives the logging object from
e.request_data rather than None.

* test(proxy): cover RejectedRequestError streaming passthrough

The streaming logging_obj fix was applied to both the ModifyResponseException
and RejectedRequestError handlers, but only the former had a regression test.
Extract a shared helper and add a parallel test for the RejectedRequestError
streaming path so both handlers stay guarded against the None-logging_obj crash.

---------

Co-authored-by: Joseph Barker <[email protected]>

…AWS SDK (#30815)

…creds per exporter (#30590)

* fix(otel): one v2 logger owns the global provider; scope tenant creds per exporter

The proxy published the OTel global TracerProvider before callbacks were
initialized, so no preset logger existed yet and a second generic logger was
built that won the global provider. Server spans then exported through a
different provider than the preset's gen-ai spans, orphaning the LLM span on
the preset backend. Publish after callback init and reuse the already-built
logger instead.

Separately, per-request tenant OTLP credentials were stamped onto every OTLP
exporter, leaking one backend's key onto a co-configured backend. Tag each
exporter with the preset that contributed it and apply dynamic credentials
only to the matching owner.

* fix(otel): satisfy Any-discipline on changed lines

Type the logger-selection parameter as Sequence[object] (isinstance narrows
it), cast the list[Any] global at the single call site, and pass model_copy a
typed dict[str, str] update so no changed line carries an Any value.

* fix(otel): annotate the untyped-global boundary with any-ok

select_global_otel_v2_logger consumes litellm._in_memory_loggers, a shared
List[Any] global this change does not own. A cast doesn't satisfy the
Any-discipline checker (it inspects the inner expression), and re-annotating the
global is out of scope, so mark the single boundary line any-ok.

* test(otel): cover the startup global-provider publish via injectable helper

The publish step lived inline in proxy_startup_event (a FastAPI lifespan unit
tests do not execute), so its lines were uncovered though the selection logic
was tested. Extract publish_global_otel_v2_provider, which selects the single v2
logger and publishes its provider through an injected setter, and unit-test that
the published provider is the selected logger's. proxy_server delegates to it.

* refactor(otel): select global provider from the registered owner, not a list scan

The startup publish picked the global TracerProvider by scanning
_in_memory_loggers for the first OpenTelemetryV2, re-deriving an answer the
factory already settled: the first logger built registers itself as
proxy_server.open_telemetry_logger, and every other v2 path (guardrail, identity
seeding, phase spans) routes through that owner via _registered_v2_logger. Pass
that owner into select_global_otel_v2_logger so the global provider reuses the
same logger instead of an independent, order-dependent guess; the list scan
remains the SDK-path fallback. The owner is injected at the proxy call site to
keep the helper free of hidden global reads.

* refactor(otel): type ExporterSpec.owner as an ExporterOwner enum

The owner field carried free-form strings that had to match preset callback
names. Introduce a str-based ExporterOwner enum (values equal to the callback
names, so per-request credential routing's owner==callback_name comparison still
holds) and have each preset tag its exporter with the enum member.

* refactor(otel): rename ExporterOwner.ARIZE to ARIZE_AX

Distinguish the hosted Arize AX backend from Arize Phoenix at the member level
while keeping the value 'arize' (the public callback name routing compares
against). Add a comment noting AX and Phoenix are separate backends.

…eams (#30787)

…treams (#30788)

A streaming request that breaks mid-flight, for example on a mid-stream read
timeout, still bills the provider for the chunks already delivered, yet the proxy
recorded that interrupted request as a zero-spend failure. An earlier revision
logged the recovered partial usage through the success path, which mislabeled a
failed request as a success and produced a misleading spend row

This recovers the partial usage where the failure is actually logged. The
streaming handler assembles the usage from the chunks seen so far and stashes it,
with its cost, on the logging object before firing the failure handlers. The
proxy failure hook lifts that usage and cost onto request_data before the
non-serialisable logging object is popped, and the spend-log writer records the
real partial spend on the failure row instead of a hardcoded zero;
get_logging_payload honors the recovered usage for the token columns and
_failure_handler_helper_fn preserves the recovered cost so the non-DB failure
loggers stay consistent

A request that recovers via a successful fallback is unaffected: the failure hook
only fires when the whole request fails, so the fallback's combined-usage success
row stays the single source of truth and there is no double counting

Resolves LIT-3825

Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>

The "View Usage Guide" button on the legacy Usage page (shown when
DISABLE_EXPENSIVE_DB_QUERIES is set, i.e. SpendLogs has 1M+ rows) linked
to docs/proxy/spending_monitoring, which was removed from the docs and
now returns 404. Point it at docs/proxy/cost_tracking, which is live.

Fixes LIT-2724

…29990)

The delete-team confirmation modal warned that a team's keys would be
deleted but said nothing about models. #29977 made team deletion also
delete the team's BYOK models, so the modal copy was understating what
gets removed.

The warning banner now mentions models alongside keys, and the
always-shown confirmation message does too so a team that has models but
no keys (the banner only renders when keys exist) still gets warned.

…he scope keys (#30675)

Adds a "valkey-semantic" cache type so semantic prompt caching can run
against Valkey clusters (for example AWS ElastiCache for Valkey) using the
valkey-search module.

The existing "redis-semantic" backend cannot drive valkey-search. RedisVL
gates the connection on a RediSearch module version that valkey-search does
not report, and its SemanticCache index declares the prompt as a TEXT field,
which valkey-search does not implement. ValkeySemanticCache therefore talks to
valkey-search directly over redis-py: it builds a vector index from the field
types valkey-search supports (TAG for caller scope, VECTOR for the prompt
embedding) and runs KNN queries for retrieval. Prompt extraction, embedding
generation, and cached-response parsing are reused from RedisSemanticCache
since those are backend agnostic. The redis dependency is imported lazily in
the cache dispatch so importing litellm without redis installed still works.

It also fixes semantic-cache scope keys so similarity matching works across
reworded prompts. get_cache_key() hashed messages / prompt / input into the
litellm_cache_key that every semantic backend filters its KNN search on, so a
paraphrase landed in a different bucket and never matched, even far above the
similarity threshold. For semantic cache types the prompt-bearing params are
now excluded from the scope key and the server-set tenant identity
(user_api_key, team, org) is appended instead, restoring embedding matching
within a tenant while keeping cache entries scoped to the authenticated
key / team / org. The three semantic backends share this key, so the same
change fixes redis-semantic and qdrant-semantic.

Connections resolve from VALKEY_HOST / VALKEY_PORT / VALKEY_PASSWORD, falling
back to REDIS_* for drop-in compatibility, and passwordless clusters (IAM or
no-auth) are supported.

Resolves #29121
Fixes #29086

)

The deprecated OldTeams component takes only accessToken, userID, userRole
and premiumUser; it ignores the teams prop these tests passed and instead
populates its table from the mocked teamListCall. The delete-warning block
never set teamListCall, and vi.clearAllMocks clears call history but not
implementations, so the table rendered the "Legacy Team" (keys.length 2)
left behind by the previous block's last test. Both delete tests therefore
ran against that leaked team: the keys-present case passed only because the
leaked count happened to be 2, and the no-keys case rendered the same warning
it asserted should be absent, so it failed.

Seed the team through the channel the component actually reads (teamListCall)
and drop the props it never consumes, so each test renders exactly the team
it declares. The keys-present case now uses a distinctive count so it can no
longer pass on a coincidental leak

… ruff gate (#30877)

* feat: add CI-parity mode and truncation-proof summary to strict ruff gate

* refactor: tolerant worktree cleanup and concrete GateInputs types

* fix: clean up temp dir when git worktree add fails

* fix: align lint-gate with CI by dropping unused --ci-parity path

The lint-gate Makefile target invoked ruff_strict_gate.py with --ci-parity,
which counted violations on a throwaway merge of base into HEAD against base
counts at the base tip. CI in test-linting.yml runs the same script without
--ci-parity on a PR-head checkout, taking the gather_fast path that counts on
the live tree against base counts at the merge-base. A local pass could
therefore disagree with CI.

Drop --ci-parity from the Makefile and remove the now-unused gather_ci_parity
branch and flag so there is one code path that both local and CI exercise.
The docstring claim that CI runs against the synthetic merge ref was also
wrong; the workflow checks out github.event.pull_request.head.sha.

---------

Co-authored-by: Cursor Agent <[email protected]>

….20.2) (#30894)

* bump: version 0.1.42 → 0.1.43

* bump: version 1.89.0 → 1.90.0

* adding uv lock

…30897)

* fix(watsonx): wrap string embedding input in array for WatsonX API

WatsonX text/embeddings expects inputs as []string; OpenAI clients often send a single string.

Co-authored-by: Cursor <[email protected]>

* style(watsonx): format watsonx embed transformation for black

Co-authored-by: Cursor <[email protected]>

* fix(watsonx): avoid UP006 in embed transformation strict lint gate

Use list[str] and branch-based input normalization instead of List and cast
so the watsonx embedding change does not add strict ruff UP006 violations.

Co-authored-by: Cursor <[email protected]>

---------

Co-authored-by: Shivam Rawat <[email protected]>
Co-authored-by: Cursor <[email protected]>

…ndpoint (#30900)

* test: point router/completion/triton tests at the local fake OpenAI endpoint

The shared Railway-hosted mock (exampleopenaiendpoint-production.up.railway.app)
takes down unrelated CI jobs whenever it is unreachable. #30695 moved the mounted
proxy configs onto a job-local fake server but left these in-Python api_base
literals pointing at the dead host, so litellm_router_testing, local_testing_part1,
local_testing_part2 and llm_translation_testing still fail with a 404
"Application not found" when Railway is down

Resolve the api_base from FAKE_OPENAI_API_BASE (default http://127.0.0.1:8190)
through a shared helper, auto-start the canned server from the local_testing and
llm_translation conftests when nothing is already serving, and extend the server
with a Triton embeddings route and a slow-endpoint delay so the triton and
latency-timeout tests run fully offline. The deliberately broken fallback URL is
left as-is so fallback handling still has a failing upstream

* fix: ignore non-loopback FAKE_OPENAI_API_BASE so the local mock is used in CI

* fix: drop 0.0.0.0 from loopback hosts, an unreliable client connect target

* fix(tests): keep fake OpenAI mock alive across xdist workers

ensure_fake_openai_endpoint registered atexit on the worker that spawned
the subprocess, so under -n 4 the first worker to drain its queue would
terminate the shared mock while siblings were still hitting it. Detach
the child via start_new_session and drop the per-worker teardown; reuse
on /health handles re-runs and CI containers clean up themselves

* feat(sandbox): add e2b code execution primitive

Add a provider-agnostic code execution primitive that runs model-generated
code in an isolated sandbox and returns the output, with e2b as the first
backend over raw httpx (no SDK dependency).

Public API: litellm.acode_interpreter_tool (ephemeral create -> run -> delete)
plus the low-level lifecycle litellm.acreate_sandbox / arun_code /
adelete_sandbox. Each is @client-decorated so operations are logged like
litellm.asearch. Backends implement BaseSandboxConfig; resolved via
ProviderConfigManager.get_provider_sandbox_config.

* fix(sandbox): address review feedback and CI gates

- document e2b provider in provider_endpoints_support.json and add a sandbox endpoint definition
- regenerate dashboard CallTypes after the sandbox call-type additions
- guard explicit timeout=0 instead of coercing it to the default
- require a ContainerHandle access token before running code; reject bare-id runs
- return False on a 404 delete now that the shared http handler raises for status
- skip non-JSON NDJSON lines and cap streamed output to bound memory
- move the real-network integration tests out of tests/test_litellm into tests/integration/sandbox

* fix(sandbox): satisfy strict ruff gate and scope star-exports

- modernize annotations in the new sandbox modules to PEP 585/604 (list/dict,
  X | None) and drop the now-unnecessary quoted forward refs so the strict-rule
  budget delta for UP006/UP037/UP045 returns to zero
- add __all__ to litellm/sandbox/main.py so 'import *' only re-exports the four
  public entrypoints instead of leaking module-level imports

* fix(sandbox): drop quotes on sandbox config return annotation

utils.py uses 'from __future__ import annotations', so the quoted forward ref
tripped UP037; the unquoted union is lazily evaluated and keeps the strict-rule
delta at zero

* chore(sandbox): re-trigger automated review after addressing feedback

… is not set" (#30903)

The migrated /ui/api-keys route gates rendering on useAuthorized() but read userID from the AuthContext (useAuth), which hydrates asynchronously. On a hard refresh or deep link the route could render UserDashboard before AuthContext had populated userID, so UserDashboard hit its `userID == null` guard and showed "User ID is not set". The legacy index page avoided this by gating on AuthContext's own authLoading; the migration switched the gate to useAuthorized without aligning the identity source.

Read identity from useAuthorized (a synchronous cookie decode) so userID is populated whenever the route is authorized. useAuth is kept only for the backfill setters UserDashboard still expects, until the planned AuthContext consolidation removes them.

Refs LIT-3687

….20.2) (#30906)