…API (BerriAI#30902)

The AI Hub MCP Hub listed each MCP server's upstream URL in a table column and in the server Details modal, on both the authenticated dashboard and the public hub. The unauthenticated GET /public/mcp_hub endpoint also returned the url field via MCPPublicServer, so the upstream address was readable by any client even with the column gone. These surfaces are for end users discovering available servers, so the gateway-internal endpoint should not be exposed there.

Drop the URL column from both MCP Hub tables and the URL field from both detail modals, and remove url from MCPPublicServer so /public/mcp_hub no longer serialises it; schema.d.ts is updated to match. The public hub MCPServerData interface no longer declares url since the response omits it. Admin surfaces that configure the endpoint (the MCP server management page, the submissions review tab, the make-public form) and the authenticated /v1/mcp/server endpoint are untouched.

publicMCPHubColumns is lifted to a module-level export so both hub column sets get a mutation-killing regression test, public_model_hub.test.tsx covers the details modal hiding the url, and test_public_endpoints.py asserts /public/mcp_hub never returns url even when the server has one.

…ettings per call (BerriAI#30989)

is_otel_v2_enabled() constructed a pydantic-settings model (_OTelV2Flag) on every
call, which re-scans the process environment and costs ~28us. The flag is read
multiple times along the proxy request hot path (auth, logging-callback setup,
proxy_server), so the cost compounded into a measurable per-request CPU overhead
and a throughput regression visible from v1.87.3 onward.

The flag is a process-level setting that is fixed at startup, so resolve it once
with lru_cache. Caching it alone restores throughput to the pre-regression
baseline in load tests. Tests that toggle the env now call cache_clear().

…ss models (BerriAI#30980)

The "day-by-day by user and model" usage export overcounted spend by
repeating each user-day total once per model. generateDailyWithModelsData
iterated day.breakdown.models crossed with the entity's own
api_key_breakdown and added the api key's full daily spend to every model,
leaving the per-model object (modelData) unused and never matching the api
key to the model. A user who called N models got N identical rows, so the
column summed to N times the real spend; one reported export totaled
$167,953 against a true $21,353 (7.87x).

Attribute each model only the spend of the keys that actually used it by
intersecting the entity's api keys with modelData.api_key_breakdown. This
mirrors the already-correct pattern in activity_metrics.tsx, so per-model
rows now sum back to the user-day total and models a user never called no
longer appear.

The existing tests passed only because their mocks omitted
models[*].api_key_breakdown and never asserted numeric values. The mocks
now match the real /user/daily/activity payload, and new tests assert that
per-model spend sums to the user-day total and that uncalled models are
omitted; both fail on the old code.

Resolves LIT-3866

…als (BerriAI#30585)

* fix: validate proxy request body and nested fields

Ensure caller-supplied request fields cannot override server-side deployment
configuration, and apply request-body validation consistently to nested
structures. Adjusts router kwarg handling and client-side credential handling
for base-url overrides

* test: cover router strip ordering and advisor clientside credential gate

* fix: clear deployment credentials on client base-url override

When a request overrides api_base/base_url, recompute the deployment's
litellm_params (clearing the deployment's own api_key) and drop the cached
client built for the original endpoint, so the deployment credential is not
reused for the client-supplied endpoint. Adds regression tests that assert the
credentials actually forwarded to litellm.completion/acompletion.

* fix(proxy): require api_key alongside api_base override

A request that overrides api_base/base_url but supplies no api_key still
left the proxy carrying a server credential: once the override clears the
banned-param opt-in, the provider re-resolves a key from the environment
(api_key or get_secret("OPENAI_API_KEY") and ~30 sibling chains in
main.py) and forwards it to the caller-controlled URL. Popping the
deployment api_key only changed which server key leaked.

Gate is_request_body_safe so a permitted api_base/base_url override must
also carry a non-empty caller api_key; reject otherwise. The env
resolution in main.py is left as the provider boundary.

* fix(proxy): extend request-body banlist with five additional credential and session targeting fields

Yuneng's review found five deployment-owned request-body params still missing
from the denylist and the router strip set. Each lets a caller reach the
operator's provider credentials or retarget the outbound request:
aws_profile_name selects a local AWS profile, oci_compartment_id and oci_region
retarget the OCI request, litellm_credential_name selects any server-loaded
credential by name with no ownership check, and runtimeSessionId resumes a
Bedrock AgentCore runtime session (AWS does not enforce session-to-user
mapping, so this is a cross-tenant session-resume vector).

Add all five to _BANNED_REQUEST_BODY_PARAMS in auth_utils.py and to
_DEPLOYMENT_OWNED_CREDENTIAL_KWARGS in router.py. Deployment litellm_params and
SDK direct calls are unaffected: the banlist gates the request body only, and
the router strip drops caller kwargs, never deployment["litellm_params"].

* test: rename arbitrary canary values in security tests to neutral placeholders

* fix(proxy): apply api_key co-presence to nested base override and warn on Router credential strip

P1-A: is_request_body_safe descended into _NESTED_CONFIG_KEYS
(litellm_embedding_config, extra_body) for the banned-param check but not for
the api_key co-presence check, so a base override smuggled into one of those
nested dicts cleared the client-side-credentials opt-in without a paired
api_key and let the provider re-resolve a server credential from the
environment. Run _check_base_override_has_api_key on each nested config dict
too, so the requirement applies wherever a base override is permitted.

P1-B: the deployment-owned credential strip in the Router runs unconditionally
on every _completion/_acompletion, which is security-correct but silently
drops per-call api_version/vertex_project/etc. for SDK Router callers. Emit a
single warning (key names only, never values) when the strip removes a
non-empty value, so the backwards-incompatible behavior is visible without
gating the strip on a context flag that does not exist.

* fix(proxy): apply api_key co-presence to tool-entry base override

is_request_body_safe scans three surfaces (root, _NESTED_CONFIG_KEYS, and
tools[]); the previous commit extended the api_key co-presence rule to root
and nested config dicts but not to tool entries. With
allow_client_side_credentials enabled, a tool entry carrying api_base/base_url
and no paired api_key cleared the gate, letting a provider interceptor fall
back to a server-side credential for a caller-controlled URL. Add the same
_check_base_override_has_api_key call to each tool dict and its nested function
dict, mirroring the symmetry already applied to the nested config keys. The
rule is unchanged: api_key must live in the same dict as the base override it
accompanies.

* test(proxy/auth): require paired api_key under extra_body opt-in

* fix(router): gate deployment-owned kwarg strip on litellm.proxy_is_running

* fix(advisor): narrow proxy-import guard to ImportError-family

* fix(router): gate api_key clear on base override behind litellm.proxy_is_running

* test(proxy/auth): scope proxy_is_running flag to dynamic-params class with autouse fixture

* style: use built-in generics in PR-added type annotations

* revert: drop proxy_is_running flag and router-level credential strip; rely on proxy gate

* revert: scope PR to LIT-3828 + LIT-3834 only; drop LIT-3830/LIT-3833 changes

* style: black-format advisor orchestration test

…30895)

Adds an optional litellm_settings.scim_admin_group. When configured, the
global proxy role is recomputed from a user's resulting groups on every SCIM
write that can change membership: user create/PUT/PATCH and group
create/PUT/PATCH/DELETE, plus the existing-email upsert path. Membership in
the admin group grants PROXY_ADMIN and its absence demotes to the non-admin
default, enabling just-in-time elevation and automatic demotion without a
re-login. When the setting is unset the role is never touched, so current
behavior is preserved and a misconfigured IdP can never unexpectedly grant
admin.

…BerriAI#30893)

Map the SCIM enterprise extension block
(urn:ietf:params:scim:schemas:extension:enterprise:2.0:User) onto SCIMUser
so create and PUT persist employeeNumber, costCenter, organization,
division, department, and manager into LiteLLM_UserTable.metadata under
scim_enterprise, and round-trip them back out on read. This lets financial
reporting group spend by fields like cost center and department.

The enterprise block holds directory-only HR attributes, so it is kept out
of the generic user management responses (/user/info, /v2/user/info, and
/user/list), which non-proxy-admin callers such as team and org admins can
use to read other users. The data still lands in metadata for reporting and
still round-trips through the SCIM read endpoints, which build their response
from the user row directly.

Resolves LIT-3617

…iAI#30992)

The Usage dashboard "Spend Per User" chart rendered raw UUIDs (and
default_user_id) instead of emails. /user/daily/activity passed
entity_metadata_field=None, so every user entity in the breakdown
carried empty metadata; the chart could only fall back to the user_id.
The frontend resolved labels from a separately paginated user list, so
any spender not on a loaded page showed as a UUID.

Resolve the email/alias for the user_ids actually on the page (mirroring
how api key metadata is already resolved) and attach it to the entity
metadata, so the chart labels each spender with their email and falls
back to the UUID only when no email is on file. get_daily_activity gains
an optional resolve_entity_metadata hook so the user endpoint can do this
page-scoped lookup without loading the whole user table.

Resolves LIT-3889

…rough body (BerriAI#30985)

* fix: prevent key-level metadata.tags from leaking into Bedrock passthrough body

* test: cover bedrock key-tag litellm_metadata pre-seed in common_checks

Add a regression test asserting key-level tags on a bedrock passthrough
request land in litellm_metadata and never leak into the provider-facing
metadata field, which closes the codecov/patch gap on the auth_checks
pre-seed line. Also drop the now-stale comment that hardcoded
metadata["headers"]; the headers are written under whichever metadata
field _get_metadata_variable_name selects.

* refactor(auth): pre-seed litellm_metadata from LITELLM_METADATA_ROUTES

The auth-time pre-seed in common_checks hardcoded "bedrock", so any other
route later added to LITELLM_METADATA_ROUTES would reintroduce GH#30629 (key
tags leaking into the provider-facing metadata field) without a matching
update here. Key off the shared constant instead, and extend the regression
test to cover a non-bedrock metadata route so the route-agnostic behavior is
locked in.

* fix(auth): pre-seed litellm_metadata before header-tag merge

apply_client_tag_policy_pre_auth runs in user_api_key_auth.py before
common_checks, so it resolved get_metadata_variable_name_from_kwargs to
'metadata' (litellm_metadata was not yet present). common_checks then
pre-seeded litellm_metadata on LITELLM_METADATA_ROUTES, after which
apply_key_tags_pre_auth and _tag_max_budget_check both targeted
litellm_metadata, leaving header tags stranded in metadata and invisible
to per-tag budget enforcement on Bedrock and other matching routes.

Extract the pre-seed into LiteLLMProxyRequestSetup.pre_seed_litellm_metadata_for_route
and invoke it before apply_client_tag_policy_pre_auth so all tag merges
and the budget-check read agree on the same metadata key.

* test(auth): guard early litellm_metadata pre-seed wiring for header tags

Bugbot's autofix added a pre-seed of litellm_metadata in
_run_centralized_common_checks before apply_client_tag_policy_pre_auth, so
x-litellm-tags header tags land in litellm_metadata and stay visible to
_tag_max_budget_check on LITELLM_METADATA_ROUTES. Its test replayed that call
order in the test body, so removing the production call site still passed.

Add a wiring-level regression that drives the real _run_centralized_common_checks
and asserts header tags land in litellm_metadata (not metadata) for bedrock and
/v1/messages. Dropping the pre-seed call site now fails the test.

---------

Co-authored-by: Zang Peiyu <[email protected]>
Co-authored-by: Cursor Agent <[email protected]>

…riAI#31009)

GET /model/info returned an empty list for a team key whose team only has team-scoped BYOK deployments, even though /v1/models and a master key both returned them. _get_caller_byok_team_scope resolved the caller's allowed teams only from user_api_key_dict.user_id and the bound user's team memberships. A team or service key has user_id=None, so the helper returned an empty set and _byok_row_outside_caller_teams then dropped every team BYOK row

This includes the key's own team_id in the allowed-team set across every non-admin branch, since a team key is authoritatively scoped to its team regardless of whether a bound user is resolvable or a formal member of that team

Completes the work in BerriAI#30025, which aligned /v1/model/info with router deployments but missed the team-key case in the scope helper it introduced

…erriAI#30867)

AWS auth parameters in the Bedrock and SageMaker path could be expanded against
the process environment when credentials were built. Config-sourced references
are already expanded at load time, so restrict expansion to that path: a
reference still present at request time is treated as caller-supplied input and
is left as-is, and the web-identity helper rejects environment-variable
references before resolving the token.

Also rework the ambient AWS_* fallback as a single pass that pairs each value
with its own env-var name, fixing a latent index misalignment that left
AWS_EXTERNAL_ID unresolved.

Adds regression tests covering the resolution behavior.

…el (BerriAI#31029)

* feat(mcp): scope a key to zero MCP servers with no-mcp-servers sentinel

A key under a team that has MCP servers had no way to opt out of them;
an empty list has always meant "inherit the team". This adds a
no-mcp-servers sentinel (mirroring no-default-models for models) so a key
can declare an explicit zero that overrides team inheritance, additive
grants, and allow_all_keys servers, surfaced as an exclusive "No MCP
Servers" option in the key create/edit UI.

* refactor(ui): centralize no-mcp-servers sentinel in a shared constant

The sentinel string was defined under two different local names and
inlined in two more files; a single exported constant removes the drift
risk flagged in review.

* fix(mcp): enforce no-mcp-servers sentinel on toolset-scoped routes

Toolset scoping replaced a key's mcp_servers with the toolset's servers,
dropping the no-mcp-servers sentinel, so a key opted out of all MCP could
still execute a granted toolset's tools via /toolset/{name}/mcp. Deny
toolset access when the key carries the sentinel, checked before the admin
branch to match get_allowed_mcp_servers.

BerriAI#31034)

The Add Model provider dropdown is driven by provider_create_fields.json
(served at /public/providers/fields), and Bedrock Mantle had no entry, so it
could not be selected even though the backend provider, its models, and the UI
enum/logo mappings already existed.

Add a bedrock_mantle entry exposing the credential fields the provider actually
honors: an optional bearer api_key for BYOK, the AWS SigV4 chain, a region, and
an api_base override. Selecting it now populates the bedrock_mantle models from
the cost map via the existing getProviderModels filter.

Also resolve the provider logo when getProviderLogoAndName is given the enum key
(e.g. BedrockMantle) rather than the slug, which the dropdown passes; previously
only slugs that lowercase-matched their key (like bedrock) resolved a logo.

…/v1/mcp/tools (BerriAI#31031)

* feat(proxy): allow llm_api_routes virtual keys to list MCP tools via /v1/mcp/tools

GET /v1/mcp/tools returns the MCP tools available to the calling key, the same
data already exposed through /mcp/tools/list and /mcp-rest/tools/list, both of
which are in llm_api_routes. The /v1/mcp/tools path was in no route group, so
virtual keys created from the UI (which default to allowed_routes=["llm_api_routes"])
got a 403 listing tools one way but not the other.

Add it to mcp_inference_routes. Unlike /v1/mcp/server, this path has no
management write counterpart, so it does not need the method-aware carve-out
used for server discovery.

* test(proxy): parametrize MCP inference route check over the full endpoint set

…AI#31037)

The request logs table column displayed "Key Name" while its accessor
(metadata.user_api_key_alias) and the corresponding filter both use the
"Key Alias" label; this aligns the column header with that naming.

…I#30317)

The login flow stores a post-login return URL in the litellm_return_url
cookie (5 minute TTL). globalSetup snapshots cookies into the per-role
storageState that every spec reuses, so when the snapshot races ahead
of the app consuming that cookie, each test inheriting it gets
redirected to the stale URL (/ui/?login=success) mid-assertion the
first time it mounts a page. That one rogue navigation is behind the
recurring e2e failures whose call logs all show "navigated to
/ui/?login=success" while waiting for an element; which specs die
varies run to run with snapshot timing. Clear the cookie right before
saving the snapshot so no test starts with a pending redirect.

* docs: add MCP server change guidelines

* Add outbound_credentials directory and update AGENTS.md

Updated AGENTS.md to include new outbound_credentials directory and its files, along with additional comments on existing files.

---------

Co-authored-by: tin-berri <[email protected]>

…nthropic streams (BerriAI#31035)

Streaming and pass-through requests could be logged with $0 cost or dropped from
SpendLogs entirely while the upstream provider still billed every token. This
closes the leak paths not already covered by BerriAI#30160, BerriAI#30787 and BerriAI#30788.

- Catch a stream_chunk_builder raise in the core CustomStreamWrapper (sync and
  async). Large agentic tool-use / thinking streams can make assembly re-raise
  as APIError from inside the except-StopIteration handler, where the sibling
  except does not catch it, so it escaped __next__/__anext__ and dropped the
  request; recover best-effort usage from the raw chunks instead
- Add a usage-only fallback for Anthropic streaming pass-through: when
  stream_chunk_builder returns None or raises, rebuild usage from the
  message_start / message_delta SSE events via AnthropicConfig.calculate_usage so
  cache, web-search and geo tokens are priced instead of left at $0
- Decode buffered pass-through bytes with errors="replace" so a stream cut
  mid-multibyte-sequence still logs the usage events already received
- Record response_cost into model_call_details on the pass-through success path
  (it is read from there, not from kwargs), matching the gemini/cohere/openai
  handlers
- Name the key (alias + masked key) in the virtual-key BudgetExceededError so
  operators don't have to reverse-map spend back to a key

…legacy completions) (BerriAI#31046)

* fix(ui): clarify OpenAI-compatible provider dropdown labels (chat vs legacy completions)

* style: change labeling to be clearer

…ct (BerriAI#31045)

POST /team/new with any budget_limits returned 500 because
jsonify_team_object serialized members_with_roles but left budget_limits
as a raw Python list, which Prisma's Json column rejects. /team/update
and /key/generate worked only because each json.dumps the windows itself.
Serialize budget_limits in the shared helper, guarded by isinstance(list)
so the pre-serialized /team/update path is unaffected.

…ary (BerriAI#31054)

Realtime websocket sessions emit events outside the OpenAIRealtimeEvents
union (e.g. rate_limits.updated, response.function_call_arguments.delta,
surfaced when logged_real_time_event_types="*"). Building
LiteLLMRealtimeStreamLoggingObject revalidated every stored event against
the 16-member union, producing thousands of ValidationErrors per session
(12,670 for a ~281-event session). That synchronous work blocked the
asyncio event loop, degrading realtime time-to-first-audio and dial latency
and tripping readiness probes, and the raised error discarded the session
usage so no cost was tracked.

Type results as SkipValidation[OpenAIRealtimeStreamList] and serialize the
events verbatim, so already-formed event dicts are not revalidated. The
flood drops from 12,670 errors to 0 and the combined usage survives to the
cost calculator.

Resolves LIT-3919
Resolves LIT-3920

…erriAI#31047)

* feat(mcp): scaffold outbound_credentials package with typed Result

PR1 of the MCP v2 outbound-credential migration. Adds the
litellm/proxy/_experimental/mcp_server/outbound_credentials/ subpackage with a
hand-rolled Ok | Error Result union (pure stdlib + typing_extensions, no new
dependency) and its package surface. Nothing imports this on a live request path
yet, so production behavior is unchanged; later PRs add the typed config
vocabulary, the resolve_credentials dispatch, and the v1 graft.

* feat(mcp): add outbound_credentials typed vocabulary (BerriAI#31049)

PR2 of the MCP v2 outbound-credential migration, stacked on the result.py
scaffolding. Adds types.py (the AuthConfig discriminated union over seven frozen
per-mode configs, CredError as an expression @tagged_union, Subject, ServerSpec,
and the parse_auth_spec_kind boundary parser) and httpx_auth.py (NoOpAuth,
StaticHeaderAuth). Pulls in expression>=5.6.0,<6.0 on the proxy extra for the
tagged union. Construction-time tests prove illegal mode/field combinations are
rejected. Nothing is wired onto a request path yet; the resolver lands next.

…g deployments (BerriAI#31021)

* fix(router): isolate all per-deployment pricing overrides from sibling deployments

CustomPricingLiteLLMParams is the authoritative set of per-deployment pricing
fields, used to strip overrides from the shared backend-alias key so one
deployment cannot pollute a sibling that shares the same backend model. It had
drifted from ModelInfoBase: tiered and per-unit cost fields such as
input_cost_per_token_above_272k_tokens, cache_read_input_token_cost_above_*,
output_vector_size, ocr_cost_per_*, and the regional uplift multipliers were
absent, so a deployment overriding any of them leaked the override into
litellm.model_cost under the shared key and every sibling read the wrong rate
via /model/info (LIT-3897).

Add the missing fields so the denylist covers every ModelInfoBase pricing
field, and guard against future drift with a test asserting the two stay in
sync, plus a regression test that a tiered override stays isolated to its own
deployment model_id key.

* chore(ui): regenerate schema.d.ts for custom pricing fields

…tool calls (BerriAI#31011)

user_api_key_auth raises ProxyException, not HTTPException, on an auth failure.
The streamable-HTTP and SSE MCP handlers only re-raised HTTPException to preserve
status and headers, so a ProxyException fell through to the catch-all and was
flattened to a generic 500, dropping the real status (for example 401) and any
WWW-Authenticate challenge. MCP clients render a 500 on the JSON-RPC POST as a
cancelled or terminated session, and an OAuth client never receives the 401 it
needs to re-authenticate. Because auth runs before server routing, one rejected
credential fails every targeted server at once.

Map ProxyException back to its real status and headers in both handlers
(handle_streamable_http_mcp, handle_sse_mcp) via a small
_proxy_exception_to_http_exception helper inserted before the generic
except Exception. A genuine auth failure now returns its real status; a key sent
without the documented Bearer prefix gets a clear 401 telling the caller to fix
the header rather than a cancelled session.

Regression tests assert that a ProxyException(401) raised during auth propagates
as a 401 with WWW-Authenticate from both the streamable-HTTP and SSE handlers,
and unit-test the converter for the 401/403/non-numeric-code cases.

… basedpyright can analyze it (BerriAI#30813)

* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (BerriAI#29693)

* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens

* test: scope local cost map env var with monkeypatch to avoid test pollution

* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (BerriAI#30764)

* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold

_mask_value did partial reveal by showing the first visible_prefix and last
visible_suffix characters, but for a value whose length was at or below
visible_prefix + visible_suffix (8 by default) it returned the value verbatim.
A value of exactly 8 chars fell through the length guard and computed
masked_length == 0, reconstructing the original string with no mask characters;
anything shorter hit the early return. Either way short credentials were emitted
in plaintext.

mask_dict routes real secrets through this path, so an 8-char-or-shorter redis
password, api key, or token could be written to logs and the UI unmasked. The
sibling helper mask_sensitive_keys already guards this case; _mask_value now does
the same by fully masking any value at or below the threshold.

* fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers

Fully masking short values is the right default for secret masking, but
CooldownCache reuses the masker purely to truncate exception messages to the
first 50 characters, and it relies on short messages being returned readable.
Masking those blanked out short exception text and broke its tests.

Add a mask_short_values flag (default True, secure) and have CooldownCache pass
False so it keeps the truncation behavior, while every secret-masking caller
still gets short values fully masked.

* fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview

MCPDebug uses the masker to preview auth tokens in debug headers and documents
that values of 10 chars or fewer are shown unchanged so token types stay
distinguishable. Pass mask_short_values=False so that diagnostic behavior is
preserved while secret maskers keep masking short values.

* fix(mcp_debug): mask short auth values in debug headers instead of echoing them

Earlier this masker opted out of short-value masking to keep a token preview, but
that echoes short authorization and token values verbatim in debug response
headers, which is the same leak this change is meant to close. Auth material
should never be emitted in full, so mask short values here too; the first/last
character preview still applies to longer tokens. Only CooldownCache keeps the
opt-out, since it truncates exception text rather than masking secrets.

* test(mcp_debug): assert masked short value preserves length

* refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (BerriAI#30917)

Fireworks AI deprecated audio inference on 2026-06-10
(https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation).
Live API testing confirms the endpoint is already non-functional: a valid
Fireworks API key receives HTTP 401 "Unauthorized" from
api.fireworks.ai/inference/v1/audio/transcriptions for every request,
regardless of payload. The audio-prod.api.fireworks.ai host referenced in
the test suite returns 401 for every path; the entire host is decommissioned.

Remove the dead FireworksAIAudioTranscriptionConfig class and every
reference to it across the codebase:

- Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line
  config class that inherited from OpenAIWhisperAudioTranscriptionConfig)
- Remove the Fireworks branch from
  ProviderConfigManager.get_provider_audio_transcription_config() in
  litellm/utils.py; update the stale comment in
  get_optional_params_transcription that referenced fireworks ai
- Remove the FireworksAIAudioTranscriptionConfig entries from
  LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in
  litellm/_lazy_imports_registry.py
- Remove the TYPE_CHECKING re-export in litellm/__init__.py
- Remove the transcription branch in the fireworks_ai case of
  get_supported_openai_params() in
  litellm/litellm_core_utils/get_supported_openai_params.py
- Remove the whisper-v3 and whisper-v3-turbo entries from
  model_prices_and_context_window.json and
  litellm/model_prices_and_context_window_backup.json (both had
  mode: audio_transcription and zero-cost pricing)
- Remove the TestFireworksAIAudioTranscription test class and its
  imports from tests/llm_translation/test_fireworks_ai_translation.py

No other provider is affected. The openai_compatible_providers list,
FireworksAIMixin, and the OpenAI Whisper transcription handler all stay
because they are shared with other Fireworks endpoints and other
providers. The provider_endpoints_support.json registry already had
audio_transcriptions set to false for fireworks_ai.

* feat: add darkbloom provider (BerriAI#30876)

* feat: add darkbloom provider

* fix: document darkbloom provider endpoints

* fix: address darkbloom review feedback

* fix: update darkbloom tool metadata

* fix: fail fast for non-Postgres database URLs (BerriAI#30883)

* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup

LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.

Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.

Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.

Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.

Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.

* fix: resolve CI failures and proxy DB URL typing issue

* fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging

* Validate DIRECT_URL alongside DATABASE_URL startup guards

* fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (BerriAI#24608) (BerriAI#30946)

* fix(bedrock): surface modeled HTTP status for mid-stream error events (BerriAI#24608)

* test(bedrock): mid-stream server errors trigger streaming fallback (BerriAI#24608)

* style(bedrock): black-format stream-error helper (BerriAI#24608)

* fix(mcp): re-land native tool preservation with typed annotations (BerriAI#30645)

* fix(mcp): preserve native tools in semantic filter hook with typed annotations

* fix(mcp): tighten _is_mcp_tool Chat Completions shape check

* fix(sambanova): return embeddings supported params instead of dropping them (BerriAI#30937)

* fix(router): send fallback metadata when streaming (BerriAI#30914)

When a streaming request triggers a fallback, there was previously no way to
know it happened. This commit addresses this in a few ways:

1. The response now correctly populates the fallback headers
    (`x-litellm-attempted-fallbacks`) so callers know a fallback happened.
2. The correct model ID is passed in the streaming chunks.
3. A streaming chunk with the fallback error can be optionally sent back
    to the client (opt-in) by passing `include_fallback_errors: true` in
    the request.

The format of the fallback errors while streaming is intentionally OpenAI
compatible to not break existing libraries that parse these events. It was
tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not
delieved unexpectedly to callers by default.

* fix(mistral): drop output-only reasoning fields from input messages (BerriAI#30884)

LiteLLM attaches reasoning_content and thinking_blocks to assistant
responses. Replaying those assistant turns verbatim forwarded the fields
back to Mistral, whose input schema forbids unknown keys, so the whole
request failed with a 422 extra_forbidden and reasoning models became
unusable across multiple turns.

Strip both fields from assistant messages before the request is built, in
a spot that runs ahead of the image/file branch so it applies on every
path. Fixes BerriAI#30835

Co-authored-by: Cursor <[email protected]>

* fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (BerriAI#30652)

* fix(perplexity): bill search queries at the per-request price, not 1/1000

The fallback cost calculator divided search_context_cost_per_query by
1000, but that field stores the per-request price in USD: sonar is
{low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published
$5/$8/$12 per 1,000 requests expressed per request. The gemini cost
calculator reads the same field per request with no division (its
docstring calls it "the per-request cost").

The division understated search cost by 1000x on every Perplexity call
that falls back to manual calculation (i.e. when the API does not return
a pre-computed usage.cost). Use the value directly.

Update the tests that had encoded the /1000 factor in their expectations,
and drop an unused import flagged by ruff in the touched test file.

* test(perplexity): update integration test search-cost expectations to per-request

The integration tests still encoded the old /1000 search-cost factor, so
they failed once the fallback calculator was corrected to bill
search_context_cost_per_query per request. Update the four expected-cost
computations (and the high-volume dollar-value comments) to match.

* test(perplexity): drop unused mock imports flagged by ruff

* fix: include model_access_groups when expanding all-team-models in get_team_models (BerriAI#30622)

* fix(fireworks_ai): return None for transcription in get_supported_openai_params

Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is
decommissioned. Without an explicit transcription branch, requests with
request_type='transcription' fell through to the else and returned
FireworksAIConfig chat-completion params. Return None instead to signal
the provider does not support transcription.

* fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting

Without an operator gate, any authenticated caller could set include_fallback_errors=True,
trigger a fallback, and read raw upstream exception messages from the
x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event.

Strip include_fallback_errors from request data in common_processing_pre_call_logic
when expose_fallback_errors_to_caller is not set, so the router never builds the
error list. Also gate _should_include_fallback_errors on the same setting as a
secondary check for the streaming SSE injection path.

* test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test

The operator gate added in e7ff3e1 means include_fallback_errors is only
honoured when general_settings.expose_fallback_errors_to_caller is True.
Set that flag via monkeypatch in the test that exercises the emit path.

* test(prompt_templates): make test_convert_url hermetic instead of hitting picsum.photos

test_convert_url called convert_url_to_base64 against a live picsum.photos
URL and asserted nothing, so it added no real signal and broke CI whenever
the host was unreachable (it was returning 522 and blocking this branch).
Replace the live call with a mocked HTTP client and assert the produced
base64 data URL, so the conversion path is exercised deterministically with
no network dependency. This suite runs under VCR, which is why a transport
level mock (respx) does not reliably intercept; mocking the client object
itself is robust regardless.

* fix(interactions): drop role from Interaction response to match Google spec

Google removed the output-only role field from the Interaction schema (it
now lives only on Turn), so the live OpenAPI compliance canary started
failing with 'role' not in spec. Reconcile our generated types by removing
role from Interaction, CreateModelInteractionParams, CreateAgentInteractionParams
and from the LiteLLM InteractionsAPIResponse/InteractionsAPIStreamingResponse,
stop stamping role=model in the responses-to-interactions transformation, and
update the compliance and integration tests accordingly. Turn.role is kept
since the spec still defines it.

* fix: align all-team-models sentinel access

* fix(router): forward include_fallback_errors through multi-hop fallbacks

run_async_fallback received include_fallback_errors as an explicit named
parameter, so it was bound out of **kwargs and never reached the nested
async_function_with_fallbacks call. Multi-hop fallback chains (a fallback
group that itself fails over) therefore stopped collecting fallback errors
beyond the first hop when a caller opted in. Re-inject the flag into kwargs
before the nested call so inner hops keep accumulating errors, which
add_fallback_headers_to_response already merges across levels.

---------

Co-authored-by: Srivatsa Kamballa <[email protected]>
Co-authored-by: Ahmad Shahzad <[email protected]>
Co-authored-by: Jeremy Chapeau <[email protected]>
Co-authored-by: KRISH SONI <[email protected]>
Co-authored-by: Kent <[email protected]>
Co-authored-by: Ayush Shekhar <[email protected]>
Co-authored-by: dav nguyxn <[email protected]>
Co-authored-by: Tal Marian <[email protected]>
Co-authored-by: Hemant K <[email protected]>
Co-authored-by: Cursor <[email protected]>
Co-authored-by: Yash Raj Pandey <[email protected]>
Co-authored-by: Zang Peiyu <[email protected]>
Co-authored-by: mateo-berri <[email protected]>

BerriAI#31103)

…30446)

* feat: add opensandbox sandbox provider

* fix: harden opensandbox sandbox startup

* fix: address opensandbox review feedback

* fix: address opensandbox sandbox review feedback

* fix: address sandbox parser nits

* fix(ci): clear opensandbox gates

* fix(review): require opensandbox api base

* chore(ci): rerun pass-through check

PR3 of the MCP v2 outbound-credential migration, stacked on the typed vocabulary.
Adds resolver.py: UpstreamCredentialProvider.resolve_credentials dispatches on the
declared AuthConfig variant with one arm per mode, a wildcard-free match plus an
assert_never tail so a missing arm fails basedpyright's exhaustiveness gate. Every
arm is a not_implemented stub returning a typed CredError; each mode's real body and
seam land in follow-up PRs. Pure v2, no v1 imports, nothing wired onto a request path.

…te COUNT(*) (BerriAI#31423)

The spend-logs UI list endpoint (/spend/logs/ui, /spend/logs/v2) ran a standalone
SELECT COUNT(*) before the page query to compute total_pages. On sharded engines
like YugabyteDB a COUNT(*) is a distributed RPC that contacts every tablet leader
and aggregates partial results regardless of row count, so it hits the distributed
RPC timeout and the logs tab 500s even on a one-minute window with a couple of rows.
The startTime range cannot prune tablets because rows hash to tablets on request_id,
not startTime.

Fold the count into the same scan as the page data with COUNT(*) OVER () and read
total off the returned rows, dropping the helper column before serialisation. One
distributed scan per page load instead of two; the response shape is unchanged. An
empty page carries no count row, in which case the total is zero.

Resolves LIT-4027

…ken (BerriAI#31412)

When STS rejects a web identity token with InvalidIdentityToken (the
"Incorrect token audience" case), litellm propagated the raw botocore
error, which never names the aud LiteLLM actually sent. Diagnosing an
audience mismatch then required enabling LITELLM_LOG=DEBUG on the prod
instance, which degrades performance.

_auth_with_web_identity_token now catches InvalidIdentityTokenException,
decodes the public aud/iss claims of the resolved JWT without verifying
its signature (no secret is read), and raises an AwsAuthError that
preserves the STS reason and names the token audience and issuer, so the
mismatch is visible from the error alone.

Resolves LIT-4026

…cy (BerriAI#31414)

litellm_guardrail_latency_seconds was only emitted for pre-call guardrails.
during_call_hook and post_call_success_hook ran guardrails without recording
any latency, so during-call and post-call guardrail time was invisible in the
metric and leaked into litellm_overhead_latency_metric, making the documented
"subtract guardrail latency from overhead" workaround under-report total
guardrail time.

Extract the find-the-PrometheusLogger-and-record step into _emit_guardrail_metrics
and add _run_guardrail_with_metrics, a single wrapper that times a guardrail
coroutine, classifies its outcome (success / intervened / error), enriches any
raised HTTPException, and records the latency under the given hook_type. Route
the pre-call emit, during_call_hook, and post_call_success_hook through it so
every guardrail phase contributes to the metric the same way.

Resolves LIT-3999

…erriAI#31418)

The auth builder cached the team object under the raw `valid_token.team_id`,
while `get_team_object`, `_cache_team_object`, and `_update_team_cache` all read
and write under `team_id:{id}`. The raw-key write was therefore never served
back, and on a non-team (personal) key, whose team_id is None, the original
unguarded version passed a None key straight to the cache layer; the in-memory
cache tolerates None keys but Redis rejects them with a NoneType key error, so
with `enable_redis_auth_cache: true` the team object never reached the L2 cache
and every request fell back to Postgres.

Write under the canonical `team_id:{id}` key, keeping the existing guard that
skips the write when team_id is None. Add a regression test that drives the real
auth builder for a team-scoped key against an in-memory cache and asserts the
team object is served back under `team_id:{id}` and never under the raw team_id
or a None key.

Resolves LIT-4000

…_python_wheel

fix(build): restore pure-Python uv_build backend to unblock PyPI publish

chore(ci): promote internal staging to main

…rail attachment (BerriAI#31421)

When a guardrail blocked a request through a flow-builder policy pipeline, the
proxy discarded the guardrail's own exception and synthesized a generic
guardrail_pipeline_error response, so the same guardrail produced a different
HTTP response and trace span depending on whether it was attached directly or
via a policy. The pipeline now carries the guardrail's original exception and
re-raises it verbatim on block, enriching it with the blocking guardrail's name
and mode exactly as the direct path does, so the two attachment methods are
indistinguishable to clients and tracing. The generic pipeline error remains
only as a fallback for blocks with no underlying exception (e.g. a guardrail
that could not be found).

Resolves LIT-4041

…rics (BerriAI#31410)

litellm_spend_metric_total and litellm_requests_metric_total previously
exposed only the resolved backend model_id and friendly model name, so
operators could not group spend or request counts by the model alias the
caller actually asked for when a router fronts multiple deployments
behind one name.

This adds the existing UserAPIKeyLabelNames.REQUESTED_MODEL to both
labelname lists; the value is already populated upstream from
standard_logging_payload["model_group"] and flows through the shared
_increment_top_level_request_and_spend_metrics call site. The sibling
token metrics (input/output/total) already carry the label, so this
also restores cross-metric consistency.

Resolves LIT-3796

BerriAI#31478)

The Add Model form's getProviderModels rolled any litellm_provider that
starts with the selected provider's slug into that provider's list. Because
"bedrock_mantle".startsWith("bedrock_") is true, bedrock_mantle/* models
(OpenAI-compatible, served at bedrock-mantle.{region}.api.aws) showed up
under plain Amazon Bedrock, where that model string routes to bedrock-runtime
and fails.

Exclude standalone sub-providers from the prefix rollup so bedrock_mantle/*
only appears under the Amazon Bedrock Mantle provider, while bedrock_converse
and other genuine sub-variants keep rolling up under Bedrock.

The MCP client logged the full tool arguments (and prompt arguments) at INFO on every call, so caller input such as user queries, model names, and instructions landed in the proxy application logs and any downstream log aggregator

Log only the tool or prompt name and drop the arguments from these INFO lines

…iry-aware cache, single-flight refresh (BerriAI#31275)

* feat(mcp): let CredError.of_unauthorized carry a 401 challenge

The unauthorized case becomes a structured Unauthorized (detail + optional WWW-Authenticate
header + optional structured body) instead of a bare string, and raise_public emits the header
and body when present. This lets a mode reproduce a rich 401 challenge (e.g. BYOK's
provisioning prompt) through the generic resolver edge. of_unauthorized's new params are
keyword-only and default to None, so existing callers and the summary string are unchanged.

* fix(mcp): make Unauthorized a frozen dataclass to keep the type budget flat

CredError's unauthorized payload was a pydantic BaseModel, whose base resolves as unknown in
this repo's basedpyright (every model in the file trips reportUntypedBaseClass plus an unknown
model_config), so the tagged-union case read as unknown and the public edge's challenge access
added reportUnknownMemberType errors over the per-rule ceiling. A frozen dataclass is fully
typed here, so error.unauthorized resolves directly with no cast or accessor and the per-rule
basedpyright counts match base.

* feat(mcp): OAuth token store seam + expiry-aware cache for authorization_code

Lay the foundation for the authorization_code resolver arm: OAuthToken (access_token,
expires_at, refresh_token), the OAuthTokenStore Protocol seam, TokenStoreUnavailable for
outages, and CachedOAuthTokenStore, an expiry-aware cache that serves a token only while
unexpired, caches the "not authorized" None for a default TTL, and propagates a store outage
without caching it. Mirrors the BYOK store/cache pattern, adapted for tokens. Refresh and
distributed single-flight are deferred to the hardening step.

* feat(mcp): proactive token refresh with self-cleaning single-flight

Add TokenRefresher (a mode-supplied seam: mint a fresh token from an expired one and persist it)
and RefreshingTokenStore: when the stored token is near expiry, the first caller refreshes while
concurrent callers await the same in-flight task and share its result, so the IdP is not
stampeded. The task self-cleans (a done-callback drops its entry), so the map is bounded by
in-flight refreshes rather than by distinct users/servers, and is detached from the caller so a
cancelled caller does not abort the refresh. An expired token the refresher cannot renew surfaces
as None so the arm challenges, never a stale bearer; it composes under CachedOAuthTokenStore.
OAuthToken's repr masks the access/refresh tokens so a stray log cannot leak them. Cross-replica
single-flight (Redis) and reactive-401 refresh are the later distributed hardening.

* style(mcp): modern type annotations (dict/tuple/X | None) + sorted imports in the token modules

* refactor(mcp): FIFO cache eviction, fix stale single-flight comment + refresh_token docstring

* refactor(mcp): cache positive tokens only, matching v1 (no negative caching)

CachedOAuthTokenStore no longer caches the "not authorized" None result; every miss re-reads the
inner store. v1's per-user token cache never caches misses, so a token written by the OAuth flow
is visible on the next request without an invalidation hook, and uniformly across replicas since
the in-process cache holds no stale None to clear. invalidate() now only covers rotation or
revocation of a cached token. Negative caching (with distributed invalidation) can return later
if a slow DB-backed v2-native source makes per-miss reads expensive.

* fix(mcp): default OAuth expiry skew to 60s, the industry standard

The proactive token-refresh / cache-expiry buffer defaulted to 30s, which is
an outlier among OAuth clients. Spring Security uses 60s as both its JWT
clock-skew tolerance and its refresh buffer, and 60s sits inside RFC 7519's
"a few minutes" leeway while preserving nearly all of a typical token's life;
30s was untested, so pin the default with two boundary-probe regression tests.

* refactor(mcp): thread user_id/server_id through the TokenRefresher seam

The refresh seam took only the OAuthToken, but a refresher needs the server's
config (token endpoint, client credentials, scopes) to run the grant and the
(user_id, server_id) key to persist the minted token, neither of which is
derivable from the token. Widen TokenRefresher.refresh to (user_id, server_id,
token) and pass them through from RefreshingTokenStore so each stacked mode PR
plugs into the final seam rather than forcing a later signature change across
the stack.

* feat(mcp): inject cache-backend and refresh-coordinator seams (cross-replica token caching)

Make CachedOAuthTokenStore's storage and RefreshingTokenStore's single-flight injectable so a
cross-replica deployment can back them with Redis without touching the resolver. The defaults preserve
today's behavior exactly: InMemoryTokenCacheBackend (the bounded per-process dict) and
InProcessRefreshCoordinator (the asyncio single-flight). A distributed deployment injects a shared
DualCache-backed backend and a SET NX PX coordinator. invalidate() is now async (the backend may be).
The cache stores via the backend with a TTL derived from the token's expiry; the coordinator threads a
reread callback for the cross-replica case (losers re-read the persisted token) that the in-process
default ignores.

* fix: reread oauth token before refresh

---------

Co-authored-by: Cursor Agent <[email protected]>

…erriAI#31485)

Pass-through success logging was scheduled with a bare asyncio.create_task
whose return value was discarded, for non-streaming HTTP, streaming, and the
vertex live websocket paths. The event loop keeps only a weak reference to such
tasks, so under GC or load the task can be collected before it finishes writing
the SpendLogs row; a request then returns 2xx to the caller yet never produces a
costed spend log. This is the most likely cause of the flaky vertex passthrough
e2e test and a rare real source of unbilled pass-through spend.

Route these coroutines through GLOBAL_LOGGING_WORKER.ensure_initialized_and_enqueue
instead, matching how the SDK completion path already enqueues async logging. The
worker holds a strong reference in its _running_tasks set and drains on shutdown
via flush/stop/clear_queue and the atexit handler, so the write can no longer be
dropped mid-flight.

…ng requests (BerriAI#31484)

* fix(websearch): wrap agentic loop response in fake stream for streaming requests

When websearch_interception converts stream=True to stream=False internally,
the agentic loop returns a plain dict. Previously this dict was returned
directly to the client expecting SSE events, resulting in empty streams.

Added _maybe_wrap_in_fake_stream() which checks the
websearch_interception_converted_stream flag and wraps dict responses in
FakeAnthropicMessagesStreamIterator. Applied to all return paths in
_call_agentic_completion_hooks:
- async_run_agentic_loop (legacy path)
- _execute_anthropic_agentic_plan (plan-based path)
- plan.response_override
- plan.terminate

Includes unit tests for _maybe_wrap_in_fake_stream().

* test(websearch): cover agentic-loop wrap paths; gate fake-stream on anthropic_messages surface

Guard _maybe_wrap_in_fake_stream on api_surface == anthropic_messages so the
responses API surface is never wrapped in an Anthropic SSE iterator, and type
logging_obj as Optional to match the None call sites. Adds regression tests
that drive the legacy, response_override, and terminate return paths of
_call_agentic_completion_hooks end to end.

* test(websearch): cover _execute_anthropic_agentic_plan and tail wrap paths

Drives the remaining two fake-stream return paths of
_call_agentic_completion_hooks (the _execute_anthropic_agentic_plan branch via
a stubbed handler, and the tail path when no agentic loop runs) so every
converted-stream return path is regression-tested.

---------

Co-authored-by: Clawd <[email protected]>

…riAI#31407)

* feat(guardrails): add headroom guardrail for message compression

Adds a headroom guardrail that compresses request messages via POST
/v1/compress before they reach the LLM. The guardrail implements
apply_guardrail so it runs on the unified guardrail path; it receives
pre-built structured_messages (OpenAI format) from the translation
layer, calls the headroom compression service, and returns the
compressed messages as structured_messages.

Set x-headroom-bypass: true on the request to skip compression.

Also adds structured_messages write-back support to the OpenAI and
Anthropic translation handlers: when apply_guardrail returns
structured_messages, those are written to data["messages"] directly
(OpenAI) or reverse-translated via anthropic_messages_pt (Anthropic)
instead of falling through to the existing text-patch path. This is a
prerequisite for any guardrail that needs to replace the full message
list rather than patch individual text spans.

* fix(guardrails/headroom): add @log_guardrail_information to populate guardrail_information in spend logs

* style: fix ruff format violations

* fix(lint): replace deprecated typing aliases with builtin generics (UP006/UP037)

* fix(guardrails): only write back structured_messages when guardrail actually changed them

* fix(guardrails/headroom): raise 502 when compression returns empty message list

* fix(guardrails/headroom): catch transport errors and fix stale debug log

* fix(guardrails/anthropic): strip system messages before anthropic_messages_pt reverse-translation

* fix(guardrails/anthropic): strip cache_control from thinking blocks after write-back

* debug(headroom): add INFO logging to trace guardrail execution

* debug(headroom): use print() for immediate visibility

* debug(headroom): print request_data keys to diagnose metadata dict mismatch

* fix(guardrails/anthropic): propagate guardrail info to logging_obj.metadata for spend log

* fix: use model_call_details litellm_params metadata on Logging object

* fix(guardrails/anthropic): write guardrail info to litellm_params attr not model_call_details copy

* fix: read slg_info from litellm_metadata when metadata key absent

* fix: write slg_info to both litellm_params attr and model_call_details copy

* chore: remove debug prints; fix now verified end-to-end

* refactor(guardrails): move spend-log sync to shared helper in custom_guardrail.py

- Add _sync_guardrail_info_to_logging_obj in custom_guardrail.py; call it from
  both async and sync wrappers in @log_guardrail_information, fixing
  guardrail_information=null in spend logs for all passthrough routes
  (/v1/messages, /v1/responses, etc.) in one place
- Remove the 35-line inline sync block from the anthropic translation handler
- Wrap response.json() in try/except in headroom.py to 502 on HTML/truncated responses
- Drop redundant headers.get(BYPASS_HEADER.lower()) — header key already lowercase
- Add regression tests for _sync_guardrail_info_to_logging_obj

* fix(lint): reduce _sync_guardrail_info_to_logging_obj complexity below C901 threshold

* fix(lint): simplify _sync_guardrail_info_to_logging_obj to reduce McCabe complexity

* fix(lint): extract _append_slg_to_litellm_params to reduce McCabe complexity

* fix(lint): extract _write_back_structured_messages to reduce process_input_messages complexity

…rriAI#31375)

failing test is not related to the pr

* fix(websearch): sync tool_choice when converting web_search tools

Claude Code forces native web search via tool_choice pointing at web_search
while websearch_interception renames the tool to litellm_web_search, causing
Anthropic 400s. Forward tool_choice into pre-request hooks and rewrite forced
tool_choice to match the converted tool name.

Co-authored-by: Cursor <[email protected]>

* fix(websearch): re-wrap agentic loop responses as SSE for streaming clients

When websearch interception converts stream=true to false for the agentic
loop, dict responses from the loop were returned as application/json even
though the client requested SSE. Wrap those responses in
FakeAnthropicMessagesStreamIterator so /v1/messages streaming callers
(e.g. Claude Code) receive text/event-stream after search completes.

Fixes BerriAI#27721

Co-authored-by: Cursor <[email protected]>

* test(websearch): cover tool_choice sync and post-loop SSE wrap; fix UP006

Add regression tests for both websearch interception fixes: _sync_forced_tool_choice
repointing a forced web_search tool_choice to litellm_web_search (the 400 fix) and
_maybe_websearch_fake_stream_wrap re-wrapping agentic loop dict responses as SSE for
streaming clients (BerriAI#27721). Switch the new helper annotations to builtin dict/list so
the ruff UP006 strict-rule ceiling stays within budget.

Co-authored-by: Cursor <[email protected]>

* fix(websearch): resolve merge conflict and unify fake stream wrapping

Remove the duplicate _maybe_websearch_fake_stream_wrap helper left by a bad merge that caused a SyntaxError in CI, and route all call sites through _maybe_wrap_in_fake_stream instead.

Co-authored-by: Cursor <[email protected]>

---------

Co-authored-by: Shivam Rawat <[email protected]>
Co-authored-by: Cursor <[email protected]>
Co-authored-by: Shivam Rawat <[email protected]>

…upload

fix(passthrough): forward all multipart files with repeated field names

…t tracking (BerriAI#31355)

* fix(cost): preserve Anthropic server_tool_use web search usage in cost tracking

Anthropic /v1/messages responses report built-in web search usage under
usage.server_tool_use.web_search_requests, but the sync cost path reconstructs
an OpenAI-shape Usage that drops server_tool_use and validates the response
through AnthropicResponse, which previously stripped the field. Either path
could leave the web-search fee uncounted.

AnthropicResponseUsageBlock now allows extra fields so model_validate/model_dump
keeps server_tool_use, and the built-in tool cost tracker reads the web search
count straight off the raw Anthropic response dict when the reconstructed Usage
lacks it, synthesizing a ServerToolUse without mutating the caller's Usage.

* fix(lint): use PEP 604 unions in anthropic web search probes to satisfy strict-rule budget

* refactor(cost): move Anthropic web search response parsing into llms/anthropic

Relocate the raw /v1/messages web-search-count probe out of the shared
built-in tool cost tracker into litellm/llms/anthropic/cost_calculation.py,
next to get_cost_for_anthropic_web_search, so provider-specific response
parsing lives under llms/. The core cost tracker now delegates to
get_anthropic_web_search_requests_from_response and keeps only the generic
Usage/ServerToolUse orchestration.

* fix(cost): price Anthropic web search when only the raw response carries the count

response_object_includes_web_search_call enters the web search branch as
soon as the raw Anthropic dict reports usage.server_tool_use.web_search_requests,
but _usage_with_anthropic_web_search bailed when the caller did not also pass
a Usage object. _handle_web_search_cost then skipped the per-request anthropic
path and fell back to the flat search_context_size_medium tier, charging a
fixed fee instead of per_query x count (or zero when the count is zero).

Synthesize a Usage from the raw dict when no Usage is supplied so count-based
pricing runs uniformly regardless of how the response reaches the tracker.

---------

Co-authored-by: Cursor Agent <[email protected]>

…31490)

The AgentOps preset hardcoded https://otlp.agentops.cloud/v1/traces, a domain
that no longer resolves (NXDOMAIN), so every span silently failed to export with
a NameResolutionError in the BatchSpanProcessor worker. The live ingest host is
otlp.agentops.ai (the auth host api.agentops.ai was already correct). Pin the
endpoint to the resolvable host and add a regression test on the constant.

…replica) [1/2] (BerriAI#31473)

* feat(mcp): implement the authorization_code resolver arm

Resolve a user's authorization_code token through the injected OAuthTokenStore: present ->
Authorization: Bearer <access_token>; absent -> the RFC 9728 WWW-Authenticate OAuth challenge;
store unavailable -> the same challenge (not a 500), since a transient outage is not a definite
absence. UpstreamCredentialProvider gains the oauth_token_store collaborator (fail-closed null
default); per-subject isolation comes from keying the fetch on subject_id. Not live until
to_server_spec maps authorization_code and a v1-backed token source is wired (next steps).

* feat(mcp): v1-backed OAuth token source for authorization_code

V1PerUserTokenStore reads the user's stored access token through v1's mcp_per_user_token_cache
(Redis-backed, encrypted) and wraps it in an OAuthToken. v1 holds only the access token (its
cache TTL is the lifetime), so no expires_at/refresh_token yet; the v2 cache holds it for its
default TTL and the OAuth challenge drives re-auth once v1's cache drops it. Additive: nothing
wires it yet, so no behavior change. Step 1b swaps it for a v2-native token store behind the
OAuthTokenStore seam.

* style(mcp): modern type annotations in the authorization_code arm and source

* refactor(mcp): share v1's OAuth egress core; make V1PerUserTokenStore refresh-capable

Extract v1's per-user OAuth egress (Redis cache, else DB read with the refresh_token grant, then
re-cache) from _get_user_oauth_extra_headers_from_db into resolve_user_oauth_access_token in db.py;
the v1 header builder is now a thin wrapper over it and its callers are unchanged.
V1PerUserTokenStore (the v2 OAuthTokenStore adapter) resolves through that same core via an injected
server lookup, so the authorization_code arm injects exactly the token v1 would, with the same silent
refresh, rather than a Redis-only read that can never refresh. One resolution implementation, two thin
adapters (header dict and OAuthToken). Behavior-preserving: the existing v1 egress tests pass
unchanged, and the arm is not wired into the live path yet (that lands with to_server_spec + the
manager).

* feat(mcp): route oauth2 per-user (authorization_code) servers through the v2 resolver

to_server_spec maps an oauth2 server to AuthorizationCodeConfig when it relies on per-user tokens
(needs_user_oauth_token and not delegate_auth_to_upstream); client_credentials (M2M), delegated
upstream OAuth, token exchange, and SigV4 still defer to v1. The manager injects V1PerUserTokenStore
(resolving through v1's shared egress core) into the credential provider. The v2 path is live but
still defers to a token v1 places in extra_headers; the cutover that makes v1 step aside lands next,
alongside the unified challenge.

* feat(mcp): per-server fail-closed OAuth challenge at the v2 egress

When an authorization_code server has no usable per-user token, the arm returns a semantic
unauthorized and the graft builds the 401 where the full MCPServer is in hand: a relative,
per-server RFC 9728 resource_metadata pointer (/.well-known/oauth-protected-resource/mcp/{name})
that names the server's own authorization server, instead of the resolver's earlier root pointer
which resolved to the gateway's generic PRM. Relative, so it is correct behind a reverse proxy
without request context. The listing-phase 401 still emits the RFC 8414 authorization_uri form;
both now target the same server, so the remaining difference is cosmetic and unifies in a later PR.

* feat(mcp): cut the call_tool egress over to v2 for authorization_code servers

_resolve_oauth2_headers_for_tool_call steps aside (builds no header) when to_server_spec maps the
server, so the v2 resolver drives the token-present case instead of being shadowed by a token v1
places in extra_headers. Non-migrated oauth2 (delegate, client_credentials) and BYOK still build
their header on v1. With this, v2 owns the authorization_code egress end to end: inject the
refreshed per-user token when present, raise the per-server fail-closed 401 when absent.

* feat(mcp): cut the tools/list connection over to v2 for authorization_code servers

The listing connection's per-user OAuth header is no longer built by v1 for migrated servers; the
v2 resolver drives it at connect time, ending the double-resolution where v1 built the token into
extra_headers and the v2 graft then deferred to it. Safe because the preemptive 401 (in the
streamable-http and SSE handlers) already challenges a missing token before the listing connection
runs, so the connection is only reached with a token present. Non-migrated oauth2 (delegate) and
the rest still build their header on v1. With this, resolve_credentials' result is honored on every
authorization_code upstream path: tool calls and listing.

* feat(mcp): route the preemptive 401 existence check through the v2 resolver

The discovery-phase 401 no longer calls v1's _get_user_oauth_extra_headers_from_db to decide
whether a migrated server has a token; it asks the v2 resolver via a new has_user_oauth_token
manager method (to_server_spec + to_subject + resolve_credentials, Ok means a token exists). With
this, every authorization_code resolution runs through the v2 resolver: the call_tool egress, the
listing connection, and the discovery challenge. Delegate servers short-circuit before the check
(the client completes PKCE with the upstream). The challenge itself still emits the RFC 8414
authorization_uri form; the format unification stays a follow-up.

* refactor(mcp): extract the authorization_code arm into a helper

Mirror the api_key arm's structure: the inline AuthorizationCodeConfig body moves into
_authorization_code(subject, server), keeping resolve_credentials a flat one-line-per-arm dispatch.
The helper is annotated with the concrete StaticHeaderAuth it returns rather than the abstract
httpx.Auth (which api_key uses) because a new method carrying the unresolved httpx.Auth return
would add reportUnknownMemberType; the concrete type is both precise and budget-neutral.

* fix(mcp): emit the canonical WWW-Authenticate header name in the OAuth challenge

raise_user_oauth_challenge emitted the header lowercase while the sibling raise_public and every
resource_metadata (RFC 9728) emitter use the canonical WWW-Authenticate; align it. HTTP header names
are case-insensitive on the wire so this is cosmetic for compliant clients, but it keeps the challenge
builders consistent and matches RFC 6750.

* feat(mcp): v2-native per-user token read store (step 1b inner store)

Reads the user's persisted authorization_code credential and returns a typed OAuthToken (access
token, epoch expiry, refresh token), validating the decoded blob at this boundary so no Any leaks
past it. The raw inner store that RefreshingTokenStore/CachedOAuthTokenStore wrap; the DB read +
decode collaborator is injected so it stays testable. Not yet wired - V1PerUserTokenStore is still
the composition-root store until the refresher and cross-worker cache land.

* feat(mcp): v2-native authorization_code token refresher (step 1b)

The refresh_token grant for the authorization_code mode: POSTs the RFC 6749 refresh_token grant to
the server's token endpoint, persists the rotated triple, and returns the new typed OAuthToken for
RefreshingTokenStore to cache. HTTP post and persist are injected so the grant + response parsing
are testable without a live IdP/DB. Also extends the TokenRefresher seam with (user_id, server_id),
which the foundation's refresh(token) lacked but the grant (server config) and persist (key) need.

* feat(mcp): wire the v2-native per-user OAuth store into the resolver (step 1b piece 4)

Assemble Cached(Refreshing(V2PerUserTokenStore)) at the composition root and replace
V1PerUserTokenStore in mcp_server_manager. The chain is built lazily on first fetch (its cache/DB/
Redis collaborators are LiteLLM globals not ready at import); when Redis is wired it uses the
cross-replica path (DualCache cache + SET NX PX coordinator), else the in-process defaults. The DB
read, refresh-grant POST, and persist acquire their globals per call like v1. authorization_code
resolution now reads/refreshes through the v2-native lifecycle, not v1's core.

* refactor(mcp): delete the unwired V1PerUserTokenStore adapter (step 1b piece 5)

Piece 4 replaced V1PerUserTokenStore with the v2-native chain at the composition root, leaving the
adapter with no callers, so remove it and its test. The shared v1 read/refresh core
(resolve_user_oauth_access_token and friends) stays - delegate's egress in server.py still uses it -
and comes out with the delegate migration.

* fix(mcp): green CI for authz_code dispatch (format + UTC expiry + v2-seam tests)

- ruff format per_user_oauth_store.py (clears the lint check)
- v2_token_store._iso_to_epoch: anchor a tz-naive expiry to UTC before
  .timestamp(), matching v1's db.py _remaining_token_seconds (Greptile P1) so a
  non-UTC host doesn't read the expiry as local time and skew refresh timing
- test_mcp_stale_session: repoint the 3 discovery tests off the removed v1
  _get_user_oauth_extra_headers_from_db onto the v2 has_user_oauth_token seam;
  the delegate test now asserts the existence check is never consulted (delegate
  short-circuits to the resource_metadata 401 before any token lookup)
- test_mcp_server_manager: repoint test_deferred_mode_uses_v1_auth_value at M2M
  (oauth2 client_credentials), which is still a deferred mode, since per-user
  oauth2 (authorization_code) now routes to the v2 resolver

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* fix(mcp): caller Authorization must not override the stored per-user OAuth token

A caller with a valid x-litellm-api-key could include their own
"Authorization: Bearer <chosen>" header and have the proxy execute tools against
that bearer instead of the user's stored OAuth credential. For a v2-migrated
authorization_code server the caller's Authorization was seeded into
extra_headers, and the graft's apply-if-absent then dropped the resolved
per-user token in its favor. v1 prevented this by overwriting a stale client
Authorization with the stored token; this restores that precedence on both
egress paths (connect + call_tool).

- _should_strip_caller_authorization: also strip for migrated per-user OAuth
  (authorization_code) servers - the v2 resolver injects the stored token, so a
  caller-forwarded Authorization must not be forwarded upstream. Delegate /
  pass-through (to_server_spec is None) keep forwarding the caller's bearer.
- both seed sites (_prepare_mcp_server_headers, _call_regular_mcp_tool) drop only
  the Authorization from the caller's oauth2_headers (via _without_authorization),
  keeping any other forwarded header and any hook/static Authorization (which
  still wins, as in v1).
- regression test for the call_tool path; updated the two tests that asserted the
  old (vulnerable) forwarding to assert the secure behavior.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* fix(mcp): preserve recorded OAuth scopes across authorization_code refresh

When a refresh response omits `scope` (RFC 6749 §5.1, where omission means unchanged), the v2 refresher persisted scopes=None and overwrote the user's recorded grant. v1 carried the prior scopes forward via `or cred.get("scopes")`; the v2 path lost that because OAuthToken did not model scopes

OAuthToken now carries scopes, V2PerUserTokenStore populates them on read, and AuthorizationCodeRefresher carries them forward for both the persisted write and the returned/cached token, so repeated refreshes do not erode them. A present `scope` in the response still replaces the prior grant

Adds regression tests: a refresh omitting `scope` preserves the prior scopes, and a present `scope` overrides them

* fix(mcp): keep user token in authorization_code tools preview

After to_server_spec maps oauth2 onto the v2 resolver, the interactive tools preview for an unsaved authorization_code server read the per-user token store, found nothing, and fail-closed with a 401, so the create/test tab could no longer list tools

The preview now routes the just-authorized token (forwarded in oauth2_headers) through mcp_auth_header, so _create_mcp_client takes the per-request-override v1 path and uses it directly, matching v1's preview. Gated to the v2-mapped oauth2 case; M2M, delegate/passthrough, and token-exchange keep their existing preview path

Adds tests: interactive oauth routes the forwarded token to mcp_auth_header, M2M and token-exchange do not

* fix(mcp): stop caller-supplied auth from overriding stored authorization_code tokens

A caller-supplied per-request override (mcp_auth_header / x-mcp-auth / x-mcp-<alias>-authorization) disabled the v2 resolver in _create_mcp_client for any spec, so an authenticated user with a stored authorization_code token could force an arbitrary upstream bearer and bypass the stored credential and its save-time validation. _create_mcp_client now keeps the v2 spec for authorization_code and ignores the override; other modes keep the client-side-credentials override

The create/test tools preview no longer relies on that override path. It resolves the just-authorized, not-yet-persisted token through the v2 resolver via a one-shot PresentedOAuthTokenStore passed as cred_provider - the same path runtime uses for the stored token - so preview and runtime resolve identically. This replaces the mcp_auth_header routing added earlier

Adds tests: a caller override cannot bypass the v2 resolver for authorization_code; the interactive preview resolves via the presented store rather than a caller header; M2M and token-exchange build no presented provider

* feat(mcp): cross-replica single-flight refresh for the v2 per-user OAuth store [2/2] (BerriAI#31474)

* feat(mcp): encrypt+serialize codec for caching OAuth tokens in Redis (step 1b §1.5)

The serialize+encrypt boundary a cross-replica cache needs: a plaintext bearer in Redis is a leak, so
encode() encrypts (NaCl in prod via the injected encrypt, identity in tests). Caches only access_token
and expires_at, never the refresh_token - the hot path needs just the bearer, and the long-lived
refresh_token stays in the DB (the refresh path is always a cache miss), matching v1. A decoded token
always has refresh_token=None. Undecryptable (key rotation) or corrupt entries read as a miss.

* feat(mcp): DualCache-backed token cache backend (step 1b §1.5)

The cross-replica TokenCacheBackend implementation that plugs into the foundation's
CachedOAuthTokenStore seam: encrypts+serializes the token via the codec and stores it in LiteLLM's
shared DualCache under the same per-(user,server) key v1 used, so workers share one refresh and a
token cached by v1 or v2 is readable by the other across the cutover. Cache and codec are injected;
a non-positive TTL (already-expired token) is not cached, and a missing/corrupt entry reads as a miss.

* feat(mcp): Redis SET NX PX refresh coordinator (step 1b §1.5)

The cross-replica RefreshCoordinator that plugs into the foundation's RefreshingTokenStore seam: a SET
NX PX lock elects one worker to refresh per (user, server) while the rest wait for it and re-read the
token it persisted, so a rotating refresh_token is used once across the fleet, not once per worker. The
lock self-expires (PX) so a crashed holder can't wedge refresh; a loser falls back to a bounded re-read
and the surrounding store re-checks expiry next fetch, so a crash self-heals. The lock (a thin Redis
SET NX/DEL/EXISTS wrapper in prod) is injected, so the single-flight logic is testable without Redis.

* feat(mcp): Redis SET NX PX distributed lock (step 1b §1.5)

The concrete DistributedLock the RedisRefreshCoordinator elects refreshers with: acquire is an atomic
SET key NX PX ttl (first caller wins, entry self-expires so a crashed holder can't wedge refresh),
release is DEL, is_held is EXISTS. The async Redis client is injected (the client from LiteLLM's
RedisCache in prod), so it is unit-testable with a fake. Any Redis error degrades to not-acquired /
not-held so a cache blip causes an extra refresh, never a crash on the resolve path.

* feat(mcp): wire the cross-replica cache + coordinator into the per-user store (step 1b §1.5)

Upgrade the composition root to use the DualCache-backed cache and SET NX PX refresh
coordinator when Redis is wired, falling back to the foundation's in-process defaults on a
single replica. Layers the cross-replica path on top of the single-replica dispatch store.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* fix(mcp): refresh on lock-backend error instead of serving a stale token

The cross-replica refresh coordinator elected refreshers with a boolean acquire:
a Redis transport error was caught and returned as False, which is
indistinguishable from "another worker holds the lock". On a total Redis
outage every worker therefore took the wait-then-reread branch and served the
still-expired token upstream (the upstream then 401s), even though the lock and
coordinator docstrings claimed a Redis blip "degrades to an extra refresh".

Make acquire tristate (LockAcquisition: ACQUIRED / HELD / ERROR) so the
coordinator can tell a busy holder from a dead backend, and refresh anyway on
ERROR. This single-flight lock is a load optimization, not a correctness mutex,
so failing open is correct: it degrades a lock-backend outage to the
no-coordinator behavior (an extra refresh), never a stale bearer.

Add a regression test asserting an acquire error refreshes rather than
re-reading the expired token, and update the docstrings to match.

* style(mcp): wrap redis lock signatures at line-length 88 for CI ruff format

* fix(mcp): a refresh loser surfaces None, not a stale token, when the winner failed

The cross-replica coordinator's losers re-read the token the winner persisted.
If the winner's refresh failed, the store still holds the expired token, so the
loser re-read it and RefreshingTokenStore handed that expired bearer to the
caller (the upstream then 401s) instead of the re-auth challenge the winner
returned via None.

Make the loser's re-read expiry-aware, mirroring refresh_latest_token: a
re-read that is still expired surfaces None so the arm challenges. This only
affects the loser path; the winner's freshly refreshed token is returned
directly by the coordinator and is unaffected.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

* Revert "feat(mcp): cross-replica single-flight refresh for the v2 per-user OA…" (BerriAI#31492)

This reverts commit cd2fb6b.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

…rowing every reload (BerriAI#31314)

The 30s add_deployment_job re-runs initialize_pass_through_endpoints, which
re-registers every config/DB pass-through endpoint. Endpoints without a
persisted id get a fresh uuid each cycle, so their route key
("{id}:{type}:{path}:{methods}") changes every reload. The stale-route cleanup
called remove_endpoint_routes(route_key), but that helper matches entries by
endpoint_id, so it never matched a route key and never deleted anything. The
registry grew by one entry per route per reload, turning the per-cycle cleanup
and the per-request is_registered_pass_through_route scan into a CPU sink that
eventually pins a core and slows every endpoint.

Pop the stale key from the registry directly in O(1). openai_routes is left
alone: its append is path-deduped and the path is still owned by the live
endpoint re-registered under a new id in the same cycle.

Resolves PERF-13

…ality_upstream_sync