…ng (BerriAI#18195)

feat: add CometAPI provider support with chat completions and streaming

…ng (BerriAI#18195)

…rvers_url_with_rootpath

fix: fixed the issue of handling root paths when processing Discovery…

…26_2026

fix(proxy): support slashes in google generateContent model names (#1…

…gmnt (#1…" (BerriAI#20393)

This reverts commit 76a399b.

…03_2026

feat(guardrails): implement team-based isolation guardrails mgmnt (#1…

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Veria admin-queue finding E3NpkuAd, Audit-B #1. The route-level gate
accepts this call when the caller is PROXY_ADMIN or ORG_ADMIN of any
org named in request_data["organization_id"]/["organizations"]. The
handler processes data.user_ids without cross-checking whether those
users belong to the caller's administered orgs, so an org-admin of
org-A could delete users in org-B via:
  {"user_ids": ["victim_in_org_B"], "organization_id": "org-A"}

Add per-target authorization: org-admins may only delete users whose
entire org membership is within their admin scope; targets with any
org outside scope (or no org at all) require PROXY_ADMIN.

Regression test confirms an ORG_ADMIN call fails with 403 and no
cascade delete_many runs.

Continuation of Veria E3NpkuAd / Audit-B hardening. All three are the
same anti-pattern PR BerriAI#25904 already addressed for _user_is_org_admin
and /user/delete: route-level gate trusts a caller-supplied scope
field, handler operates on a different scope.

1. /user/update no longer silently creates a user when the target
   email doesn't exist. Pre-fix, an org admin could supply a fresh
   email + caller-chosen budget/models/metadata; the INSERT path
   created the user with no org attachment, bypassing /user/new's
   org/team authorization. Now require PROXY_ADMIN for the create
   branch; return 404 otherwise. Also fixes /user/bulk_update because
   it dispatches through the same _update_single_user_helper.

2. /team/bulk_member_add with all_users=true restricted to PROXY_ADMIN.
   The flag pulls every user in the database into the target team,
   ignoring org scope — any team admin could use it to capture every
   user across every org into their team.

3. /team/update now verifies destination-org admin rights. When the
   request carries an organization_id that differs from the team's
   current org, an org admin of the caller's current org could
   previously relocate the team into any other org (draining their
   resources, or capturing a team they once administered). Require
   PROXY_ADMIN or org-admin of the DESTINATION org for the relocation.

Regression tests for #1 and BerriAI#3; BerriAI#2 covered by the existing bulk_add
suite after the gate addition.

* fix(proxy): allow non-admin virtual keys to call GA Realtime WebRTC HTTP routes (BerriAI#30089)

* fix(proxy): allow non-admin virtual keys to call GA Realtime WebRTC HTTP routes

Add the realtime WebRTC HTTP sub-routes (/realtime/client_secrets,
/realtime/calls and their /v1 + /openai/v1 variants) to
LiteLLMRoutes.openai_routes so is_llm_api_route() classifies them as
LLM API routes. Without this, non-admin virtual keys received
401 'Only proxy admin can be used to generate, delete, update info
for new keys/users/teams' when calling these endpoints.

Fixes BerriAI#29923

* fix(proxy): validate session.model for realtime routes in model-access check

The GA Realtime WebRTC HTTP routes resolve the effective model from the
nested session.model (falling back to the top-level model), but the auth
layer's get_model_from_request() only extracted the top-level model. A
model-restricted virtual key could therefore place a disallowed model in
session.model, leave the top-level model unset, and skip can_key_call_model()
entirely - obtaining an ephemeral token for a model it is not allowed to use.

Extract session.model for the realtime client_secrets/calls routes so the
model-access check runs against the model the request will actually use.
Legitimate callers are unaffected; their permitted model still validates.

Relates to BerriAI#29923

* fix(proxy): classify realtime transcription_sessions routes as LLM API routes

Add the GA Realtime WebRTC transcription_sessions HTTP routes to
openai_routes so is_llm_api_route() returns True for them, matching the
client_secrets and calls routes already fixed. These endpoints are
registered with user_api_key_auth in realtime_endpoints/endpoints.py, so
without this a non-admin virtual key calling
POST /v1/realtime/transcription_sessions would hit the admin-only 401
branch. Extends the regression test parametrization accordingly.

---------

Co-authored-by: habonlaci <[email protected]>

* feat(proxy): surface max_input_tokens/max_output_tokens on /v1/models (BerriAI#30272)

* feat(proxy): surface max_input_tokens/max_output_tokens on /v1/models

* fix(proxy): degrade /v1/models gracefully when model-group lookup fails

---------

Co-authored-by: Sameer Kankute <[email protected]>

* fix: sort tiered token-cost thresholds numerically (BerriAI#30375)

* fix: sort tiered token-cost thresholds numerically

_get_token_base_cost iterated input_cost_per_token_above_<N>_tokens keys with a
lexicographic sort, so for tiers whose thresholds have different digit lengths
(e.g. 90k vs 128k) a request crossing both was billed at the lower tier that
sorted first. Sort by the parsed numeric threshold instead, so the highest tier
the request actually crosses is applied.

* refactor: reuse _parse_above_token_threshold for inline threshold parse

---------

Co-authored-by: Eric (GabiDevFamily) <[email protected]>

* fix(openai): preserve cache_control for openai-compatible custom endpoints (BerriAI#30387)

* fix(openai): preserve cache_control for openai-compatible custom endpoints

* fix(openai): use parsed hostname to detect real OpenAI for cache_control preservation

* fix(proxy): drain all daily-spend batches per flush cycle (BerriAI#30281) (BerriAI#30505)

* fix(types): prevent internal parallel_request_limiter fields from leaking to upstream providers (BerriAI#30545)

* fix(types): add internal parallel_request_limiter fields to all_litellm_params to prevent forwarding to upstream providers

* test(types): add regression test for internal rate-limit fields in all_litellm_params

* fix(init): add bool type annotation to suppress_debug_info (BerriAI#30531)

Module-level `suppress_debug_info = False` had no annotation, so strict
type checkers (e.g. ty) infer it as `Literal[False]`. Reassigning it to
`True` (as done in proxy_server.py and router.py) then fails with an
invalid-assignment error. Annotate it as `bool` to match every other
flag in this module.

* fix: coalesce null aggregates in update_metrics for no-spend keys (BerriAI#29945)

* feat(team_endpoints): add query parameter `key_limit` to `/team/info` endpoint (BerriAI#30006)

* feat(team_endpoints): Add query parameter key_limit to /team/info

* feat(team_endpoints): update schema.d.ts to include the new query parameter

* feat(team_endpoints): add tests for limitting key count in /team/info response

* feat(team_endpoints): Apply suggestions from greptile

* Set greater-than constraint on key-limit
* Fix type

* fix(router): release aiohttp connection when stream iteration ends abnormally (BerriAI#30271)

* fix(router): release aiohttp connection when stream iteration ends abnormally

A streaming response that terminates with a mid-stream read timeout, a task
cancellation (client disconnect), or GeneratorExit never closed the underlying
aiohttp ClientResponse. aiohttp only auto-releases the connector slot at body
EOF, so each abnormally terminated stream permanently leaked one slot from the
shared TCPConnector pool. During a backend traffic spike the pool drains; once
exhausted every subsequent request to that host waits for a slot, times out
and surfaces as a 408, indefinitely, even after the backend recovers. Only a
proxy restart cleared the in-memory sessions, which matched the reported
symptom of a router stuck returning 408 for a healthy vLLM backend.

Close the response in a finally clause when iteration ends. On a fully read
response the connection was already released at EOF and close() is a no-op,
so keep-alive reuse for normal requests is unchanged.

Fixes BerriAI#30192

* test(aiohttp): cover GeneratorExit path with a mock instead of a live socket

The previous slot-release test started a real aiohttp TCP server, which can
flake in offline CI and does not exercise this fix's code path directly.
Replace it with a dependency-injected mock that closes the stream generator
(GeneratorExit) and asserts the response is closed, covering the third
abnormal-exit path the finally block handles

* feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery (BerriAI#30273)

* feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery

* refactor(proxy): move Anthropic model-list formatter into llms/anthropic/common_utils

* fix(proxy): make model_list request param optional for direct callers

* feat(dashscope): add Responses API support (BerriAI#30286)

* feat(dashscope): add Responses API support

DashScope's OpenAI-compatible endpoint serves /responses, so register a
DashScopeResponsesAPIConfig that routes dashscope/* responses calls to
{api_base}/responses without rewriting the upstream model id, instead of
falling back to the chat-completions -> responses emulation pipeline.

Closes BerriAI#29780

* feat(dashscope): mark responses API as not supporting native websocket

Matches the hosted_vllm/perplexity/openrouter responses configs, which all
override supports_native_websocket() to False since the OpenAI-compatible
endpoint has no native wss:// responses transport.

---------

Co-authored-by: Sameer Kankute <[email protected]>

* fix(spend-logs): preserve error_message on ProxyException failures (BerriAI#30381)

* fix(spend-logs): preserve error_message on ProxyException failures

`StandardLoggingPayloadSetup.get_error_information` used
`str(original_exception)` to populate the human-readable error message
stored in `spend_logs.metadata.error_information.error_message`.

`ProxyException` (litellm/proxy/_types.py:3453) sets `self.message` in
its constructor but does NOT call `super().__init__(message)` and does
NOT define `__str__`. As a result, `str(ProxyException(...))` returns
the empty string, and every auth/budget/quota rejection was landing
in spend_logs with `error_message=""` despite a fully populated
traceback.

Operator impact: dashboard "LLM Failure" rows became untriageable —
the only way to tell a 401 from a 429 was to manually unpack the
traceback JSON via psql. Burst failure patterns (e.g. a UI session
polling with a stale token) produced 20-30 indistinguishable
`error_code=401` rows per second.

Fix: prefer the `.message` attribute (set by ProxyException and every
litellm.exceptions.* class) over `str(exc)`. The `str(exc)` fallback
is retained for non-litellm exception types, preserving prior behavior.

Test plan:
  - 2 new unit tests in tests/test_litellm/litellm_core_utils/
    test_litellm_logging.py:
    * test_get_error_information_prefers_message_attribute_over_str
    * test_get_error_information_falls_back_to_str_when_no_message_attr
  - Existing test_get_error_information_error_code_priority still passes
  - End-to-end verified: bad-key 401 now stores full
    "Authentication Error, Invalid proxy server token passed..."
    message in spend_logs.metadata.error_information.error_message

* fix(spend-logs): preserve explicit empty .message + drop dead reference

Greptile P2 on BerriAI#30381. The truthiness check `if message_attr:`
silently skipped an explicit empty-string `.message` and fell
through to `str(original_exception)`. For ProxyException-shaped
objects both produce empty, so the bug was latent; for other
exception types it would inject a different string into
error_information.error_message and corrupt the signal.

Use `is not None` so an empty string survives verbatim.

Also drop the stale `See e2e/cases/11.` comment reference — that
path does not exist anywhere in the repo and confuses future
readers.

Regression test added: an exception with `.message=""` and a
non-empty `super().__init__()` arg must yield error_message == "".

* ci: retrigger workflows after base branch change to litellm_internal_staging

* fix(anthropic): strip LiteLLM-injected total_tokens from /v1/messages response (BerriAI#30382)

* fix(anthropic): strip LiteLLM-injected total_tokens from /v1/messages response

The non-streaming /v1/messages response carries a LiteLLM-injected
usage.total_tokens = input_tokens + output_tokens that is not part of
the Anthropic API spec. This caused three problems:

1. Shape divergence with streaming on the same endpoint.
   message_delta.usage in the SSE path never carries total_tokens.
   Clients parsing both paths get two different schemas from one endpoint.

2. Shape divergence with upstream. Direct calls to
   https://api.anthropic.com/v1/messages return no total_tokens field,
   so clients using the official Anthropic SDK couldn't rely on it,
   and clients that did rely on the LiteLLM-injected one broke when
   bypassing the proxy.

3. Numerical misuse. total = input + output undercounts when
   cache_read_input_tokens and cache_creation_input_tokens are
   non-zero, because cache tokens are reported in their own fields.
   A 100k-token cached prompt with 1 non-cache input token + 200
   output tokens reports total_tokens = 201, off by ~99.8% from any
   reasonable definition of "total."

Fix: add _strip_total_tokens_from_anthropic_response in
litellm/proxy/anthropic_endpoints/endpoints.py and invoke it in the
success path of anthropic_response right before returning. Only mutates
dict-shaped responses; streaming (which already lacks the field) is
left untouched.

spend_logs / Prometheus continue to compute total_tokens internally
for billing — this fix only strips the field from the wire response.

Scope: only the Anthropic passthrough endpoint /v1/messages. The
OpenAI-shape /v1/chat/completions is unaffected.

* fix(anthropic): gate total_tokens strip behind flag + handle Pydantic .usage

Two P1 greptile threads on BerriAI#30382:

P1 — **Backwards-incompatible removal without a feature flag**
  Stripping `usage.total_tokens` unconditionally breaks any client
  currently reading the LiteLLM-shaped non-streaming /v1/messages
  response. Per the codebase's policy (mirrors BerriAI#30418), gate behind
  a new flag.

  - `litellm.strip_anthropic_total_tokens: bool = False` (default —
    backward-compat: clients keep seeing total_tokens).
  - Env override: `LITELLM_STRIP_ANTHROPIC_TOTAL_TOKENS=true`.
  - Docstring: planned to flip to True in a future major release;
    opt in early.

P1 — **Silent no-op if `result` is a Pydantic model**
  `base_process_llm_request` may return a Pydantic-style object
  whose `.usage` is a plain dict (the most common shape — e.g.
  objects wrapping raw upstream JSON). The original
  `isinstance(response, dict)` guard skipped strip on those, so
  `total_tokens` would still hit the wire. Helper now also reads
  `getattr(response, "usage", None)` and strips when that's a dict.

  Strongly-typed Pydantic `Usage` sub-models with required
  `total_tokens` fields are still skipped — those impose type
  constraints the helper doesn't try to subvert.

Tests:
- `test_strips_total_tokens_on_pydantic_model_with_dict_usage`
- `test_flag_defaults_off`
8/8 pass locally.

* fix(anthropic): drop env var for strip flag (docs CI)

Mirrors BerriAI#30418's pattern (`expose_router_debug_in_errors: bool = True`,
no `os.getenv`). The `LITELLM_STRIP_ANTHROPIC_TOTAL_TOKENS` env var
introduced in the prior commit was flagged by
`tests/documentation_tests/test_env_keys.py` because the documentation
file `docs/my-website/docs/proxy/config_settings.md` lives in
`BerriAI/litellm-docs` (separate repo) and registering a new env key
requires a parallel docs PR — a friction we avoid here by exposing
the flag only as a Python attribute + `litellm_settings` config key,
both of which load through the existing proxy config plumbing without
needing the env-var registry to be updated.

No semantic change: default still False, behavior identical when set
via `litellm.strip_anthropic_total_tokens = True` or
`litellm_settings.strip_anthropic_total_tokens: true` in config.yaml.

Verified locally: env scan no longer surfaces the key; 8/8 tests pass.

* ci: retrigger workflows after base branch change to litellm_internal_staging

* fix(pricing): correct swapped input/output token costs for command-r7b-12-2024 (BerriAI#30413)

* fix(pricing): correct swapped input/output token costs for command-r7b-12-2024

* test: resolve model prices JSON relative to test file for pip installs

* fix(exception-mapping): map Gemini upstream-error body code 429 to RateLimitError (BerriAI#30417)

* fix(exception-mapping): map Gemini upstream-error body code 429 to RateLimitError

Some Gemini-compatible gateways (e.g. new-api) wrap a 429 rate-limit
signal from upstream inside an HTTP 500/503 envelope, with the real
code only surfaced in the JSON body:

    {"error":{"message":"...high demand...","type":"upstream_error",
              "param":"","code":429}}

Previously LiteLLM only looked at the HTTP status and mapped this to
InternalServerError, which Router treats as non-retryable for many
configs — so users got hard 500s instead of fallback/retry.

Now the Gemini/Vertex exception mapper parses error.code from the body
and routes code 429 to RateLimitError before falling through to the
HTTP-status branches. Other body codes fall through unchanged.

Tests cover:
- new-api gateway's `code:429` payload now maps to RateLimitError
- Genuine 500-body responses stay InternalServerError
- Non-JSON body strings fall through to status-code mapping unchanged

* fix(exception-mapping): scope body-code 429 promotion to 5xx envelopes

Addresses greptile P1/P2 + @Sameerlite's review on BerriAI#30417. The new
elif branch was firing for any HTTP status, so a gateway response of
HTTP 400 with body {"error":{"code":429,...}} would be incorrectly
promoted to RateLimitError (retryable) instead of falling through
to BadRequestError. Same trap for 401 -> AuthenticationError.

Scoped the body-code 429 check to `500 <= status_code < 600` —
covers 500/502/503/504 (gateways wrapping upstream 429 in any 5xx
envelope) without inviting the 4xx misclassification.

Tests: parametrized table now covers 5xx (500/502/503), 4xx (400/401),
and the existing fall-through cases, asserting each maps to the
exception type that matches the HTTP status code. 50/50 pass locally.

* ci: retrigger workflows after base branch change to litellm_internal_staging

* feat(router): add expose_router_debug_in_errors flag (default True) to redact internal model_group/fallback names (BerriAI#30418)

* feat(router)!: redact internal model_group/fallback names from exception messages

The Router was unconditionally appending internal config names onto
exception.message:
  - "Received Model Group=..."
  - "Available Model Group Fallbacks=..."
  - "No fallback model group found... Fallbacks={...}"
  - "context_window_fallbacks={...}"
  - Deployment-timeout messages including model_group
  - Fallback failure detail listing fallback chain

ProxyException forwards .message verbatim to clients, so gateways were
leaking their model_name / fallback wiring in every failed call.

Fix: gate all five mutation sites on a new
`litellm.expose_router_debug_in_errors` flag (default False). Set to
True to restore upstream debug behavior for local debugging.

Why: matches the redaction posture this codebase already has for
upstream model identifiers (cf. _litellm_returned_model_name) and
removes the last common error-path leak of internal model_group names.

Breaking change marker (!): if anything parses "Received Model Group="
out of client error messages, flip the flag on or migrate to the
x-litellm-* response headers instead.

Tests: 7 cases covering each of the 5 redaction sites + the flag-on
inverse path, plus a "default off" sanity check.

* test(router): cover sites 1 + 3 of expose_router_debug_in_errors gate

Addresses Greptile / codecov feedback on BerriAI#30418: patch coverage was
55.6% with 4 lines uncovered in litellm/router.py. The existing tests
exercised sites 2 (ContextWindowExceededError), 4 (no-fallback-found),
and 5 (Received Model Group) — both default and flag-on. Sites 1 and 3
were declared in the PR description as covered by "site 5 also fires"
but the gate body lines for each (the `e.message +=` inside the
`if litellm.expose_router_debug_in_errors:` branch) only execute when
the flag is on AND the specific exception path is taken, which neither
existing test triggered.

Added 4 new tests (default + flag-on × 2 sites):

  - test_default_does_not_leak_deployment_timeout_debug
  - test_flag_on_leaks_deployment_timeout_debug
  - test_default_does_not_leak_content_policy_fallback_hint
  - test_flag_on_leaks_content_policy_fallback_hint

Trigger details:

  - Site 1 (litellm.Timeout in _acompletion) is reached via the
    Router-supported `mock_timeout=True` + `timeout=0.001` kwargs on
    `acompletion(...)`. Cannot embed a Timeout instance in model_list
    because Router.__init__ deep-copies it and Timeout.__reduce__ does
    not preserve the required positional args.
  - Site 3 (ContentPolicyViolationError without content_policy_fallbacks
    set, in async_function_with_fallbacks_common_utils) is reached by
    passing a `mock_response=litellm.ContentPolicyViolationError(...)`
    instance via the call-site kwarg — same deepcopy-avoidance reason.

11/11 tests pass locally. Patch coverage on litellm/router.py for this
PR's diff should now be 100%.

* chore(router): flip expose_router_debug_in_errors default to True

Addresses @Sameerlite's review on BerriAI#30418 — maintain backward
compat on the wire. Redact becomes opt-in via setting the flag
to False; the historical behavior (leak internal model_group /
fallback wiring through exception messages) is preserved as the
default.

- litellm/__init__.py: default flipped to True, docstring rewritten
  with deprecation note pointing at a future flip to False (redact
  by default) in a major release.
- tests/test_litellm/test_router_exception_redaction.py: fixture
  resets to True (was False); the "off" tests now explicitly set
  False; the "default_leaks_*" tests rely on the fixture default.
  test_flag_defaults_off -> test_flag_defaults_on.
- No router.py change needed; the gate keys off the same flag,
  only the default changes.
- PR title no longer needs the breaking-change `!` marker — no
  client sees a behavior change at default settings.

11/11 pass locally.

* ci: retrigger workflows after base branch change to litellm_internal_staging

* feat(guardrails): integrate Repelloai Argus guardrail (BerriAI#30465)

* feat(guardrails): add RepelloAI Argus guardrail integration (#1)

* feat(guardrails): add RepelloAI Argus guardrail integration

Add a new guardrail hook backed by RepelloAI Argus, with dashboard-managed
asset policies enforced via an asset_id and X-API-Key auth.

* fix(guardrails): harden RepelloAI Argus guardrail

- scan streaming responses on output (was bypassing the guardrail)
- log blocked verdicts as guardrail_intervened instead of success
- treat auth/config errors (401/403/404/422) as misconfiguration that
  always blocks, not a fail-open-able unreachable error
- default unreachable_fallback to fail_closed and read it directly;
  block on unknown/malformed verdicts so an API change can't silently
  disable enforcement
- type unreachable_fallback as a Literal, drop the duplicate config model,
  expose unreachable_fallback in the config schema, and stop leaking the
  raw provider response / exception strings to the client

* fix(guardrails): address RepelloAI Argus review feedback

- support ARGUS_API_KEY (with REPELLOAI_API_KEY fallback)
- make asset_id required in the config model
- normalize unreachable_fallback so only fail_open opens; block on 400 misconfig
- correct the shared unreachable_fallback field description

* docs(guardrails): add RepelloAI Argus docs page and dashboard listing

- add docs page covering config, env vars, modes, verdicts, failure semantics
- list RepelloAI Argus in the Guardrail Garden with provider/logo mappings
- add a regression test for the provider logo and display-name resolution

* fix(guardrails): keep RepelloAI asset_id optional in config model

A required asset_id leaked onto the shared LitellmParams (which inherits
RepelloAIGuardrailConfigModel), breaking validation for every other
guardrail. Keep it optional like sibling models; the guardrail __init__
still raises when asset_id is missing, which is the real enforcement.

* Add comment for last user turn scanning

* feat(guardrails): harden repelloai scanning

* feat(guardrails): expand repelloai scanning to include tool definitions

Add extraction of tool definitions and tool call arguments to the RepelloAI
guardrail scanning. Improves detection coverage by including function schemas
and parameters in the prompt sent to the guardrail service. Also captures
detailed error responses in logs and adds guardrail header to streaming responses.

* refactor(guardrails): fix and harden repelloai schema text extraction

- Fix duplicate text in _iter_schema_text: previously all dict values were
  re-queued onto the stack even after scalar/list keys were already extracted
  explicitly, causing names/descriptions to appear twice in the scanned prompt
- Extract schema key frozensets to module-level constants so they are not
  reconstructed on every call
- Change _iter_schema_text from @classmethod to @staticmethod (cls unused)
- Narrow _call_analyze stage param from str to Literal["prompt", "response"]
- Add HttpxResponse type annotation to _raise_for_config_error
- Add LLMResponseTypes annotation to async_post_call_success_hook response param

* fix(guardrails): resolve pyright type errors in repelloai guardrail

- Narrow async_handler.post return from Response|None to Response with
  explicit None guard before calling raise_for_status/json
- Fix list comprehension returning str|None by switching to explicit loop
  with isinstance guard so pyright tracks the narrowing
- Cast model_dump() result to Dict since hasattr does not narrow object
  type in pyright

* fix(guardrails/repello): include Responses API instructions field in prompt scan

The /v1/responses top-level `instructions` field was not included in
_extract_prompt_text, allowing a caller to bypass guardrail policy checks
by putting blocked content in `instructions` while keeping `input` benign.

* feat: add api_key to config model and read prompt from data dict

* fix(guardrails/repello): plug input_text and tool-call response bypass gaps

Responses API input content parts with type 'input_text' were silently
dropped by build_inspection_messages (which only handles type='text'),
allowing callers to send blocked content via that path without triggering
the pre-call scan. Fix: add _extract_input_text_parts to RepelloAIGuardrail
and call it when walking the Responses API input messages.

Post-call scanning skipped responses whose choices contained only tool_calls
or function_call (message.content=None), letting models put blocked output in
function arguments undetected. Fix: _extract_chat_completion_text now calls
_extract_tool_call_args_from_message on each choice message.

Also replace typing.Dict/List with builtin dict/list to clear TID251 strict
ruff violations introduced by this file.

* fix(guardrails/repello): scan Responses API function_call output arguments

Output items with type 'function_call' in a /v1/responses response were
skipped by _extract_responses_api_text; only 'message' items were walked.
A model could return blocked content in function_call.arguments undetected.
Now extract arguments from function_call output items before scanning.

* fix(anthropic): drop orphaned server_tool_use on multi-turn replay from generic OpenAI clients (BerriAI#30486)

* fix(anthropic): drop orphaned server_tool_use on multi-turn replay from generic OpenAI clients

When an Anthropic server-side tool (web_search, id `srvtoolu_...`) is used, its
result is carried in `provider_specific_fields.web_search_results` — PRs BerriAI#17746
/ BerriAI#17798 restore it for callers that round-trip provider_specific_fields. A
generic OpenAI client that does NOT preserve provider_specific_fields (e.g. Open
WebUI talking to a Vertex/Anthropic model over /chat/completions) drops it on
replay and instead sends back an assistant `tool_call` + a `tool` message both
keyed to the `srvtoolu_` id. The transform then produced a bare `server_tool_use`
(with no following *_tool_result) plus a user `tool_result` for the same id —
both invalid, so the next turn 400s:

  messages.N.content.0: unexpected `tool_use_id` found in `tool_result` blocks:
  srvtoolu_... Each `tool_result` block must have a corresponding `tool_use`
  block in the previous message.

This is the commonly-reported vertex_ai symptom where Gemini works but Claude
400s on the 2nd turn of a web-search chat.

Fix (litellm/litellm_core_utils/prompt_templates/factory.py):
- convert_to_anthropic_tool_invoke: only emit a server_tool_use when its matching
  *_tool_result is available to pair with it; otherwise skip it (a bare
  server_tool_use is itself rejected).
- anthropic_messages_pt: drop a replayed `tool`/`function` message whose
  tool_call_id starts with `srvtoolu_` (a server-executed tool produces no client
  result; a user tool_result for it is invalid).

The existing reconstruction path (provider_specific_fields present, e.g. the
litellm SDK) is unchanged, as is regular client tool_use/tool_result.

Tests (tests/llm_translation/test_prompt_factory.py):
- update test_convert_to_anthropic_tool_invoke_server_tool ->
  test_convert_to_anthropic_tool_invoke_server_tool_without_result_is_dropped
- add test_anthropic_messages_pt_generic_client_drops_orphan_server_tool

Follow-up to BerriAI#17746 / BerriAI#17798; addresses the generic-client (no
provider_specific_fields) case of BerriAI#17737.


* test(anthropic): cover the srvtoolu_ round-trip fix in the test_litellm unit suite

The regression tests added in tests/llm_translation/test_prompt_factory.py aren't
run by the coverage CI job (it runs tests/test_litellm), so the new factory.py
branches showed as uncovered (codecov patch coverage). Add equivalent focused
tests in the unit suite so both new branches are exercised there:
- convert_to_anthropic_tool_invoke drops a srvtoolu_ server_tool_use when no
  matching *_tool_result is available.
- anthropic_messages_pt drops the orphaned srvtoolu_ tool message a generic
  OpenAI client replays.

Refs BerriAI#17737


* test(anthropic): cover the server_tool_use + result valid-pair path in unit suite

Covers the remaining patch-coverage lines codecov flagged: convert_to_anthropic_tool_invoke
emitting server_tool_use followed by its web_search_tool_result when the matching
result is present (the litellm-SDK round-trip path). Refs BerriAI#17737


* style(anthropic): flatten srvtoolu_ tool-message guard to a negated if

Addresses the Greptile style nit: replace the if-pass/else with a single negated
`if not (...)` guard around the tool_result append. Behavior unchanged. Refs BerriAI#17737


---------


* fix(proxy): require premium only when enabling premium metadata fields (BerriAI#30285) (BerriAI#30506)

Co-authored-by: Sameer Kankute <[email protected]>

* fix(perplexity): stop double-billing reasoning tokens in manual cost fallback (BerriAI#30488)

* fix(perplexity): stop double-billing reasoning tokens in manual cost fallback

When perplexity_cost_per_token cannot use the API-provided usage.cost.total_cost short-circuit and falls back to manual calculation, it multiplies the full usage.completion_tokens by output_cost_per_token and then adds reasoning_tokens * output_cost_per_reasoning_token on top. Per the OpenAI/Perplexity usage convention codified for the central path in PR BerriAI#18607, completion_tokens already INCLUDES reasoning_tokens, so the manual fallback double-bills reasoning at both the output and reasoning rate.

Concrete impact on perplexity/sonar-deep-research (input 2e-6, output 8e-6, reasoning 3e-6): for the exact usage shape exercised by the live response fixture in tests/llm_translation/test_perplexity_reasoning.py (prompt_tokens=9, completion_tokens=20, reasoning_tokens=15) the current code charges 0.000223 vs the convention-correct 0.000103, a 2.165x overcharge. The bug is reachable whenever Perplexity omits the cost object (streaming chunks, fixture-driven paths, older API versions).

Subtracts reasoning_tokens (clamped at zero) from completion_tokens before applying the output rate, mirroring how dashscope/cost_calculator.py and the central generic_cost_per_token already handle it. Preserves the existing fallback behaviour when output_cost_per_reasoning_token is unset (all completion_tokens stay at the output rate).

Existing tests in tests/test_litellm/llms/perplexity/test_perplexity_cost_calculator.py asserted the buggy math and are updated to the convention-correct math. Adds a focused regression test using the exact usage shape from the live response fixture so this class of bug cannot be silently reintroduced.

* style(perplexity): drop redundant type annotation on else branch to satisfy mypy

mypy [no-redef] flagged 'completion_cost' as declared in both if and else arms; keeping the annotation only on the first declaration matches existing patterns in this file.

* fix(perplexity): update integration test expected costs for non-double-billed math

Three tests in test_perplexity_integration.py asserted the old buggy expectation
that reasoning_tokens are billed in addition to the full completion_tokens
count. After the fix in cost_per_token, reasoning_tokens are billed at the
reasoning rate and the remaining (completion_tokens - reasoning_tokens) at the
standard output rate, matching OpenAI/Perplexity convention (PR BerriAI#18607).

Updates: test_end_to_end_cost_calculation_with_transformation,
test_main_cost_calculator_integration, test_high_volume_cost_calculation.
The high-volume sanity threshold drops to 0.25 to reflect the corrected total.

* fix(ui): use dynamic proxy base URL in MCP usage examples (BerriAI#30487)

Replace hardcoded http://localhost:4000 with getProxyBaseUrl() in the
MCP server usage example and copy-to-clipboard snippet so the generated
configuration works for non-local deployments.

Fixes BerriAI#30466

* feat: add missing UK PII entity types to Presidio guardrail (BerriAI#30537)

* feat: add missing UK PII entity types to Presidio guardrail

Add UK_PASSPORT, UK_POSTCODE, and UK_VEHICLE_REGISTRATION to PiiEntityType enum and PII_ENTITY_CATEGORIES_MAP. These entity types are supported by Microsoft Presidio but were missing from litellm's type definitions, preventing users from configuring UK-specific PII detection.

* test: remove fragile hardcoded entity count test

Remove test_uk_category_entity_count which hardcodes len() == 5. The test_uk_entities_match_presidio_recognizers test already verifies exact set equality, making the count test redundant and fragile to future Presidio additions.

* style: apply Black formatting to match CI requirements

* fix: route volcengine (Doubao) tiered-pricing models to the tiered cost handler (BerriAI#30357)

Volcengine (Doubao) models define `tiered_pricing` but no flat per-token cost, so cost_per_token fell through to generic_cost_per_token (which only reads flat costs) and tracked them at $0

Route custom_llm_provider == "volcengine" to the shared tiered-pricing handler in litellm/llms/dashscope/cost_calculator.py, which already computes graduated tier costs. Make that handler provider-agnostic by adding a custom_llm_provider argument (default "dashscope" preserves existing behavior) so get_model_info resolves the correct model map entry

Fixes BerriAI#30346

* feat(mcp): make MCP gateway name and description configurable via env vars (BerriAI#30473)

* feat(mcp): make MCP gateway name and description configurable via env vars

* Rename function _restore_env to _apply_env

* docs(mcp): document import-time capture of env-backed identity constants

Address Greptile review feedback: clarify that LITELLM_MCP_SERVER_NAME and
LITELLM_MCP_SERVER_DESCRIPTION are read once at import and require a module
reload to observe env changes after import.



---------

Co-authored-by: Yevhen Luhovtsov <[email protected]>

* fix(mcp): preserve native tools in semantic filter hook (BerriAI#26650)

* fix(mcp): preserve native tools in semantic filter hook

The SemanticToolFilterHook.async_pre_call_hook passed ALL tools (MCP +
native) to filter_tools(), which only knows MCP-registered tool names.
Native tools silently failed the name match in _get_tools_by_names()
and were dropped from the request.

Fix: partition tools into native and MCP-registered before filtering.
Run the semantic filter only on MCP tools, then merge native tools
back unconditionally.

Changes:
- Robust _is_mcp_tool() using shape-based detection for OpenAI-format
  dicts, safe regardless of future _extract_tool_info changes
- Single-pass partition loop (no double _is_mcp_tool calls)
- Preserve native tools in MCP expansion path (mixed requests)
- Track MCP expansion to prevent expanded tools bypassing filtering
- filter_stats reports MCP-only counts for accurate metrics
- Extracted _emit_filter_metadata() helper
- Skip spurious filter headers for all-native tool requests

Closes BerriAI#26212

* remove stale docstring note referencing tools_expanded_from_mcp

* fix: handle Responses API name collision and preserve tool ordering

- Classify Responses API tools ({type: 'function', name: '...'}) as
  native to prevent name collisions with MCP canonical names
- Preserve original request tool ordering using id()-based merge
  instead of naive native+mcp concatenation
- Add 2 regression tests: name collision and ordering preservation

* style: apply black formatting

* fix(mcp): harden semantic filter — preserve all native tool formats, safe metadata access, graceful expansion failure, name-based merge

* lint: suppress PLR0915 on async_pre_call_hook (matches codebase convention)

* ci: retrigger checks after rebase onto litellm_internal_staging

* feat(fireworks): sync Fireworks AI model registry with current platform catalog (BerriAI#30616)

Adds 12 new Fireworks serverless models and updates 3 existing entries in
model_prices_and_context_window.json and its bundled backup to match the
current Fireworks platform model list. New direct models: glm-5p2,
qwen3p7-plus, minimax-m3, minimax-m2p7, kimi-k2p7-code, kimi-k2p6,
deepseek-v4-pro, deepseek-v4-flash. New router endpoints: glm-5p1-fast,
kimi-k2p6-fast, kimi-k2p7-code-fast. Updated: glm-5p1, gpt-oss-120b, and
gpt-oss-20b now carry correct output token caps, cache-read pricing, and
explicit capability flags

max_tokens is set equal to max_output_tokens (not the full context window)
for models whose generation cap is below their context window. This avoids
the shared input+output budget path in get_modified_max_tokens, which would
otherwise let callers request output sizes the model cannot produce. The
same fix corrects the pre-existing glm-5p1, gpt-oss-120b, and gpt-oss-20b
entries that had max_tokens equal to the full context window

Short-form aliases (fireworks_ai/<model>) are added for every direct
accounts/fireworks/models/ entry so cost attribution works for callers
using bare model names. Router endpoints get short-form aliases too, and
transform_request now routes bare names ending in -fast to the
accounts/fireworks/routers/ path instead of defaulting every bare name to
models/. This keeps the kimi-k2p6-fast router from being misrouted to the
nonexistent models/kimi-k2p6-fast endpoint

kimi-k2p6-turbo is intentionally excluded; kimi-k2p6-fast is its
replacement. Context windows for deepseek-v4 and kimi models use the
power-of-two values (1048576 and 262144) published on the Fireworks model
pages, matching the convention already used by existing entries

Two regression tests in test_utils.py assert the exact per-token costs,
token limits, capability flags, and short-form-to-long-form equality for
all 15 models against both the main and backup cost maps. Two routing
tests in test_fireworks_ai_chat_transformation.py verify bare -fast names
route to routers/ and bare direct-model names route to models/

* fix(bedrock): handle role:"system" inside the messages array on /v1/messages (BerriAI#29698) (BerriAI#30443)

* feat(anthropic): hoist leading in-array system to top-level (helper)

* test(anthropic): cover _system_content_to_blocks edge cases; deepcopy cache_control

* test(anthropic): mid-conversation system normalization cases

* feat: add supports_mid_conversation_system flag to Claude Opus 4.8

Add supports_mid_conversation_system: true to all 9 claude-opus-4-8 cost-map
entries (Anthropic-native, Bedrock, Vertex, Azure AI) in both the root cost
map and the bundled package backup, since the runtime helper and tests read
the backup in local/offline mode.

Pin the mid-system passthrough regression test to the local cost map via the
existing local_model_cost_map fixture so it reads the branch-local flag rather
than the network-fetched main copy.

* fix(bedrock): normalize in-array system in /v1/messages handler (BerriAI#29698)

Wire normalize_system_messages_for_anthropic into anthropic_messages_handler
so all Bedrock /v1/messages paths (Invoke / Mantle / ClaudePlatform /
Converse-bridge) hoist leading in-array system entries (and demote
mid-conversation ones on models lacking supports_mid_conversation_system) into
the top-level system field. The normalized messages/system are written back
into the local_vars snapshot the base_llm branch reads from, otherwise the
Invoke/Mantle fix would silently no-op.

Also fix the helper to resolve supports_mid_conversation_system through the
prefix-aware AnthropicModelInfo._supports_model_capability resolver. The raw
_supports_factory could not see the flag once get_llm_provider left the
invoke/ prefix on the model id, which would have wrongly demoted
mid-conversation system on a Bedrock invoke opus-4-8 path.

* fix(bedrock): resolve mid-conversation-system flag through mantle/invoke/converse route prefixes; drop unused param

* fix(types): widen system param to Union[str, List] for hoisted system blocks

* refactor(bedrock): drop dead local_vars messages writeback

* fix(bedrock/converse): translate in-array system in anthropic->openai adapter (BerriAI#29698)

* fix(bedrock/converse): preserve cache_control on in-array system; test drop-empty

* fix(bedrock/converse): rename colliding local to satisfy mypy; test handler system-merge branches

* fix(types): register supports_mid_conversation_system in model-info schema

The cost-map JSON-schema validation test (test_aaamodel_prices_and_context_window_json_is_valid)
rejects unknown properties, so adding supports_mid_conversation_system to the opus-4-8
cost-map entries failed CI with 'Additional properties are not allowed'. Register the flag
in the INTENDED_SCHEMA allow-list and in the ProviderSpecificModelInfo TypedDict so it is a
typed, first-class capability flag alongside its peers (supports_output_config, etc.).

---------

Co-authored-by: Sameer Kankute <[email protected]>

* fix(bedrock/agentcore): optionally forward multimodal content blocks in InvokeAgentRuntime payload (BerriAI#28885)

* fix(bedrock/agentcore): optionally forward multimodal content blocks in InvokeAgentRuntime payload

By default the agentcore provider flattens the last message to a text-only
{"prompt": "..."} payload via convert_content_list_to_str, silently dropping
OpenAI multimodal blocks (image_url, file, input_audio, ...).

This adds an opt-in `forward_multimodal_content` litellm param. When truthy and
the last message's content is a list containing a non-text block, the original
OpenAI content list is forwarded verbatim under a new "content" field so an
attachment-aware AgentCore agent can read it. Default off keeps the payload
byte-identical to the legacy {"prompt": "..."} shape — existing agents are
unaffected.

The flag is read from optional_params (where other AgentCore params land) with a
litellm_params fallback, and accepts a bool or a config/env string ('true', '1', ...).

AgentCore Runtime is schemaless on the agent side — the agent's @app.entrypoint
parses arbitrary JSON up to 100 MB (per
https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-invoke-agent.html),
so this is a purely upstream change; no AgentCore-side schema is asserted.

* fix(bedrock/agentcore): shallow-copy forwarded multimodal content list

Address review feedback (Sameerlite): payload["content"] = last_content
aliased the caller's mutable messages[-1]["content"] list. Harmless today
because the payload is JSON-serialized immediately, but a latent footgun if
a future caller mutates the returned payload before serialization. Forward
list(last_content) so the payload owns its own list. Block dicts stay shared
on purpose — a deep copy would clone potentially large base64 media on the
request hot path, and the flagged risk was the shared list, not the blocks.

Update the passthrough tests to assert equality + distinct identity, and add
a regression test that mutating the payload list can't leak back into the
original message content.

* Revert "fix(mcp): preserve native tools in semantic filter hook (BerriAI#26650)"

This reverts commit 438c825.

* Revert "feat(guardrails): integrate Repelloai Argus guardrail (BerriAI#30465)"

This reverts commit 54da785.

* Revert "feat(dashscope): add Responses API support (BerriAI#30286)"

This reverts commit 6766256.

* Revert "fix(bedrock): handle role:"system" inside the messages array on /v1/messages (BerriAI#29698) (BerriAI#30443)"

This reverts commit b8a8083.

* Revert "fix(anthropic): drop orphaned server_tool_use on multi-turn replay from generic OpenAI clients (BerriAI#30486)"

This reverts commit 6e9c0b0.

* Revert "fix: route volcengine (Doubao) tiered-pricing models to the tiered cost handler (BerriAI#30357)"

This reverts commit 172e302.

* Revert "feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery (BerriAI#30273)"

This reverts commit 4e31885.

* fix: pass key_limit=None in team_member_update and patch model_cost in pricing test

team_member_update called team_info without key_limit, so the fastapi.Query
default object (not None) was passed through to get_data, which failed when
serializing it. Pass key_limit=None explicitly to avoid this.

test_get_model_info_costs patched litellm.model_cost from the local backup so
the assertion holds before the PR is merged and the remote main URL is updated.

* fix(security): validate resolved model in /realtime/client_secrets for non-transcription sessions (BerriAI#30710)

Omitting both model and session.model caused the endpoint to default to
gpt-4o-realtime-preview without running can_key_call_resolved_model, so
any key could access that model regardless of its allowed-model list.

The transcription path already called can_key_call_resolved_model; this
adds the same call for the realtime path before returning.

* fix(lint): fix F821 undefined model_info and F841 unused metadata in create_model_info_response

* fix: black formatting and stub get_model_group_info in third team translation test

* fix: reformat utils.py with black 26.3.1 to match CI

* fix: replace Optional[X] with X | None to satisfy UP045 ruff strict gate

---------

Co-authored-by: Habon Laszlo <[email protected]>
Co-authored-by: habonlaci <[email protected]>
Co-authored-by: Armaan Sandhu <[email protected]>
Co-authored-by: santino18727-debug <[email protected]>
Co-authored-by: Eric (GabiDevFamily) <[email protected]>
Co-authored-by: Nitish Agarwal <[email protected]>
Co-authored-by: jho1-godaddy <[email protected]>
Co-authored-by: 安妮的心动录 <[email protected]>
Co-authored-by: Harshith Gujjeti <[email protected]>
Co-authored-by: Tomoya Tabuchi <[email protected]>
Co-authored-by: Vedant Agarwal <[email protected]>
Co-authored-by: Prathamesh Jadhav <[email protected]>
Co-authored-by: songkuan-zheng <[email protected]>
Co-authored-by: Kropiunig <[email protected]>
Co-authored-by: Lavish Bansal <[email protected]>
Co-authored-by: Shane Emmons <[email protected]>
Co-authored-by: Anuj ojha <[email protected]>
Co-authored-by: Nahrin <[email protected]>
Co-authored-by: Nbouyaa <[email protected]>
Co-authored-by: Vineeth Sai <[email protected]>
Co-authored-by: Eugene Lugovtsov <[email protected]>
Co-authored-by: Yevhen Luhovtsov <[email protected]>
Co-authored-by: Ayush Shekhar <[email protected]>
Co-authored-by: Ahmad Shahzad <[email protected]>
Co-authored-by: Kent <[email protected]>
Co-authored-by: Jón Levy <[email protected]>

* fix(proxy): bump health-check max_tokens default to 16 for GPT-5 compatibility (BerriAI#30708)

OpenAI GPT-5 models require max_completion_tokens >= 16.
Health checks were using 5 (proxy/health_check.py) and 10
(health_check_helpers.py), causing failures on GPT-5 models.

Fixes BerriAI#23836

* fix: increase health check max_tokens from 5 to 16 (BerriAI#23836) (BerriAI#26610)

GPT-5 models enforce a minimum of 16 for max_output_tokens. The current
default of 5 still causes health checks to fail for these models. Bump
the non-wildcard default to 16 — the smallest value that satisfies all
known provider minimums while keeping health checks lightweight.

Also tightens the wildcard test assertion from a weak disjunctive check
to strict key-absence.

Co-authored-by: Sameer Kankute <[email protected]>

* fix: ensure checks show gemini-3-flash-preview supports responseJsonS… (BerriAI#30696)

* fix: ensure checks show gemini-3-flash-preview supports responseJsonSchema.

* fix: remove async keyword from test.

* fix: make Bedrock Mantle Responses routing data-driven per model (BerriAI#30700)

* Make Bedrock Mantle Responses routing data-driven per model

Route Bedrock Mantle models to the native Responses API based on each
model's price-map capability signal instead of a hardcoded model-name
heuristic, and derive the OpenAI-compatible base path segment per model.

Responses dispatch now selects the native config when the model advertises
responses support (/v1/responses in supported_endpoints, or mode=responses),
both overridable via register_model and proxy model_info. This enables
native Responses for gpt-oss-120b/20b and the gemma-4 family while keeping
chat-only models (gpt-oss safeguard, nvidia, mistral, ...) on the existing
chat-completions emulation. Capability is per-model, so gpt-oss-120b routes
natively while gpt-oss-safeguard-120b does not despite sharing the gpt-oss
substring.

The wire path is a separate concern, driven by the existing
use_openai_responses_path flag rather than a model-name match: gpt-5.x and
gemma-4-* on /openai/v1, everything else (incl. gpt-oss) on /v1. The chat
config now derives its base from the same flag, fixing gemma-4
chat-completions requests that previously went to /v1 instead of /openai/v1.

Cost maps: add supported_endpoints to the gpt-oss entries (responses for the
non-safeguard variants, chat-only for safeguard) and supported_endpoints +
use_openai_responses_path to all three gemma-4 entries.


* Address review: move capability helper into bedrock_mantle package

Move the Responses capability check out of utils.py into
litellm/llms/bedrock_mantle/common_utils.py as mantle_supports_responses,
alongside its companion wire-path helper mantle_base_segment. Both are now
pure functions of (model, model_cost): the price-map mode/supported_endpoints
read replaces the get_model_info call, so the rules are unit-testable without
patching global state and the Bedrock Mantle package is self-contained.

Use str | None instead of Optional[str] on the new signatures to satisfy the
ruff UP045 strict-rule gate. Add direct unit tests for both helpers.

Fix test_register_model_restore_undoes_existing_key_overwrite: gpt-oss-120b
now legitimately supports Responses, so it can no longer be the
"None after restore" vehicle; use the chat-only safeguard variant, which
isolates the register/restore effect from the model's own capability.


---------

Co-authored-by: Sameer Kankute <[email protected]>

* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (BerriAI#30366)

* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup

LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.

Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.

Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.

Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.

Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.

* fix: resolve CI failures and proxy DB URL typing issue

* fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (BerriAI#30653)

The tiered cost calculator resolved a tier's per-token cost with
`tier.get(cost_key) or tier.get(fallback_cost_key, 0)`. Because `or`
short-circuits on any falsy value, a tier that legitimately prices a
component at 0.0 (e.g. a free-cache-read tier with
cache_read_input_token_cost: 0.0, or a free-reasoning tier) is treated
as missing and silently billed at the full fallback rate
(input_cost_per_token / output_cost_per_token).

The flat-pricing path in the same module already handles this correctly
with an `is None` guard. Resolve tier costs through a small helper that
mirrors it, so 0.0 is honored at both the in-range and overflow sites.

No shipped model currently has a 0.0 tier cost, so this is a latent
defect; the fix makes the tiered path consistent with the flat path and
prevents over-charging the first time such a tier appears. Adds unit
tests covering the in-range and overflow paths, and drops an unused
import flagged by ruff in the touched test file.

* feat(proxy): show session-aggregate cost and duration in request logs (BerriAI#25708) (BerriAI#30507)

* fix(anthropic): don't leak tool 'type' into OpenAI function parameters schema (BerriAI#30618)

In the messages->chat/completions bridge, translate_anthropic_tools_to_openai
merged every non-mapped tool key into the function parameters dict. The
Anthropic tool 'type' (e.g. 'custom') thus overwrote parameters.type ('object'
-> 'custom'), and providers reject it ('custom' is not a valid JSON-Schema type).
Exclude 'type' from the passthrough. Fixes BerriAI#30557.

* fix(proxy): stop IAM-refresh engine restart from cascading reconnects (BerriAI#29176) (BerriAI#30183)

An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the
running query-engine and spawns a new one. That planned kill was
indistinguishable from a crash, and three reconnect paths used two
uncoordinated locks, so a single refresh triggered a cascade of engine
kill/respawn cycles:

  1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old
     engine, spawn new one.
  2. The engine-death watcher sees that kill, assumes a crash, and calls
     `attempt_db_reconnect(force=True)` (a different lock,
     `_db_reconnect_lock`) -> recreate again -> kills the fresh engine.
  3. In-flight queries failing during the swap are classified as transport
     errors and trigger their own `attempt_db_reconnect` -> recreate again.

Fix coordinates planned restarts across the wrapper and the watcher:

  - PrismaWrapper records the old engine PID in `_expected_engine_deaths`
    before killing it; all four watcher death-detectors (waitpid thread,
    pidfd, already-dead probe, os.kill poll) consume that PID and skip the
    reconnect instead of treating it as a crash.
  - `recreate_prisma_client` now serializes through `_reconnection_lock` and
    bumps a monotonic `_engine_generation`. Callers pass `expected_generation`
    as an optimistic-lock token, so racing/cascading recreates collapse into a
    single restart (losers no-op). This closes the two-lock gap.
  - The direct reconnect path probes the writer with SELECT 1 before
    recreating; a healthy connection (e.g. engine already replaced by a
    refresh) skips the recreate entirely.
  - `_safe_refresh_token` coalesces: it skips when the current token still has
    more than the refresh buffer of runway, so stacked triggers (proactive
    loop + __getattr__ fallback) don't each restart the engine. An
    `on_engine_replaced` hook re-arms the watcher on the new PID.

RoutingPrismaWrapper forwards `expected_generation` and skips recreating the
reader when the writer recreate was skipped.

* feat(bedrock): support file content retrieval for batch output files (BerriAI#30595)

Implements transform_file_content_request and transform_file_content_response
in BedrockFilesConfig so GET /v1/files/{id}/content works for Bedrock batch
files. The request transform resolves the file id (direct s3:// URI or base64
unified id) to its S3 object, validates bucket and key prefix against the
server-configured bucket, and SigV4-signs an S3 GetObject using the same
credential and region resolution as the existing upload path. The credential
and region params are validated into a typed model at the boundary, so the only
untyped values left are the botocore signing primitives.

Also fixes the proxy managed-files path: CredentialLiteLLMParams now carries
s3_bucket_name (previously dropped when building deployment credentials) and
the managed-files hook passes the deployment credential snapshot when routing
afile_content, so unified-id content retrieval works with per-model bucket
config instead of only the AWS_S3_BUCKET_NAME env var.

Preserves managed-file access control: the proxy file-content endpoint now
rejects raw cloud-storage ids (s3://, gs://), which would otherwise skip the
owner/team check that only runs for unified ids and let a caller read another
tenant's batch output by its object key. Managed outputs are reachable only
through their unified file id. The afile_content "not found" error now reports
the caller's unified id rather than the resolved internal S3 URI.

Fixes BerriAI#16186, BerriAI#15563

* fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (BerriAI#30646)

* fix(oci): map Cohere tool array/object params to lowercase builtins

OCI's Cohere backend returns HTTP 500 on a tool parameter typed as a bare
"List", which is what OCI_JSON_TO_PYTHON_TYPES produced for JSON-schema
arrays. MLflow {{trace}} judges trip this: their tools (get_root_span,
get_span) take an attributes_to_fetch array. The lowercase builtins list/dict
are accepted; only the bare "List" 500s ("Dict" happens to be tolerated, but
both are lowercased for consistency).

Verified live against us-chicago-1 (cohere.command-a-03-2025 and
command-latest). Adds a unit regression on the transformed parameterDefinitions
plus a gated integration test exercising an array-param tool end to end.

* fix(oci): make Cohere agentic tool-calling continuation work

Two bugs broke the OCI Cohere tool-calling loop that MLflow {{trace}} judges
drive once a tool has been executed and its result is fed back.

Request side: litellm pulled the last user message into the top-level `message`
and emitted the tool result as a TOOL entry in chatHistory. OCI rejects that
("cannot specify message if the last entry in chat history contains tool
results"), and an empty message alone is rejected too ("message must be at least
1 token long or tool results must be specified"). OCI carries the current turn's
results in a dedicated top-level `toolResults` field. The Cohere transform now
sends an empty message, keeps the user turn in chatHistory, and puts the results
in `toolResults`, matching the langchain-oracle reference. Tool results are no
longer represented as chatHistory entries.

Response side: tool-grounded answers come back with citations carrying
`documentIds` (camelCase) and no `document_ids`, which made the required
`CohereCitation.document_ids` field fail validation and sink the whole response
parse. Those citations are never surfaced, so the field (and CohereSearchQuery's
generation_id) is now optional.

Verified live against us-chicago-1 (cohere.command-a-03-2025 and command-latest),
single and multi-round tool loops. Adds unit regressions on the transformed
request shape and on citation parsing, plus gated integration tests for the
continuation.

* feat: integrate Repelloai Argus guardrail (BerriAI#30673)

* feat(guardrails): add RepelloAI Argus guardrail integration (#1)

* feat(guardrails): add RepelloAI Argus guardrail integration

Add a new guardrail hook backed by RepelloAI Argus, with dashboard-managed
asset policies enforced via an asset_id and X-API-Key auth.

* fix(guardrails): harden RepelloAI Argus guardrail

- scan streaming responses on output (was bypassing the guardrail)
- log blocked verdicts as guardrail_intervened instead of success
- treat auth/config errors (401/403/404/422) as misconfiguration that
  always blocks, not a fail-open-able unreachable error
- default unreachable_fallback to fail_closed and read it directly;
  block on unknown/malformed verdicts so an API change can't silently
  disable enforcement
- type unreachable_fallback as a Literal, drop the duplicate config model,
  expose unreachable_fallback in the config schema, and stop leaking the
  raw provider response / exception strings to the client

* fix(guardrails): address RepelloAI Argus review feedback

- support ARGUS_API_KEY (with REPELLOAI_API_KEY fallback)
- make asset_id required in the config model
- normalize unreachable_fallback so only fail_open opens; block on 400 misconfig
- correct the shared unreachable_fallback field description

* docs(guardrails): add RepelloAI Argus docs page and dashboard listing

- add docs page covering config, env vars, modes, verdicts, failure semantics
- list RepelloAI Argus in the Guardrail Garden with provider/logo mappings
- add a regression test for the provider logo and display-name resolution

* fix(guardrails): keep RepelloAI asset_id optional in config model

A required asset_id leaked onto the shared LitellmParams (which inherits
RepelloAIGuardrailConfigModel), breaking validation for every other
guardrail. Keep it optional like sibling models; the guardrail __init__
still raises when asset_id is missing, which is the real enforcement.

* Add comment for last user turn scanning

* feat(guardrails): harden repelloai scanning

* feat(guardrails): expand repelloai scanning to include tool definitions

Add extraction of tool definitions and tool call arguments to the RepelloAI
guardrail scanning. Improves detection coverage by including function schemas
and parameters in the prompt sent to the guardrail service. Also captures
detailed error responses in logs and adds guardrail header to streaming responses.

* refactor(guardrails): fix and harden repelloai schema text extraction

- Fix duplicate text in _iter_schema_text: previously all dict values were
  re-queued onto the stack even after scalar/list keys were already extracted
  explicitly, causing names/descriptions to appear twice in the scanned prompt
- Extract schema key frozensets to module-level constants so they are not
  reconstructed on every call
- Change _iter_schema_text from @classmethod to @staticmethod (cls unused)
- Narrow _call_analyze stage param from str to Literal["prompt", "response"]
- Add HttpxResponse type annotation to _raise_for_config_error
- Add LLMResponseTypes annotation to async_post_call_success_hook response param

* fix(guardrails): resolve pyright type errors in repelloai guardrail

- Narrow async_handler.post return from Response|None to Response with
  explicit None guard before calling raise_for_status/json
- Fix list comprehension returning str|None by switching to explicit loop
  with isinstance guard so pyright tracks the narrowing
- Cast model_dump() result to Dict since hasattr does not narrow object
  type in pyright

* fix(guardrails/repello): include Responses API instructions field in prompt scan

The /v1/responses top-level `instructions` field was not included in
_extract_prompt_text, allowing a caller to bypass guardrail policy checks
by putting blocked content in `instructions` while keeping `input` benign.

* feat: add api_key to config model and read prompt from data dict

* fix(guardrails/repello): plug input_text and tool-call response bypass gaps

Responses API input content parts with type 'input_text' were silently
dropped by build_inspection_messages (which only handles type='text'),
allowing callers to send blocked content via that path without triggering
the pre-call scan. Fix: add _extract_input_text_parts to RepelloAIGuardrail
and call it when walking the Responses API input messages.

Post-call scanning skipped responses whose choices contained only tool_calls
or function_call (message.content=None), letting models put blocked output in
function arguments undetected. Fix: _extract_chat_completion_text now calls
_extract_tool_call_args_from_message on each choice message.

Also replace typing.Dict/List with builtin dict/list to clear TID251 strict
ruff violations introduced by this file.

* fix(guardrails/repello): scan Responses API function_call output arguments

Output items with type 'function_call' in a /v1/responses response were
skipped by _extract_responses_api_text; only 'message' items were walked.
A model could return blocked content in function_call.arguments undetected.
Now extract arguments from function_call output items before scanning.

* refactor(guardrails/repello): clean up typing and remove lint-any workarounds

- Replace Optional[X]/Union[X,Y] with X|None/X|Y union syntax throughout
- Use dict[str, object] instead of bare dict in all signatures
- Remove **kwargs from __init__; declare guardrail_name, event_hook, default_on explicitly
- Replace getattr(litellm_params, ...) with direct attribute access now that LitellmParams inherits RepelloAIGuardrailConfigModel
- Add _event_hook_from_mode() to convert str|list[str]|Mode to typed GuardrailEventHooks
- Use TypeAdapter.validate_json() instead of response.json() + manual dict construction
- Add _is_object_dict/_is_object_list TypeGuard helpers to narrow object types without Any
- Remove cast() workarounds and typed intermediate variables that existed only for the now-removed lint-any CI check
- Drop _AddLiteLLMCallback Protocol; budget has sufficient slack for the one reportUnknownMemberType
- Fix GuardrailConfigModel missing type arg: GuardrailConfigModel[BaseModel]

* fix(guardrails/repello): suppress LIT007 on TypeGuard helpers and add streaming scan-skip warning

- Add guard-ok suppressions to _is_object_dict and _is_object_list to satisfy the LIT007 hard-zero budget gate
- Emit verbose_proxy_logger.warning when the streaming hook finds no inspectable text after assembly, matching observability of pre/post hooks

* refactor: modifications for lint check

* feat: add Pinstripes as an OpenAI-compatible provider (BerriAI#30567)

* feat: add Pinstripes as an OpenAI-compatible provider

Pinstripes (https://pinstripes.io) is an OpenAI-compatible inference
provider serving open-source models (GLM-4.5-Air, Qwen3, DeepSeek, etc.)
with per-token pricing and no subscriptions.

Changes:
- `litellm/llms/openai_like/providers.json`: register pinstripes with
  base_url, api_key_env, and max_completion_tokens→max_tokens mapping
- `litellm/types/utils.py`: add `PINSTRIPES = "pinstripes"` to LlmProviders
- `litellm/constants.py`: add to openai_compatible_providers and
  openai_compatible_endpoints lists
- `litellm/litellm_core_utils/get_llm_provider_logic.py`: auto-detect
  provider when api_base is "https://pinstripes.io/v1"
- `provider_endpoints_support.json`: document supported endpoints
- `tests/`: 7 unit tests covering provider registration, resolution,
  URL auto-detection, api_base override, and Router config

Usage:
    import litellm
    response = litellm.completion(
        model="pinstripes/ps/glm-4.5-air",
        messages=[{"role": "user", "content": "Hello"}],
        api_key=os.environ["PINSTRIPES_API_KEY"],
    )


* fix(pinstripes): resolve Greptile P1 review comments

- Add api_base_env: PINSTRIPES_API_BASE to providers.json so env var override works
- Set responses: false in provider_endpoints_support.json — not actually wired up
- Remove docs/my-website/docs/providers/pinstripes.md — belongs in litellm-docs repo


* fix(pinstripes): add api_base_env and correct responses capability

- Add api_base_env: PINSTRIPES_API_BASE to providers.json
- Set responses: false in provider_endpoints_support.json


* fix(pinstripes): wire up Responses API — add supported_endpoints

Adds supported_endpoints: ["/v1/chat/completions", "/v1/responses"] so
JSONProviderRegistry.supports_responses_api returns true correctly,
matching what provider_endpoints_support.json advertises.


* feat(pinstripes): enable embeddings endpoint

Pinstripes serves nomic-embed-text-v1.5 and bge-m3 via /v1/embeddings.
Add /v1/embeddings to supported_endpoints and set embeddings: true.


* fix(pinstripes): use 4-space indentation in model_prices_and_context_window.json

Matches the file's existing convention. Flagged by Greptile review.


* fix(pinstripes): set a2a: false — A2A protocol not implemented

All comparable JSON-configured providers (tensormesh, parasail, empiriolabs,
libertai, neosantara) have a2a: false. Pinstripes does not implement the
Google A2A protocol, so this should be false to match.


---------

Co-authored-by: inference_provider <[email protected]>

* fix(rag): attach existing OpenAI file ids (BerriAI#30628)

* fix(rag): attach existing OpenAI file ids

* chore: use modern typing in rag ingest fix

* chore: retrigger ci

* fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (BerriAI#30341)

cache_control_injection_points was only consumed by the chat/completions
prompt-management hook; on the native Anthropic /v1/messages path it was
forwarded unused, so deployment-level cache injection was silently dropped
(cache_creation_input_tokens stayed 0 for Anthropic-native clients).

Add AnthropicCacheControlHook.apply_to_anthropic_messages_request to inject
cache_control at block level for system / tools / message locations (the only
forms /v1/messages accepts), wire it into the native anthropic_messages
handler, and pop the param so it does not leak upstream as an unknown field.
A {location: message, role: system} config is redirected to the top-level
system prompt so the same YAML works on both endpoints.

Injection respects Anthropic's 4-block cache_control limit shared across
system, tools, and messages: client-supplied markers count toward the cap and
are never overwritten, a slot is reserved per Bedrock tool_config point, and
injection stops once the budget is exhausted. Locations this path cannot
represent (tool_config) are forwarded downstream instead of being silently
consumed, mirroring get_chat_completion_prompt's remaining_points pass-through.

Built on litellm_internal_staging. Refs BerriAI#30293

* fix(proxy): release budget reservation when a request is cancelled mid-flight (BerriAI#30522)

* fix(proxy): release budget reservation on cancel when no chunk was delivered

The pre-call budget reservation increments the cross-pod spend counter by a
request's worst-case cost, then reconciles it on success (cost callback) or
error (failure hook). A client disconnect or timeout cancels the request and
surfaces as CancelledError / GeneratorExit, which neither path catches, so the
reservation leaks. Under a retry storm the leaked holds accumulate, pin the
counter above real spend, and return spurious 429 "Budget has been exceeded" to
keys whose spend is far below budget; the counter only recovers when its TTL
lapses, so the failure is intermittent and self-healing.

Release the reservation in async_streaming_data_generator (which the Anthropic
and Google SSE generators delegate to) on the (CancelledError, GeneratorExit)
path, alongside the existing max_parallel_requests release. release_budget_
reservation_on_cancel runs under asyncio.shield so it completes despite the
in-progress cancellation, is guarded by the reservation's finalized flag, and
swallows a failing release so it cannot replace the in-flight cancellation.

The refund is gated on whether a chunk reached the client. The flag is set
immediately before the yield, after the slow-path hook await: an async generator
suspends at the yield, so a GeneratorExit on disconnect after a delivered chunk
sees it True (keep the hold), while a cancellation during the slow-path await
leaves it False (refund, nothing sent). A non-streaming cancellation delivers
nothing and a completed non-streaming response is reconciled by the success
callback, so neither needs a release here.

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* fix(proxy): reconcile a cancelled reservation to input cost, not zero

A streaming request cancelled before the first chunk previously reconciled its
reservation to zero and finalized it. But by the time the generator is
consuming the response the provider call was already dispatched, so the input
tokens were billed even though no chunk reached the client, and the
success/failure cost callbacks are skipped on cancellation. Refunding to zero
let a caller send an expensive request and abort pre-token to dodge the input
charge.

Compute the request's input-token cost at reservation time and reconcile the
cancelled reservation to it instead of zero. The worst-case output portion of
the reservation is still released (so a legitimate mid-flight cancellation no
longer pins the counter and 429s the key), while the input the provider already
processed is charged.

---------

Co-authored-by: Bytechoreographer <[email protected]>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* fix(caching): encode object name in GCS cache GET path (BerriAI#30378)

GCS cache reads always missed when gcs_path was set. The GET methods
interpolated the object name directly into the URL path, while the GCS
JSON API requires it to be URL-encoded (a "/" must be sent as %2F).

With gcs_path configured the object name is "<prefix>/<sha256>", so the
raw slash produced a malformed object path and GCS returned 404. httpx
does not raise on 4xx, so the status_code == 200 check fell through and
get/async_get returned None, silently missing on every read. Without
gcs_path the key has no slash, which is why this went unnoticed.

Wrap the object name with urllib.parse.quote(..., safe="") in get_cache
and async_get_cache. Apply the same encoding to the name= query
parameter in set_cache and async_set_cache so the key written matches
the key read back.

Adds regression tests asserting the GET path and SET query are encoded
(%2F) when gcs_path is set, for both sync and async paths; these fail on
the unpatched code.

Fixes BerriAI#30377

* chore: add soniox stt-async-v5 model (BerriAI#30672)

* fix(proxy): include model group aliases in v1 model info (BerriAI#30626)

* Include model group aliases in v1 model info

* Fix model info alias implementation

* removed extra blank line

* chore: rerun CI

* fix(lint): remove redundant noqa directive in proxy_cli.py

* fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme

* Revert "fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme"

This reverts commit 52c7a07.

* Revert "fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (BerriAI#30341)"

This reverts commit c9e8a17.

* Revert "fix(proxy): stop IAM-refresh engine restart from cascading reconnects (BerriAI#29176) (BerriAI#30183)"

This reverts commit 85828da.

* fix(proxy): stop IAM-refresh engine restart from cascading reconnects (BerriAI#29176) (BerriAI#30183)

An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the
running query-engine and spawns a new one. That planned kill was
indistinguishable from a crash, and three reconnect paths used two
uncoordinated locks, so a single refresh triggered a cascade of engine
kill/respawn cycles:

  1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old
     engine, spawn new one.
  2. The engine-death watcher sees that kill, assumes a crash, and calls
     `attempt_db_reconnect(force=True)` (a different lock,
     `_db_reconnect_lock`) -> recreate again -> kills the fresh engine.
  3. In-flight queries failing during the swap are classified as transport
     errors and trigger their own `attempt_db_reconnect` -> recreate again.

Fix coordinates planned restarts across the wrapper and the watcher:

  - PrismaWrapper records the old engine PID in `_expected_engine_deaths`
    before killing it; all four watcher death-detectors (waitpid thread,
    pidfd, already-dead probe, os.kill poll) consume that PID and skip the
    reconnect instead of treating it as a crash.
  - `recreate_prisma_client` now serializes through `_reconnection_lock` and
    bumps a monotonic `_engine_generation`. Callers pass `expected_generation`
    as an optimistic-lock token, so racing/cascading recreates collapse into a
    single restart (losers no-op). This closes the two-lock gap.
  - The direct reconnect path probes the writer with SELECT 1 before
    recreating; a healthy connection (e.g. engine already replaced by a
    refresh) skips the recreate entirely.
  - `_safe_refresh_token` coalesces: it skips when the current token still has
    more than the refresh buffer of runway, so stacked triggers (proactive
    loop + __getattr__ fallback) don't each restart the engine. An
    `on_engine_replaced` hook re-arms the watcher on the new PID.

RoutingPrismaWrapper forwards `expected_generation` and skips recreating the
reader when the writer recreate was skipped.

* fix(lint): modernize type annotations in IAM-refresh prisma client files (UP006/UP045)

* Revert "feat(proxy): show session-aggregate cost and duration in request logs (BerriAI#25708) (BerriAI#30507)"

This reverts commit f530b22.

* Revert "fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (BerriAI#30653)"

This reverts commit 4f58bd0.

* Revert "fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (BerriAI#30646)"

This reverts commit 50f34e0.

* Revert "fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (BerriAI#30366)"

This reverts commit 0544eed.

* fix(bedrock_mantle): restore BedrockMantleAuthMixin and constants removed by routing rewrite

* fix(key management): restore exact /key/list user_id & key_alias matching by default (BerriAI#30593)

Before substring search was added (commit 33bd570), /key/list matched user_id
and key_alias exactly. That change made admin-authenticated calls substring-match
by default, breaking the prior contract: a caller passing an exact user_id as an
access filter (e.g. an integration scoping to one user with an admin key) then
received other users' keys -- user_id="alice" also returned "alice2",
"alice-test", etc. This is a cross-user key disclosure.

Make substring matching opt-in via a new admin-only substring_matching=true query
param; default to exact, restoring the prior behavior. The dashboard search box
(keyListCall) passes the flag so partial search still works. Non-admins remain
exact and scoped to their own keys.

Updates the proxy-behavior key_alias test to opt in and adds an exact-by-default
guard; adds list_keys unit coverage for the opt-in gate.

---------

Co-authored-by: perseus <[email protected]>
Co-authored-by: Hannah Smith <[email protected]>
Co-authored-by: Charlie Patterson <[email protected]>
Co-authored-by: Matthew Lapointe <[email protected]>
Co-authored-by: KRISH SONI <[email protected]>
Co-authored-by: Yash Raj Pandey <[email protected]>
Co-authored-by: Nitish Agarwal <[email protected]>
Co-authored-by: hcl <[email protected]>
Co-authored-by: tushar8408 <[email protected]>
Co-authored-by: AD Mohanraj <[email protected]>
Co-authored-by: Fede Kamelhar <[email protected]>
Co-authored-by: Lavish Bansal <[email protected]>
Co-authored-by: max-amos <[email protected]>
Co-authored-by: inference_provider <[email protected]>
Co-authored-by: NK <[email protected]>
Co-authored-by: 安妮的心动录 <[email protected]>
Co-authored-by: Rick <[email protected]>
Co-authored-by: Bytechoreographer <[email protected]>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Burak Ömür <[email protected]>
Co-authored-by: Dan Lemon <[email protected]>
Co-authored-by: Vanika Dangi <[email protected]>
Co-authored-by: Jay Gowdy <[email protected]>

…rvers_url_with_rootpath

fix: fixed the issue of handling root paths when processing Discovery…

…26_2026

fix(proxy): support slashes in google generateContent model names (#1…

…gmnt (#1…" (BerriAI#20393)

This reverts commit 76a399b.

…03_2026

feat(guardrails): implement team-based isolation guardrails mgmnt (#1…

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

Veria admin-queue finding E3NpkuAd, Audit-B #1. The route-level gate
accepts this call when the caller is PROXY_ADMIN or ORG_ADMIN of any
org named in request_data["organization_id"]/["organizations"]. The
handler processes data.user_ids without cross-checking whether those
users belong to the caller's administered orgs, so an org-admin of
org-A could delete users in org-B via:
  {"user_ids": ["victim_in_org_B"], "organization_id": "org-A"}

Add per-target authorization: org-admins may only delete users whose
entire org membership is within their admin scope; targets with any
org outside scope (or no org at all) require PROXY_ADMIN.

Regression test confirms an ORG_ADMIN call fails with 403 and no
cascade delete_many runs.

Continuation of Veria E3NpkuAd / Audit-B hardening. All three are the
same anti-pattern PR BerriAI#25904 already addressed for _user_is_org_admin
and /user/delete: route-level gate trusts a caller-supplied scope
field, handler operates on a different scope.

1. /user/update no longer silently creates a user when the target
   email doesn't exist. Pre-fix, an org admin could supply a fresh
   email + caller-chosen budget/models/metadata; the INSERT path
   created the user with no org attachment, bypassing /user/new's
   org/team authorization. Now require PROXY_ADMIN for the create
   branch; return 404 otherwise. Also fixes /user/bulk_update because
   it dispatches through the same _update_single_user_helper.

2. /team/bulk_member_add with all_users=true restricted to PROXY_ADMIN.
   The flag pulls every user in the database into the target team,
   ignoring org scope — any team admin could use it to capture every
   user across every org into their team.

3. /team/update now verifies destination-org admin rights. When the
   request carries an organization_id that differs from the team's
   current org, an org admin of the caller's current org could
   previously relocate the team into any other org (draining their
   resources, or capturing a team they once administered). Require
   PROXY_ADMIN or org-admin of the DESTINATION org for the relocation.

Regression tests for #1 and BerriAI#3; BerriAI#2 covered by the existing bulk_add
suite after the gate addition.

* fix(proxy): allow non-admin virtual keys to call GA Realtime WebRTC HTTP routes (BerriAI#30089)

* fix(proxy): allow non-admin virtual keys to call GA Realtime WebRTC HTTP routes

Add the realtime WebRTC HTTP sub-routes (/realtime/client_secrets,
/realtime/calls and their /v1 + /openai/v1 variants) to
LiteLLMRoutes.openai_routes so is_llm_api_route() classifies them as
LLM API routes. Without this, non-admin virtual keys received
401 'Only proxy admin can be used to generate, delete, update info
for new keys/users/teams' when calling these endpoints.

Fixes BerriAI#29923

* fix(proxy): validate session.model for realtime routes in model-access check

The GA Realtime WebRTC HTTP routes resolve the effective model from the
nested session.model (falling back to the top-level model), but the auth
layer's get_model_from_request() only extracted the top-level model. A
model-restricted virtual key could therefore place a disallowed model in
session.model, leave the top-level model unset, and skip can_key_call_model()
entirely - obtaining an ephemeral token for a model it is not allowed to use.

Extract session.model for the realtime client_secrets/calls routes so the
model-access check runs against the model the request will actually use.
Legitimate callers are unaffected; their permitted model still validates.

Relates to BerriAI#29923

* fix(proxy): classify realtime transcription_sessions routes as LLM API routes

Add the GA Realtime WebRTC transcription_sessions HTTP routes to
openai_routes so is_llm_api_route() returns True for them, matching the
client_secrets and calls routes already fixed. These endpoints are
registered with user_api_key_auth in realtime_endpoints/endpoints.py, so
without this a non-admin virtual key calling
POST /v1/realtime/transcription_sessions would hit the admin-only 401
branch. Extends the regression test parametrization accordingly.

---------

Co-authored-by: habonlaci <[email protected]>

* feat(proxy): surface max_input_tokens/max_output_tokens on /v1/models (BerriAI#30272)

* feat(proxy): surface max_input_tokens/max_output_tokens on /v1/models

* fix(proxy): degrade /v1/models gracefully when model-group lookup fails

---------

Co-authored-by: Sameer Kankute <[email protected]>

* fix: sort tiered token-cost thresholds numerically (BerriAI#30375)

* fix: sort tiered token-cost thresholds numerically

_get_token_base_cost iterated input_cost_per_token_above_<N>_tokens keys with a
lexicographic sort, so for tiers whose thresholds have different digit lengths
(e.g. 90k vs 128k) a request crossing both was billed at the lower tier that
sorted first. Sort by the parsed numeric threshold instead, so the highest tier
the request actually crosses is applied.

* refactor: reuse _parse_above_token_threshold for inline threshold parse

---------

Co-authored-by: Eric (GabiDevFamily) <[email protected]>

* fix(openai): preserve cache_control for openai-compatible custom endpoints (BerriAI#30387)

* fix(openai): preserve cache_control for openai-compatible custom endpoints

* fix(openai): use parsed hostname to detect real OpenAI for cache_control preservation

* fix(proxy): drain all daily-spend batches per flush cycle (BerriAI#30281) (BerriAI#30505)

* fix(types): prevent internal parallel_request_limiter fields from leaking to upstream providers (BerriAI#30545)

* fix(types): add internal parallel_request_limiter fields to all_litellm_params to prevent forwarding to upstream providers

* test(types): add regression test for internal rate-limit fields in all_litellm_params

* fix(init): add bool type annotation to suppress_debug_info (BerriAI#30531)

Module-level `suppress_debug_info = False` had no annotation, so strict
type checkers (e.g. ty) infer it as `Literal[False]`. Reassigning it to
`True` (as done in proxy_server.py and router.py) then fails with an
invalid-assignment error. Annotate it as `bool` to match every other
flag in this module.

* fix: coalesce null aggregates in update_metrics for no-spend keys (BerriAI#29945)

* feat(team_endpoints): add query parameter `key_limit` to `/team/info` endpoint (BerriAI#30006)

* feat(team_endpoints): Add query parameter key_limit to /team/info

* feat(team_endpoints): update schema.d.ts to include the new query parameter

* feat(team_endpoints): add tests for limitting key count in /team/info response

* feat(team_endpoints): Apply suggestions from greptile

* Set greater-than constraint on key-limit
* Fix type

* fix(router): release aiohttp connection when stream iteration ends abnormally (BerriAI#30271)

* fix(router): release aiohttp connection when stream iteration ends abnormally

A streaming response that terminates with a mid-stream read timeout, a task
cancellation (client disconnect), or GeneratorExit never closed the underlying
aiohttp ClientResponse. aiohttp only auto-releases the connector slot at body
EOF, so each abnormally terminated stream permanently leaked one slot from the
shared TCPConnector pool. During a backend traffic spike the pool drains; once
exhausted every subsequent request to that host waits for a slot, times out
and surfaces as a 408, indefinitely, even after the backend recovers. Only a
proxy restart cleared the in-memory sessions, which matched the reported
symptom of a router stuck returning 408 for a healthy vLLM backend.

Close the response in a finally clause when iteration ends. On a fully read
response the connection was already released at EOF and close() is a no-op,
so keep-alive reuse for normal requests is unchanged.

Fixes BerriAI#30192

* test(aiohttp): cover GeneratorExit path with a mock instead of a live socket

The previous slot-release test started a real aiohttp TCP server, which can
flake in offline CI and does not exercise this fix's code path directly.
Replace it with a dependency-injected mock that closes the stream generator
(GeneratorExit) and asserts the response is closed, covering the third
abnormal-exit path the finally block handles

* feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery (BerriAI#30273)

* feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery

* refactor(proxy): move Anthropic model-list formatter into llms/anthropic/common_utils

* fix(proxy): make model_list request param optional for direct callers

* feat(dashscope): add Responses API support (BerriAI#30286)

* feat(dashscope): add Responses API support

DashScope's OpenAI-compatible endpoint serves /responses, so register a
DashScopeResponsesAPIConfig that routes dashscope/* responses calls to
{api_base}/responses without rewriting the upstream model id, instead of
falling back to the chat-completions -> responses emulation pipeline.

Closes BerriAI#29780

* feat(dashscope): mark responses API as not supporting native websocket

Matches the hosted_vllm/perplexity/openrouter responses configs, which all
override supports_native_websocket() to False since the OpenAI-compatible
endpoint has no native wss:// responses transport.

---------

Co-authored-by: Sameer Kankute <[email protected]>

* fix(spend-logs): preserve error_message on ProxyException failures (BerriAI#30381)

* fix(spend-logs): preserve error_message on ProxyException failures

`StandardLoggingPayloadSetup.get_error_information` used
`str(original_exception)` to populate the human-readable error message
stored in `spend_logs.metadata.error_information.error_message`.

`ProxyException` (litellm/proxy/_types.py:3453) sets `self.message` in
its constructor but does NOT call `super().__init__(message)` and does
NOT define `__str__`. As a result, `str(ProxyException(...))` returns
the empty string, and every auth/budget/quota rejection was landing
in spend_logs with `error_message=""` despite a fully populated
traceback.

Operator impact: dashboard "LLM Failure" rows became untriageable —
the only way to tell a 401 from a 429 was to manually unpack the
traceback JSON via psql. Burst failure patterns (e.g. a UI session
polling with a stale token) produced 20-30 indistinguishable
`error_code=401` rows per second.

Fix: prefer the `.message` attribute (set by ProxyException and every
litellm.exceptions.* class) over `str(exc)`. The `str(exc)` fallback
is retained for non-litellm exception types, preserving prior behavior.

Test plan:
  - 2 new unit tests in tests/test_litellm/litellm_core_utils/
    test_litellm_logging.py:
    * test_get_error_information_prefers_message_attribute_over_str
    * test_get_error_information_falls_back_to_str_when_no_message_attr
  - Existing test_get_error_information_error_code_priority still passes
  - End-to-end verified: bad-key 401 now stores full
    "Authentication Error, Invalid proxy server token passed..."
    message in spend_logs.metadata.error_information.error_message

* fix(spend-logs): preserve explicit empty .message + drop dead reference

Greptile P2 on BerriAI#30381. The truthiness check `if message_attr:`
silently skipped an explicit empty-string `.message` and fell
through to `str(original_exception)`. For ProxyException-shaped
objects both produce empty, so the bug was latent; for other
exception types it would inject a different string into
error_information.error_message and corrupt the signal.

Use `is not None` so an empty string survives verbatim.

Also drop the stale `See e2e/cases/11.` comment reference — that
path does not exist anywhere in the repo and confuses future
readers.

Regression test added: an exception with `.message=""` and a
non-empty `super().__init__()` arg must yield error_message == "".

* ci: retrigger workflows after base branch change to litellm_internal_staging

* fix(anthropic): strip LiteLLM-injected total_tokens from /v1/messages response (BerriAI#30382)

* fix(anthropic): strip LiteLLM-injected total_tokens from /v1/messages response

The non-streaming /v1/messages response carries a LiteLLM-injected
usage.total_tokens = input_tokens + output_tokens that is not part of
the Anthropic API spec. This caused three problems:

1. Shape divergence with streaming on the same endpoint.
   message_delta.usage in the SSE path never carries total_tokens.
   Clients parsing both paths get two different schemas from one endpoint.

2. Shape divergence with upstream. Direct calls to
   https://api.anthropic.com/v1/messages return no total_tokens field,
   so clients using the official Anthropic SDK couldn't rely on it,
   and clients that did rely on the LiteLLM-injected one broke when
   bypassing the proxy.

3. Numerical misuse. total = input + output undercounts when
   cache_read_input_tokens and cache_creation_input_tokens are
   non-zero, because cache tokens are reported in their own fields.
   A 100k-token cached prompt with 1 non-cache input token + 200
   output tokens reports total_tokens = 201, off by ~99.8% from any
   reasonable definition of "total."

Fix: add _strip_total_tokens_from_anthropic_response in
litellm/proxy/anthropic_endpoints/endpoints.py and invoke it in the
success path of anthropic_response right before returning. Only mutates
dict-shaped responses; streaming (which already lacks the field) is
left untouched.

spend_logs / Prometheus continue to compute total_tokens internally
for billing — this fix only strips the field from the wire response.

Scope: only the Anthropic passthrough endpoint /v1/messages. The
OpenAI-shape /v1/chat/completions is unaffected.

* fix(anthropic): gate total_tokens strip behind flag + handle Pydantic .usage

Two P1 greptile threads on BerriAI#30382:

P1 — **Backwards-incompatible removal without a feature flag**
  Stripping `usage.total_tokens` unconditionally breaks any client
  currently reading the LiteLLM-shaped non-streaming /v1/messages
  response. Per the codebase's policy (mirrors BerriAI#30418), gate behind
  a new flag.

  - `litellm.strip_anthropic_total_tokens: bool = False` (default —
    backward-compat: clients keep seeing total_tokens).
  - Env override: `LITELLM_STRIP_ANTHROPIC_TOTAL_TOKENS=true`.
  - Docstring: planned to flip to True in a future major release;
    opt in early.

P1 — **Silent no-op if `result` is a Pydantic model**
  `base_process_llm_request` may return a Pydantic-style object
  whose `.usage` is a plain dict (the most common shape — e.g.
  objects wrapping raw upstream JSON). The original
  `isinstance(response, dict)` guard skipped strip on those, so
  `total_tokens` would still hit the wire. Helper now also reads
  `getattr(response, "usage", None)` and strips when that's a dict.

  Strongly-typed Pydantic `Usage` sub-models with required
  `total_tokens` fields are still skipped — those impose type
  constraints the helper doesn't try to subvert.

Tests:
- `test_strips_total_tokens_on_pydantic_model_with_dict_usage`
- `test_flag_defaults_off`
8/8 pass locally.

* fix(anthropic): drop env var for strip flag (docs CI)

Mirrors BerriAI#30418's pattern (`expose_router_debug_in_errors: bool = True`,
no `os.getenv`). The `LITELLM_STRIP_ANTHROPIC_TOTAL_TOKENS` env var
introduced in the prior commit was flagged by
`tests/documentation_tests/test_env_keys.py` because the documentation
file `docs/my-website/docs/proxy/config_settings.md` lives in
`BerriAI/litellm-docs` (separate repo) and registering a new env key
requires a parallel docs PR — a friction we avoid here by exposing
the flag only as a Python attribute + `litellm_settings` config key,
both of which load through the existing proxy config plumbing without
needing the env-var registry to be updated.

No semantic change: default still False, behavior identical when set
via `litellm.strip_anthropic_total_tokens = True` or
`litellm_settings.strip_anthropic_total_tokens: true` in config.yaml.

Verified locally: env scan no longer surfaces the key; 8/8 tests pass.

* ci: retrigger workflows after base branch change to litellm_internal_staging

* fix(pricing): correct swapped input/output token costs for command-r7b-12-2024 (BerriAI#30413)

* fix(pricing): correct swapped input/output token costs for command-r7b-12-2024

* test: resolve model prices JSON relative to test file for pip installs

* fix(exception-mapping): map Gemini upstream-error body code 429 to RateLimitError (BerriAI#30417)

* fix(exception-mapping): map Gemini upstream-error body code 429 to RateLimitError

Some Gemini-compatible gateways (e.g. new-api) wrap a 429 rate-limit
signal from upstream inside an HTTP 500/503 envelope, with the real
code only surfaced in the JSON body:

    {"error":{"message":"...high demand...","type":"upstream_error",
              "param":"","code":429}}

Previously LiteLLM only looked at the HTTP status and mapped this to
InternalServerError, which Router treats as non-retryable for many
configs — so users got hard 500s instead of fallback/retry.

Now the Gemini/Vertex exception mapper parses error.code from the body
and routes code 429 to RateLimitError before falling through to the
HTTP-status branches. Other body codes fall through unchanged.

Tests cover:
- new-api gateway's `code:429` payload now maps to RateLimitError
- Genuine 500-body responses stay InternalServerError
- Non-JSON body strings fall through to status-code mapping unchanged

* fix(exception-mapping): scope body-code 429 promotion to 5xx envelopes

Addresses greptile P1/P2 + @Sameerlite's review on BerriAI#30417. The new
elif branch was firing for any HTTP status, so a gateway response of
HTTP 400 with body {"error":{"code":429,...}} would be incorrectly
promoted to RateLimitError (retryable) instead of falling through
to BadRequestError. Same trap for 401 -> AuthenticationError.

Scoped the body-code 429 check to `500 <= status_code < 600` —
covers 500/502/503/504 (gateways wrapping upstream 429 in any 5xx
envelope) without inviting the 4xx misclassification.

Tests: parametrized table now covers 5xx (500/502/503), 4xx (400/401),
and the existing fall-through cases, asserting each maps to the
exception type that matches the HTTP status code. 50/50 pass locally.

* ci: retrigger workflows after base branch change to litellm_internal_staging

* feat(router): add expose_router_debug_in_errors flag (default True) to redact internal model_group/fallback names (BerriAI#30418)

* feat(router)!: redact internal model_group/fallback names from exception messages

The Router was unconditionally appending internal config names onto
exception.message:
  - "Received Model Group=..."
  - "Available Model Group Fallbacks=..."
  - "No fallback model group found... Fallbacks={...}"
  - "context_window_fallbacks={...}"
  - Deployment-timeout messages including model_group
  - Fallback failure detail listing fallback chain

ProxyException forwards .message verbatim to clients, so gateways were
leaking their model_name / fallback wiring in every failed call.

Fix: gate all five mutation sites on a new
`litellm.expose_router_debug_in_errors` flag (default False). Set to
True to restore upstream debug behavior for local debugging.

Why: matches the redaction posture this codebase already has for
upstream model identifiers (cf. _litellm_returned_model_name) and
removes the last common error-path leak of internal model_group names.

Breaking change marker (!): if anything parses "Received Model Group="
out of client error messages, flip the flag on or migrate to the
x-litellm-* response headers instead.

Tests: 7 cases covering each of the 5 redaction sites + the flag-on
inverse path, plus a "default off" sanity check.

* test(router): cover sites 1 + 3 of expose_router_debug_in_errors gate

Addresses Greptile / codecov feedback on BerriAI#30418: patch coverage was
55.6% with 4 lines uncovered in litellm/router.py. The existing tests
exercised sites 2 (ContextWindowExceededError), 4 (no-fallback-found),
and 5 (Received Model Group) — both default and flag-on. Sites 1 and 3
were declared in the PR description as covered by "site 5 also fires"
but the gate body lines for each (the `e.message +=` inside the
`if litellm.expose_router_debug_in_errors:` branch) only execute when
the flag is on AND the specific exception path is taken, which neither
existing test triggered.

Added 4 new tests (default + flag-on × 2 sites):

  - test_default_does_not_leak_deployment_timeout_debug
  - test_flag_on_leaks_deployment_timeout_debug
  - test_default_does_not_leak_content_policy_fallback_hint
  - test_flag_on_leaks_content_policy_fallback_hint

Trigger details:

  - Site 1 (litellm.Timeout in _acompletion) is reached via the
    Router-supported `mock_timeout=True` + `timeout=0.001` kwargs on
    `acompletion(...)`. Cannot embed a Timeout instance in model_list
    because Router.__init__ deep-copies it and Timeout.__reduce__ does
    not preserve the required positional args.
  - Site 3 (ContentPolicyViolationError without content_policy_fallbacks
    set, in async_function_with_fallbacks_common_utils) is reached by
    passing a `mock_response=litellm.ContentPolicyViolationError(...)`
    instance via the call-site kwarg — same deepcopy-avoidance reason.

11/11 tests pass locally. Patch coverage on litellm/router.py for this
PR's diff should now be 100%.

* chore(router): flip expose_router_debug_in_errors default to True

Addresses @Sameerlite's review on BerriAI#30418 — maintain backward
compat on the wire. Redact becomes opt-in via setting the flag
to False; the historical behavior (leak internal model_group /
fallback wiring through exception messages) is preserved as the
default.

- litellm/__init__.py: default flipped to True, docstring rewritten
  with deprecation note pointing at a future flip to False (redact
  by default) in a major release.
- tests/test_litellm/test_router_exception_redaction.py: fixture
  resets to True (was False); the "off" tests now explicitly set
  False; the "default_leaks_*" tests rely on the fixture default.
  test_flag_defaults_off -> test_flag_defaults_on.
- No router.py change needed; the gate keys off the same flag,
  only the default changes.
- PR title no longer needs the breaking-change `!` marker — no
  client sees a behavior change at default settings.

11/11 pass locally.

* ci: retrigger workflows after base branch change to litellm_internal_staging

* feat(guardrails): integrate Repelloai Argus guardrail (BerriAI#30465)

* feat(guardrails): add RepelloAI Argus guardrail integration (#1)

* feat(guardrails): add RepelloAI Argus guardrail integration

Add a new guardrail hook backed by RepelloAI Argus, with dashboard-managed
asset policies enforced via an asset_id and X-API-Key auth.

* fix(guardrails): harden RepelloAI Argus guardrail

- scan streaming responses on output (was bypassing the guardrail)
- log blocked verdicts as guardrail_intervened instead of success
- treat auth/config errors (401/403/404/422) as misconfiguration that
  always blocks, not a fail-open-able unreachable error
- default unreachable_fallback to fail_closed and read it directly;
  block on unknown/malformed verdicts so an API change can't silently
  disable enforcement
- type unreachable_fallback as a Literal, drop the duplicate config model,
  expose unreachable_fallback in the config schema, and stop leaking the
  raw provider response / exception strings to the client

* fix(guardrails): address RepelloAI Argus review feedback

- support ARGUS_API_KEY (with REPELLOAI_API_KEY fallback)
- make asset_id required in the config model
- normalize unreachable_fallback so only fail_open opens; block on 400 misconfig
- correct the shared unreachable_fallback field description

* docs(guardrails): add RepelloAI Argus docs page and dashboard listing

- add docs page covering config, env vars, modes, verdicts, failure semantics
- list RepelloAI Argus in the Guardrail Garden with provider/logo mappings
- add a regression test for the provider logo and display-name resolution

* fix(guardrails): keep RepelloAI asset_id optional in config model

A required asset_id leaked onto the shared LitellmParams (which inherits
RepelloAIGuardrailConfigModel), breaking validation for every other
guardrail. Keep it optional like sibling models; the guardrail __init__
still raises when asset_id is missing, which is the real enforcement.

* Add comment for last user turn scanning

* feat(guardrails): harden repelloai scanning

* feat(guardrails): expand repelloai scanning to include tool definitions

Add extraction of tool definitions and tool call arguments to the RepelloAI
guardrail scanning. Improves detection coverage by including function schemas
and parameters in the prompt sent to the guardrail service. Also captures
detailed error responses in logs and adds guardrail header to streaming responses.

* refactor(guardrails): fix and harden repelloai schema text extraction

- Fix duplicate text in _iter_schema_text: previously all dict values were
  re-queued onto the stack even after scalar/list keys were already extracted
  explicitly, causing names/descriptions to appear twice in the scanned prompt
- Extract schema key frozensets to module-level constants so they are not
  reconstructed on every call
- Change _iter_schema_text from @classmethod to @staticmethod (cls unused)
- Narrow _call_analyze stage param from str to Literal["prompt", "response"]
- Add HttpxResponse type annotation to _raise_for_config_error
- Add LLMResponseTypes annotation to async_post_call_success_hook response param

* fix(guardrails): resolve pyright type errors in repelloai guardrail

- Narrow async_handler.post return from Response|None to Response with
  explicit None guard before calling raise_for_status/json
- Fix list comprehension returning str|None by switching to explicit loop
  with isinstance guard so pyright tracks the narrowing
- Cast model_dump() result to Dict since hasattr does not narrow object
  type in pyright

* fix(guardrails/repello): include Responses API instructions field in prompt scan

The /v1/responses top-level `instructions` field was not included in
_extract_prompt_text, allowing a caller to bypass guardrail policy checks
by putting blocked content in `instructions` while keeping `input` benign.

* feat: add api_key to config model and read prompt from data dict

* fix(guardrails/repello): plug input_text and tool-call response bypass gaps

Responses API input content parts with type 'input_text' were silently
dropped by build_inspection_messages (which only handles type='text'),
allowing callers to send blocked content via that path without triggering
the pre-call scan. Fix: add _extract_input_text_parts to RepelloAIGuardrail
and call it when walking the Responses API input messages.

Post-call scanning skipped responses whose choices contained only tool_calls
or function_call (message.content=None), letting models put blocked output in
function arguments undetected. Fix: _extract_chat_completion_text now calls
_extract_tool_call_args_from_message on each choice message.

Also replace typing.Dict/List with builtin dict/list to clear TID251 strict
ruff violations introduced by this file.

* fix(guardrails/repello): scan Responses API function_call output arguments

Output items with type 'function_call' in a /v1/responses response were
skipped by _extract_responses_api_text; only 'message' items were walked.
A model could return blocked content in function_call.arguments undetected.
Now extract arguments from function_call output items before scanning.

* fix(anthropic): drop orphaned server_tool_use on multi-turn replay from generic OpenAI clients (BerriAI#30486)

* fix(anthropic): drop orphaned server_tool_use on multi-turn replay from generic OpenAI clients

When an Anthropic server-side tool (web_search, id `srvtoolu_...`) is used, its
result is carried in `provider_specific_fields.web_search_results` — PRs BerriAI#17746
/ BerriAI#17798 restore it for callers that round-trip provider_specific_fields. A
generic OpenAI client that does NOT preserve provider_specific_fields (e.g. Open
WebUI talking to a Vertex/Anthropic model over /chat/completions) drops it on
replay and instead sends back an assistant `tool_call` + a `tool` message both
keyed to the `srvtoolu_` id. The transform then produced a bare `server_tool_use`
(with no following *_tool_result) plus a user `tool_result` for the same id —
both invalid, so the next turn 400s:

  messages.N.content.0: unexpected `tool_use_id` found in `tool_result` blocks:
  srvtoolu_... Each `tool_result` block must have a corresponding `tool_use`
  block in the previous message.

This is the commonly-reported vertex_ai symptom where Gemini works but Claude
400s on the 2nd turn of a web-search chat.

Fix (litellm/litellm_core_utils/prompt_templates/factory.py):
- convert_to_anthropic_tool_invoke: only emit a server_tool_use when its matching
  *_tool_result is available to pair with it; otherwise skip it (a bare
  server_tool_use is itself rejected).
- anthropic_messages_pt: drop a replayed `tool`/`function` message whose
  tool_call_id starts with `srvtoolu_` (a server-executed tool produces no client
  result; a user tool_result for it is invalid).

The existing reconstruction path (provider_specific_fields present, e.g. the
litellm SDK) is unchanged, as is regular client tool_use/tool_result.

Tests (tests/llm_translation/test_prompt_factory.py):
- update test_convert_to_anthropic_tool_invoke_server_tool ->
  test_convert_to_anthropic_tool_invoke_server_tool_without_result_is_dropped
- add test_anthropic_messages_pt_generic_client_drops_orphan_server_tool

Follow-up to BerriAI#17746 / BerriAI#17798; addresses the generic-client (no
provider_specific_fields) case of BerriAI#17737.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* test(anthropic): cover the srvtoolu_ round-trip fix in the test_litellm unit suite

The regression tests added in tests/llm_translation/test_prompt_factory.py aren't
run by the coverage CI job (it runs tests/test_litellm), so the new factory.py
branches showed as uncovered (codecov patch coverage). Add equivalent focused
tests in the unit suite so both new branches are exercised there:
- convert_to_anthropic_tool_invoke drops a srvtoolu_ server_tool_use when no
  matching *_tool_result is available.
- anthropic_messages_pt drops the orphaned srvtoolu_ tool message a generic
  OpenAI client replays.

Refs BerriAI#17737

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* test(anthropic): cover the server_tool_use + result valid-pair path in unit suite

Covers the remaining patch-coverage lines codecov flagged: convert_to_anthropic_tool_invoke
emitting server_tool_use followed by its web_search_tool_result when the matching
result is present (the litellm-SDK round-trip path). Refs BerriAI#17737

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* style(anthropic): flatten srvtoolu_ tool-message guard to a negated if

Addresses the Greptile style nit: replace the if-pass/else with a single negated
`if not (...)` guard around the tool_result append. Behavior unchanged. Refs BerriAI#17737

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

* fix(proxy): require premium only when enabling premium metadata fields (BerriAI#30285) (BerriAI#30506)

Co-authored-by: Sameer Kankute <[email protected]>

* fix(perplexity): stop double-billing reasoning tokens in manual cost fallback (BerriAI#30488)

* fix(perplexity): stop double-billing reasoning tokens in manual cost fallback

When perplexity_cost_per_token cannot use the API-provided usage.cost.total_cost short-circuit and falls back to manual calculation, it multiplies the full usage.completion_tokens by output_cost_per_token and then adds reasoning_tokens * output_cost_per_reasoning_token on top. Per the OpenAI/Perplexity usage convention codified for the central path in PR BerriAI#18607, completion_tokens already INCLUDES reasoning_tokens, so the manual fallback double-bills reasoning at both the output and reasoning rate.

Concrete impact on perplexity/sonar-deep-research (input 2e-6, output 8e-6, reasoning 3e-6): for the exact usage shape exercised by the live response fixture in tests/llm_translation/test_perplexity_reasoning.py (prompt_tokens=9, completion_tokens=20, reasoning_tokens=15) the current code charges 0.000223 vs the convention-correct 0.000103, a 2.165x overcharge. The bug is reachable whenever Perplexity omits the cost object (streaming chunks, fixture-driven paths, older API versions).

Subtracts reasoning_tokens (clamped at zero) from completion_tokens before applying the output rate, mirroring how dashscope/cost_calculator.py and the central generic_cost_per_token already handle it. Preserves the existing fallback behaviour when output_cost_per_reasoning_token is unset (all completion_tokens stay at the output rate).

Existing tests in tests/test_litellm/llms/perplexity/test_perplexity_cost_calculator.py asserted the buggy math and are updated to the convention-correct math. Adds a focused regression test using the exact usage shape from the live response fixture so this class of bug cannot be silently reintroduced.

* style(perplexity): drop redundant type annotation on else branch to satisfy mypy

mypy [no-redef] flagged 'completion_cost' as declared in both if and else arms; keeping the annotation only on the first declaration matches existing patterns in this file.

* fix(perplexity): update integration test expected costs for non-double-billed math

Three tests in test_perplexity_integration.py asserted the old buggy expectation
that reasoning_tokens are billed in addition to the full completion_tokens
count. After the fix in cost_per_token, reasoning_tokens are billed at the
reasoning rate and the remaining (completion_tokens - reasoning_tokens) at the
standard output rate, matching OpenAI/Perplexity convention (PR BerriAI#18607).

Updates: test_end_to_end_cost_calculation_with_transformation,
test_main_cost_calculator_integration, test_high_volume_cost_calculation.
The high-volume sanity threshold drops to 0.25 to reflect the corrected total.

* fix(ui): use dynamic proxy base URL in MCP usage examples (BerriAI#30487)

Replace hardcoded http://localhost:4000 with getProxyBaseUrl() in the
MCP server usage example and copy-to-clipboard snippet so the generated
configuration works for non-local deployments.

Fixes BerriAI#30466

* feat: add missing UK PII entity types to Presidio guardrail (BerriAI#30537)

* feat: add missing UK PII entity types to Presidio guardrail

Add UK_PASSPORT, UK_POSTCODE, and UK_VEHICLE_REGISTRATION to PiiEntityType enum and PII_ENTITY_CATEGORIES_MAP. These entity types are supported by Microsoft Presidio but were missing from litellm's type definitions, preventing users from configuring UK-specific PII detection.

* test: remove fragile hardcoded entity count test

Remove test_uk_category_entity_count which hardcodes len() == 5. The test_uk_entities_match_presidio_recognizers test already verifies exact set equality, making the count test redundant and fragile to future Presidio additions.

* style: apply Black formatting to match CI requirements

* fix: route volcengine (Doubao) tiered-pricing models to the tiered cost handler (BerriAI#30357)

Volcengine (Doubao) models define `tiered_pricing` but no flat per-token cost, so cost_per_token fell through to generic_cost_per_token (which only reads flat costs) and tracked them at $0

Route custom_llm_provider == "volcengine" to the shared tiered-pricing handler in litellm/llms/dashscope/cost_calculator.py, which already computes graduated tier costs. Make that handler provider-agnostic by adding a custom_llm_provider argument (default "dashscope" preserves existing behavior) so get_model_info resolves the correct model map entry

Fixes BerriAI#30346

* feat(mcp): make MCP gateway name and description configurable via env vars (BerriAI#30473)

* feat(mcp): make MCP gateway name and description configurable via env vars

* Rename function _restore_env to _apply_env

* docs(mcp): document import-time capture of env-backed identity constants

Address Greptile review feedback: clarify that LITELLM_MCP_SERVER_NAME and
LITELLM_MCP_SERVER_DESCRIPTION are read once at import and require a module
reload to observe env changes after import.

Generated with AI assistance

Co-Authored-By: Claude <[email protected]>

---------

Co-authored-by: Yevhen Luhovtsov <[email protected]>
Co-authored-by: Claude <[email protected]>

* fix(mcp): preserve native tools in semantic filter hook (BerriAI#26650)

* fix(mcp): preserve native tools in semantic filter hook

The SemanticToolFilterHook.async_pre_call_hook passed ALL tools (MCP +
native) to filter_tools(), which only knows MCP-registered tool names.
Native tools silently failed the name match in _get_tools_by_names()
and were dropped from the request.

Fix: partition tools into native and MCP-registered before filtering.
Run the semantic filter only on MCP tools, then merge native tools
back unconditionally.

Changes:
- Robust _is_mcp_tool() using shape-based detection for OpenAI-format
  dicts, safe regardless of future _extract_tool_info changes
- Single-pass partition loop (no double _is_mcp_tool calls)
- Preserve native tools in MCP expansion path (mixed requests)
- Track MCP expansion to prevent expanded tools bypassing filtering
- filter_stats reports MCP-only counts for accurate metrics
- Extracted _emit_filter_metadata() helper
- Skip spurious filter headers for all-native tool requests

Closes BerriAI#26212

* remove stale docstring note referencing tools_expanded_from_mcp

* fix: handle Responses API name collision and preserve tool ordering

- Classify Responses API tools ({type: 'function', name: '...'}) as
  native to prevent name collisions with MCP canonical names
- Preserve original request tool ordering using id()-based merge
  instead of naive native+mcp concatenation
- Add 2 regression tests: name collision and ordering preservation

* style: apply black formatting

* fix(mcp): harden semantic filter — preserve all native tool formats, safe metadata access, graceful expansion failure, name-based merge

* lint: suppress PLR0915 on async_pre_call_hook (matches codebase convention)

* ci: retrigger checks after rebase onto litellm_internal_staging

* feat(fireworks): sync Fireworks AI model registry with current platform catalog (BerriAI#30616)

Adds 12 new Fireworks serverless models and updates 3 existing entries in
model_prices_and_context_window.json and its bundled backup to match the
current Fireworks platform model list. New direct models: glm-5p2,
qwen3p7-plus, minimax-m3, minimax-m2p7, kimi-k2p7-code, kimi-k2p6,
deepseek-v4-pro, deepseek-v4-flash. New router endpoints: glm-5p1-fast,
kimi-k2p6-fast, kimi-k2p7-code-fast. Updated: glm-5p1, gpt-oss-120b, and
gpt-oss-20b now carry correct output token caps, cache-read pricing, and
explicit capability flags

max_tokens is set equal to max_output_tokens (not the full context window)
for models whose generation cap is below their context window. This avoids
the shared input+output budget path in get_modified_max_tokens, which would
otherwise let callers request output sizes the model cannot produce. The
same fix corrects the pre-existing glm-5p1, gpt-oss-120b, and gpt-oss-20b
entries that had max_tokens equal to the full context window

Short-form aliases (fireworks_ai/<model>) are added for every direct
accounts/fireworks/models/ entry so cost attribution works for callers
using bare model names. Router endpoints get short-form aliases too, and
transform_request now routes bare names ending in -fast to the
accounts/fireworks/routers/ path instead of defaulting every bare name to
models/. This keeps the kimi-k2p6-fast router from being misrouted to the
nonexistent models/kimi-k2p6-fast endpoint

kimi-k2p6-turbo is intentionally excluded; kimi-k2p6-fast is its
replacement. Context windows for deepseek-v4 and kimi models use the
power-of-two values (1048576 and 262144) published on the Fireworks model
pages, matching the convention already used by existing entries

Two regression tests in test_utils.py assert the exact per-token costs,
token limits, capability flags, and short-form-to-long-form equality for
all 15 models against both the main and backup cost maps. Two routing
tests in test_fireworks_ai_chat_transformation.py verify bare -fast names
route to routers/ and bare direct-model names route to models/

* fix(bedrock): handle role:"system" inside the messages array on /v1/messages (BerriAI#29698) (BerriAI#30443)

* feat(anthropic): hoist leading in-array system to top-level (helper)

* test(anthropic): cover _system_content_to_blocks edge cases; deepcopy cache_control

* test(anthropic): mid-conversation system normalization cases

* feat: add supports_mid_conversation_system flag to Claude Opus 4.8

Add supports_mid_conversation_system: true to all 9 claude-opus-4-8 cost-map
entries (Anthropic-native, Bedrock, Vertex, Azure AI) in both the root cost
map and the bundled package backup, since the runtime helper and tests read
the backup in local/offline mode.

Pin the mid-system passthrough regression test to the local cost map via the
existing local_model_cost_map fixture so it reads the branch-local flag rather
than the network-fetched main copy.

* fix(bedrock): normalize in-array system in /v1/messages handler (BerriAI#29698)

Wire normalize_system_messages_for_anthropic into anthropic_messages_handler
so all Bedrock /v1/messages paths (Invoke / Mantle / ClaudePlatform /
Converse-bridge) hoist leading in-array system entries (and demote
mid-conversation ones on models lacking supports_mid_conversation_system) into
the top-level system field. The normalized messages/system are written back
into the local_vars snapshot the base_llm branch reads from, otherwise the
Invoke/Mantle fix would silently no-op.

Also fix the helper to resolve supports_mid_conversation_system through the
prefix-aware AnthropicModelInfo._supports_model_capability resolver. The raw
_supports_factory could not see the flag once get_llm_provider left the
invoke/ prefix on the model id, which would have wrongly demoted
mid-conversation system on a Bedrock invoke opus-4-8 path.

* fix(bedrock): resolve mid-conversation-system flag through mantle/invoke/converse route prefixes; drop unused param

* fix(types): widen system param to Union[str, List] for hoisted system blocks

* refactor(bedrock): drop dead local_vars messages writeback

* fix(bedrock/converse): translate in-array system in anthropic->openai adapter (BerriAI#29698)

* fix(bedrock/converse): preserve cache_control on in-array system; test drop-empty

* fix(bedrock/converse): rename colliding local to satisfy mypy; test handler system-merge branches

* fix(types): register supports_mid_conversation_system in model-info schema

The cost-map JSON-schema validation test (test_aaamodel_prices_and_context_window_json_is_valid)
rejects unknown properties, so adding supports_mid_conversation_system to the opus-4-8
cost-map entries failed CI with 'Additional properties are not allowed'. Register the flag
in the INTENDED_SCHEMA allow-list and in the ProviderSpecificModelInfo TypedDict so it is a
typed, first-class capability flag alongside its peers (supports_output_config, etc.).

---------

Co-authored-by: Sameer Kankute <[email protected]>

* fix(bedrock/agentcore): optionally forward multimodal content blocks in InvokeAgentRuntime payload (BerriAI#28885)

* fix(bedrock/agentcore): optionally forward multimodal content blocks in InvokeAgentRuntime payload

By default the agentcore provider flattens the last message to a text-only
{"prompt": "..."} payload via convert_content_list_to_str, silently dropping
OpenAI multimodal blocks (image_url, file, input_audio, ...).

This adds an opt-in `forward_multimodal_content` litellm param. When truthy and
the last message's content is a list containing a non-text block, the original
OpenAI content list is forwarded verbatim under a new "content" field so an
attachment-aware AgentCore agent can read it. Default off keeps the payload
byte-identical to the legacy {"prompt": "..."} shape — existing agents are
unaffected.

The flag is read from optional_params (where other AgentCore params land) with a
litellm_params fallback, and accepts a bool or a config/env string ('true', '1', ...).

AgentCore Runtime is schemaless on the agent side — the agent's @app.entrypoint
parses arbitrary JSON up to 100 MB (per
https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-invoke-agent.html),
so this is a purely upstream change; no AgentCore-side schema is asserted.

* fix(bedrock/agentcore): shallow-copy forwarded multimodal content list

Address review feedback (Sameerlite): payload["content"] = last_content
aliased the caller's mutable messages[-1]["content"] list. Harmless today
because the payload is JSON-serialized immediately, but a latent footgun if
a future caller mutates the returned payload before serialization. Forward
list(last_content) so the payload owns its own list. Block dicts stay shared
on purpose — a deep copy would clone potentially large base64 media on the
request hot path, and the flagged risk was the shared list, not the blocks.

Update the passthrough tests to assert equality + distinct identity, and add
a regression test that mutating the payload list can't leak back into the
original message content.

* Revert "fix(mcp): preserve native tools in semantic filter hook (BerriAI#26650)"

This reverts commit 438c825.

* Revert "feat(guardrails): integrate Repelloai Argus guardrail (BerriAI#30465)"

This reverts commit 54da785.

* Revert "feat(dashscope): add Responses API support (BerriAI#30286)"

This reverts commit 6766256.

* Revert "fix(bedrock): handle role:"system" inside the messages array on /v1/messages (BerriAI#29698) (BerriAI#30443)"

This reverts commit b8a8083.

* Revert "fix(anthropic): drop orphaned server_tool_use on multi-turn replay from generic OpenAI clients (BerriAI#30486)"

This reverts commit 6e9c0b0.

* Revert "fix: route volcengine (Doubao) tiered-pricing models to the tiered cost handler (BerriAI#30357)"

This reverts commit 172e302.

* Revert "feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery (BerriAI#30273)"

This reverts commit 4e31885.

* fix: pass key_limit=None in team_member_update and patch model_cost in pricing test

team_member_update called team_info without key_limit, so the fastapi.Query
default object (not None) was passed through to get_data, which failed when
serializing it. Pass key_limit=None explicitly to avoid this.

test_get_model_info_costs patched litellm.model_cost from the local backup so
the assertion holds before the PR is merged and the remote main URL is updated.

* fix(security): validate resolved model in /realtime/client_secrets for non-transcription sessions (BerriAI#30710)

Omitting both model and session.model caused the endpoint to default to
gpt-4o-realtime-preview without running can_key_call_resolved_model, so
any key could access that model regardless of its allowed-model list.

The transcription path already called can_key_call_resolved_model; this
adds the same call for the realtime path before returning.

* fix(lint): fix F821 undefined model_info and F841 unused metadata in create_model_info_response

* fix: black formatting and stub get_model_group_info in third team translation test

* fix: reformat utils.py with black 26.3.1 to match CI

* fix: replace Optional[X] with X | None to satisfy UP045 ruff strict gate

---------

Co-authored-by: Habon Laszlo <[email protected]>
Co-authored-by: habonlaci <[email protected]>
Co-authored-by: Armaan Sandhu <[email protected]>
Co-authored-by: santino18727-debug <[email protected]>
Co-authored-by: Eric (GabiDevFamily) <[email protected]>
Co-authored-by: Nitish Agarwal <[email protected]>
Co-authored-by: jho1-godaddy <[email protected]>
Co-authored-by: 安妮的心动录 <[email protected]>
Co-authored-by: Harshith Gujjeti <[email protected]>
Co-authored-by: Tomoya Tabuchi <[email protected]>
Co-authored-by: Vedant Agarwal <[email protected]>
Co-authored-by: Prathamesh Jadhav <[email protected]>
Co-authored-by: songkuan-zheng <[email protected]>
Co-authored-by: Kropiunig <[email protected]>
Co-authored-by: Lavish Bansal <[email protected]>
Co-authored-by: Shane Emmons <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Co-authored-by: Anuj ojha <[email protected]>
Co-authored-by: Nahrin <[email protected]>
Co-authored-by: Nbouyaa <[email protected]>
Co-authored-by: Vineeth Sai <[email protected]>
Co-authored-by: Eugene Lugovtsov <[email protected]>
Co-authored-by: Yevhen Luhovtsov <[email protected]>
Co-authored-by: Ayush Shekhar <[email protected]>
Co-authored-by: Ahmad Shahzad <[email protected]>
Co-authored-by: Kent <[email protected]>
Co-authored-by: Jón Levy <[email protected]>

* fix(proxy): bump health-check max_tokens default to 16 for GPT-5 compatibility (BerriAI#30708)

OpenAI GPT-5 models require max_completion_tokens >= 16.
Health checks were using 5 (proxy/health_check.py) and 10
(health_check_helpers.py), causing failures on GPT-5 models.

Fixes BerriAI#23836

* fix: increase health check max_tokens from 5 to 16 (BerriAI#23836) (BerriAI#26610)

GPT-5 models enforce a minimum of 16 for max_output_tokens. The current
default of 5 still causes health checks to fail for these models. Bump
the non-wildcard default to 16 — the smallest value that satisfies all
known provider minimums while keeping health checks lightweight.

Also tightens the wildcard test assertion from a weak disjunctive check
to strict key-absence.

Co-authored-by: Sameer Kankute <[email protected]>

* fix: ensure checks show gemini-3-flash-preview supports responseJsonS… (BerriAI#30696)

* fix: ensure checks show gemini-3-flash-preview supports responseJsonSchema.

* fix: remove async keyword from test.

* fix: make Bedrock Mantle Responses routing data-driven per model (BerriAI#30700)

* Make Bedrock Mantle Responses routing data-driven per model

Route Bedrock Mantle models to the native Responses API based on each
model's price-map capability signal instead of a hardcoded model-name
heuristic, and derive the OpenAI-compatible base path segment per model.

Responses dispatch now selects the native config when the model advertises
responses support (/v1/responses in supported_endpoints, or mode=responses),
both overridable via register_model and proxy model_info. This enables
native Responses for gpt-oss-120b/20b and the gemma-4 family while keeping
chat-only models (gpt-oss safeguard, nvidia, mistral, ...) on the existing
chat-completions emulation. Capability is per-model, so gpt-oss-120b routes
natively while gpt-oss-safeguard-120b does not despite sharing the gpt-oss
substring.

The wire path is a separate concern, driven by the existing
use_openai_responses_path flag rather than a model-name match: gpt-5.x and
gemma-4-* on /openai/v1, everything else (incl. gpt-oss) on /v1. The chat
config now derives its base from the same flag, fixing gemma-4
chat-completions requests that previously went to /v1 instead of /openai/v1.

Cost maps: add supported_endpoints to the gpt-oss entries (responses for the
non-safeguard variants, chat-only for safeguard) and supported_endpoints +
use_openai_responses_path to all three gemma-4 entries.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* Address review: move capability helper into bedrock_mantle package

Move the Responses capability check out of utils.py into
litellm/llms/bedrock_mantle/common_utils.py as mantle_supports_responses,
alongside its companion wire-path helper mantle_base_segment. Both are now
pure functions of (model, model_cost): the price-map mode/supported_endpoints
read replaces the get_model_info call, so the rules are unit-testable without
patching global state and the Bedrock Mantle package is self-contained.

Use str | None instead of Optional[str] on the new signatures to satisfy the
ruff UP045 strict-rule gate. Add direct unit tests for both helpers.

Fix test_register_model_restore_undoes_existing_key_overwrite: gpt-oss-120b
now legitimately supports Responses, so it can no longer be the
"None after restore" vehicle; use the chat-only safeguard variant, which
isolates the register/restore effect from the model's own capability.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Co-authored-by: Sameer Kankute <[email protected]>

* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (BerriAI#30366)

* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup

LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.

Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.

Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.

Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.

Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.

* fix: resolve CI failures and proxy DB URL typing issue

* fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (BerriAI#30653)

The tiered cost calculator resolved a tier's per-token cost with
`tier.get(cost_key) or tier.get(fallback_cost_key, 0)`. Because `or`
short-circuits on any falsy value, a tier that legitimately prices a
component at 0.0 (e.g. a free-cache-read tier with
cache_read_input_token_cost: 0.0, or a free-reasoning tier) is treated
as missing and silently billed at the full fallback rate
(input_cost_per_token / output_cost_per_token).

The flat-pricing path in the same module already handles this correctly
with an `is None` guard. Resolve tier costs through a small helper that
mirrors it, so 0.0 is honored at both the in-range and overflow sites.

No shipped model currently has a 0.0 tier cost, so this is a latent
defect; the fix makes the tiered path consistent with the flat path and
prevents over-charging the first time such a tier appears. Adds unit
tests covering the in-range and overflow paths, and drops an unused
import flagged by ruff in the touched test file.

* feat(proxy): show session-aggregate cost and duration in request logs (BerriAI#25708) (BerriAI#30507)

* fix(anthropic): don't leak tool 'type' into OpenAI function parameters schema (BerriAI#30618)

In the messages->chat/completions bridge, translate_anthropic_tools_to_openai
merged every non-mapped tool key into the function parameters dict. The
Anthropic tool 'type' (e.g. 'custom') thus overwrote parameters.type ('object'
-> 'custom'), and providers reject it ('custom' is not a valid JSON-Schema type).
Exclude 'type' from the passthrough. Fixes BerriAI#30557.

* fix(proxy): stop IAM-refresh engine restart from cascading reconnects (BerriAI#29176) (BerriAI#30183)

An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the
running query-engine and spawns a new one. That planned kill was
indistinguishable from a crash, and three reconnect paths used two
uncoordinated locks, so a single refresh triggered a cascade of engine
kill/respawn cycles:

  1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old
     engine, spawn new one.
  2. The engine-death watcher sees that kill, assumes a crash, and calls
     `attempt_db_reconnect(force=True)` (a different lock,
     `_db_reconnect_lock`) -> recreate again -> kills the fresh engine.
  3. In-flight queries failing during the swap are classified as transport
     errors and trigger their own `attempt_db_reconnect` -> recreate again.

Fix coordinates planned restarts across the wrapper and the watcher:

  - PrismaWrapper records the old engine PID in `_expected_engine_deaths`
    before killing it; all four watcher death-detectors (waitpid thread,
    pidfd, already-dead probe, os.kill poll) consume that PID and skip the
    reconnect instead of treating it as a crash.
  - `recreate_prisma_client` now serializes through `_reconnection_lock` and
    bumps a monotonic `_engine_generation`. Callers pass `expected_generation`
    as an optimistic-lock token, so racing/cascading recreates collapse into a
    single restart (losers no-op). This closes the two-lock gap.
  - The direct reconnect path probes the writer with SELECT 1 before
    recreating; a healthy connection (e.g. engine already replaced by a
    refresh) skips the recreate entirely.
  - `_safe_refresh_token` coalesces: it skips when the current token still has
    more than the refresh buffer of runway, so stacked triggers (proactive
    loop + __getattr__ fallback) don't each restart the engine. An
    `on_engine_replaced` hook re-arms the watcher on the new PID.

RoutingPrismaWrapper forwards `expected_generation` and skips recreating the
reader when the writer recreate was skipped.

* feat(bedrock): support file content retrieval for batch output files (BerriAI#30595)

Implements transform_file_content_request and transform_file_content_response
in BedrockFilesConfig so GET /v1/files/{id}/content works for Bedrock batch
files. The request transform resolves the file id (direct s3:// URI or base64
unified id) to its S3 object, validates bucket and key prefix against the
server-configured bucket, and SigV4-signs an S3 GetObject using the same
credential and region resolution as the existing upload path. The credential
and region params are validated into a typed model at the boundary, so the only
untyped values left are the botocore signing primitives.

Also fixes the proxy managed-files path: CredentialLiteLLMParams now carries
s3_bucket_name (previously dropped when building deployment credentials) and
the managed-files hook passes the deployment credential snapshot when routing
afile_content, so unified-id content retrieval works with per-model bucket
config instead of only the AWS_S3_BUCKET_NAME env var.

Preserves managed-file access control: the proxy file-content endpoint now
rejects raw cloud-storage ids (s3://, gs://), which would otherwise skip the
owner/team check that only runs for unified ids and let a caller read another
tenant's batch output by its object key. Managed outputs are reachable only
through their unified file id. The afile_content "not found" error now reports
the caller's unified id rather than the resolved internal S3 URI.

Fixes BerriAI#16186, BerriAI#15563

* fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (BerriAI#30646)

* fix(oci): map Cohere tool array/object params to lowercase builtins

OCI's Cohere backend returns HTTP 500 on a tool parameter typed as a bare
"List", which is what OCI_JSON_TO_PYTHON_TYPES produced for JSON-schema
arrays. MLflow {{trace}} judges trip this: their tools (get_root_span,
get_span) take an attributes_to_fetch array. The lowercase builtins list/dict
are accepted; only the bare "List" 500s ("Dict" happens to be tolerated, but
both are lowercased for consistency).

Verified live against us-chicago-1 (cohere.command-a-03-2025 and
command-latest). Adds a unit regression on the transformed parameterDefinitions
plus a gated integration test exercising an array-param tool end to end.

* fix(oci): make Cohere agentic tool-calling continuation work

Two bugs broke the OCI Cohere tool-calling loop that MLflow {{trace}} judges
drive once a tool has been executed and its result is fed back.

Request side: litellm pulled the last user message into the top-level `message`
and emitted the tool result as a TOOL entry in chatHistory. OCI rejects that
("cannot specify message if the last entry in chat history contains tool
results"), and an empty message alone is rejected too ("message must be at least
1 token long or tool results must be specified"). OCI carries the current turn's
results in a dedicated top-level `toolResults` field. The Cohere transform now
sends an empty message, keeps the user turn in chatHistory, and puts the results
in `toolResults`, matching the langchain-oracle reference. Tool results are no
longer represented as chatHistory entries.

Response side: tool-grounded answers come back with citations carrying
`documentIds` (camelCase) and no `document_ids`, which made the required
`CohereCitation.document_ids` field fail validation and sink the whole response
parse. Those citations are never surfaced, so the field (and CohereSearchQuery's
generation_id) is now optional.

Verified live against us-chicago-1 (cohere.command-a-03-2025 and command-latest),
single and multi-round tool loops. Adds unit regressions on the transformed
request shape and on citation parsing, plus gated integration tests for the
continuation.

* feat: integrate Repelloai Argus guardrail (BerriAI#30673)

* feat(guardrails): add RepelloAI Argus guardrail integration (#1)

* feat(guardrails): add RepelloAI Argus guardrail integration

Add a new guardrail hook backed by RepelloAI Argus, with dashboard-managed
asset policies enforced via an asset_id and X-API-Key auth.

* fix(guardrails): harden RepelloAI Argus guardrail

- scan streaming responses on output (was bypassing the guardrail)
- log blocked verdicts as guardrail_intervened instead of success
- treat auth/config errors (401/403/404/422) as misconfiguration that
  always blocks, not a fail-open-able unreachable error
- default unreachable_fallback to fail_closed and read it directly;
  block on unknown/malformed verdicts so an API change can't silently
  disable enforcement
- type unreachable_fallback as a Literal, drop the duplicate config model,
  expose unreachable_fallback in the config schema, and stop leaking the
  raw provider response / exception strings to the client

* fix(guardrails): address RepelloAI Argus review feedback

- support ARGUS_API_KEY (with REPELLOAI_API_KEY fallback)
- make asset_id required in the config model
- normalize unreachable_fallback so only fail_open opens; block on 400 misconfig
- correct the shared unreachable_fallback field description

* docs(guardrails): add RepelloAI Argus docs page and dashboard listing

- add docs page covering config, env vars, modes, verdicts, failure semantics
- list RepelloAI Argus in the Guardrail Garden with provider/logo mappings
- add a regression test for the provider logo and display-name resolution

* fix(guardrails): keep RepelloAI asset_id optional in config model

A required asset_id leaked onto the shared LitellmParams (which inherits
RepelloAIGuardrailConfigModel), breaking validation for every other
guardrail. Keep it optional like sibling models; the guardrail __init__
still raises when asset_id is missing, which is the real enforcement.

* Add comment for last user turn scanning

* feat(guardrails): harden repelloai scanning

* feat(guardrails): expand repelloai scanning to include tool definitions

Add extraction of tool definitions and tool call arguments to the RepelloAI
guardrail scanning. Improves detection coverage by including function schemas
and parameters in the prompt sent to the guardrail service. Also captures
detailed error responses in logs and adds guardrail header to streaming responses.

* refactor(guardrails): fix and harden repelloai schema text extraction

- Fix duplicate text in _iter_schema_text: previously all dict values were
  re-queued onto the stack even after scalar/list keys were already extracted
  explicitly, causing names/descriptions to appear twice in the scanned prompt
- Extract schema key frozensets to module-level constants so they are not
  reconstructed on every call
- Change _iter_schema_text from @classmethod to @staticmethod (cls unused)
- Narrow _call_analyze stage param from str to Literal["prompt", "response"]
- Add HttpxResponse type annotation to _raise_for_config_error
- Add LLMResponseTypes annotation to async_post_call_success_hook response param

* fix(guardrails): resolve pyright type errors in repelloai guardrail

- Narrow async_handler.post return from Response|None to Response with
  explicit None guard before calling raise_for_status/json
- Fix list comprehension returning str|None by switching to explicit loop
  with isinstance guard so pyright tracks the narrowing
- Cast model_dump() result to Dict since hasattr does not narrow object
  type in pyright

* fix(guardrails/repello): include Responses API instructions field in prompt scan

The /v1/responses top-level `instructions` field was not included in
_extract_prompt_text, allowing a caller to bypass guardrail policy checks
by putting blocked content in `instructions` while keeping `input` benign.

* feat: add api_key to config model and read prompt from data dict

* fix(guardrails/repello): plug input_text and tool-call response bypass gaps

Responses API input content parts with type 'input_text' were silently
dropped by build_inspection_messages (which only handles type='text'),
allowing callers to send blocked content via that path without triggering
the pre-call scan. Fix: add _extract_input_text_parts to RepelloAIGuardrail
and call it when walking the Responses API input messages.

Post-call scanning skipped responses whose choices contained only tool_calls
or function_call (message.content=None), letting models put blocked output in
function arguments undetected. Fix: _extract_chat_completion_text now calls
_extract_tool_call_args_from_message on each choice message.

Also replace typing.Dict/List with builtin dict/list to clear TID251 strict
ruff violations introduced by this file.

* fix(guardrails/repello): scan Responses API function_call output arguments

Output items with type 'function_call' in a /v1/responses response were
skipped by _extract_responses_api_text; only 'message' items were walked.
A model could return blocked content in function_call.arguments undetected.
Now extract arguments from function_call output items before scanning.

* refactor(guardrails/repello): clean up typing and remove lint-any workarounds

- Replace Optional[X]/Union[X,Y] with X|None/X|Y union syntax throughout
- Use dict[str, object] instead of bare dict in all signatures
- Remove **kwargs from __init__; declare guardrail_name, event_hook, default_on explicitly
- Replace getattr(litellm_params, ...) with direct attribute access now that LitellmParams inherits RepelloAIGuardrailConfigModel
- Add _event_hook_from_mode() to convert str|list[str]|Mode to typed GuardrailEventHooks
- Use TypeAdapter.validate_json() instead of response.json() + manual dict construction
- Add _is_object_dict/_is_object_list TypeGuard helpers to narrow object types without Any
- Remove cast() workarounds and typed intermediate variables that existed only for the now-removed lint-any CI check
- Drop _AddLiteLLMCallback Protocol; budget has sufficient slack for the one reportUnknownMemberType
- Fix GuardrailConfigModel missing type arg: GuardrailConfigModel[BaseModel]

* fix(guardrails/repello): suppress LIT007 on TypeGuard helpers and add streaming scan-skip warning

- Add guard-ok suppressions to _is_object_dict and _is_object_list to satisfy the LIT007 hard-zero budget gate
- Emit verbose_proxy_logger.warning when the streaming hook finds no inspectable text after assembly, matching observability of pre/post hooks

* refactor: modifications for lint check

* feat: add Pinstripes as an OpenAI-compatible provider (BerriAI#30567)

* feat: add Pinstripes as an OpenAI-compatible provider

Pinstripes (https://pinstripes.io) is an OpenAI-compatible inference
provider serving open-source models (GLM-4.5-Air, Qwen3, DeepSeek, etc.)
with per-token pricing and no subscriptions.

Changes:
- `litellm/llms/openai_like/providers.json`: register pinstripes with
  base_url, api_key_env, and max_completion_tokens→max_tokens mapping
- `litellm/types/utils.py`: add `PINSTRIPES = "pinstripes"` to LlmProviders
- `litellm/constants.py`: add to openai_compatible_providers and
  openai_compatible_endpoints lists
- `litellm/litellm_core_utils/get_llm_provider_logic.py`: auto-detect
  provider when api_base is "https://pinstripes.io/v1"
- `provider_endpoints_support.json`: document supported endpoints
- `tests/`: 7 unit tests covering provider registration, resolution,
  URL auto-detection, api_base override, and Router config

Usage:
    import litellm
    response = litellm.completion(
        model="pinstripes/ps/glm-4.5-air",
        messages=[{"role": "user", "content": "Hello"}],
        api_key=os.environ["PINSTRIPES_API_KEY"],
    )

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix(pinstripes): resolve Greptile P1 review comments

- Add api_base_env: PINSTRIPES_API_BASE to providers.json so env var override works
- Set responses: false in provider_endpoints_support.json — not actually wired up
- Remove docs/my-website/docs/providers/pinstripes.md — belongs in litellm-docs repo

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix(pinstripes): add api_base_env and correct responses capability

- Add api_base_env: PINSTRIPES_API_BASE to providers.json
- Set responses: false in provider_endpoints_support.json

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix(pinstripes): wire up Responses API — add supported_endpoints

Adds supported_endpoints: ["/v1/chat/completions", "/v1/responses"] so
JSONProviderRegistry.supports_responses_api returns true correctly,
matching what provider_endpoints_support.json advertises.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* feat(pinstripes): enable embeddings endpoint

Pinstripes serves nomic-embed-text-v1.5 and bge-m3 via /v1/embeddings.
Add /v1/embeddings to supported_endpoints and set embeddings: true.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix(pinstripes): use 4-space indentation in model_prices_and_context_window.json

Matches the file's existing convention. Flagged by Greptile review.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix(pinstripes): set a2a: false — A2A protocol not implemented

All comparable JSON-configured providers (tensormesh, parasail, empiriolabs,
libertai, neosantara) have a2a: false. Pinstripes does not implement the
Google A2A protocol, so this should be false to match.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

---------

Co-authored-by: inference_provider <[email protected]>
Co-authored-by: Claude Sonnet 4.6 <[email protected]>

* fix(rag): attach existing OpenAI file ids (BerriAI#30628)

* fix(rag): attach existing OpenAI file ids

* chore: use modern typing in rag ingest fix

* chore: retrigger ci

* fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (BerriAI#30341)

cache_control_injection_points was only consumed by the chat/completions
prompt-management hook; on the native Anthropic /v1/messages path it was
forwarded unused, so deployment-level cache injection was silently dropped
(cache_creation_input_tokens stayed 0 for Anthropic-native clients).

Add AnthropicCacheControlHook.apply_to_anthropic_messages_request to inject
cache_control at block level for system / tools / message locations (the only
forms /v1/messages accepts), wire it into the native anthropic_messages
handler, and pop the param so it does not leak upstream as an unknown field.
A {location: message, role: system} config is redirected to the top-level
system prompt so the same YAML works on both endpoints.

Injection respects Anthropic's 4-block cache_control limit shared across
system, tools, and messages: client-supplied markers count toward the cap and
are never overwritten, a slot is reserved per Bedrock tool_config point, and
injection stops once the budget is exhausted. Locations this path cannot
represent (tool_config) are forwarded downstream instead of being silently
consumed, mirroring get_chat_completion_prompt's remaining_points pass-through.

Built on litellm_internal_staging. Refs BerriAI#30293

* fix(proxy): release budget reservation when a request is cancelled mid-flight (BerriAI#30522)

* fix(proxy): release budget reservation on cancel when no chunk was delivered

The pre-call budget reservation increments the cross-pod spend counter by a
request's worst-case cost, then reconciles it on success (cost callback) or
error (failure hook). A client disconnect or timeout cancels the request and
surfaces as CancelledError / GeneratorExit, which neither path catches, so the
reservation leaks. Under a retry storm the leaked holds accumulate, pin the
counter above real spend, and return spurious 429 "Budget has been exceeded" to
keys whose spend is far below budget; the counter only recovers when its TTL
lapses, so the failure is intermittent and self-healing.

Release the reservation in async_streaming_data_generator (which the Anthropic
and Google SSE generators delegate to) on the (CancelledError, GeneratorExit)
path, alongside the existing max_parallel_requests release. release_budget_
reservation_on_cancel runs under asyncio.shield so it completes despite the
in-progress cancellation, is guarded by the reservation's finalized flag, and
swallows a failing release so it cannot replace the in-flight cancellation.

The refund is gated on whether a chunk reached the client. The flag is set
immediately before the yield, after the slow-path hook await: an async generator
suspends at the yield, so a GeneratorExit on disconnect after a delivered chunk
sees it True (keep the hold), while a cancellation during the slow-path await
leaves it False (refund, nothing sent). A non-streaming cancellation delivers
nothing and a completed non-streaming response is reconciled by the success
callback, so neither needs a release here.

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* fix(proxy): reconcile a cancelled reservation to input cost, not zero

A streaming request cancelled before the first chunk previously reconciled its
reservation to zero and finalized it. But by the time the generator is
consuming the response the provider call was already dispatched, so the input
tokens were billed even though no chunk reached the client, and the
success/failure cost callbacks are skipped on cancellation. Refunding to zero
let a caller send an expensive request and abort pre-token to dodge the input
charge.

Compute the request's input-token cost at reservation time and reconcile the
cancelled reservation to it instead of zero. The worst-case output portion of
the reservation is still released (so a legitimate mid-flight cancellation no
longer pins the counter and 429s the key), while the input the provider already
processed is charged.

---------

Co-authored-by: Bytechoreographer <[email protected]>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* fix(caching): encode object name in GCS cache GET path (BerriAI#30378)

GCS cache reads always missed when gcs_path was set. The GET methods
interpolated the object name directly into the URL path, while the GCS
JSON API requires it to be URL-encoded (a "/" must be sent as %2F).

With gcs_path configured the object name is "<prefix>/<sha256>", so the
raw slash produced a malformed object path and GCS returned 404. httpx
does not raise on 4xx, so the status_code == 200 check fell through and
get/async_get returned None, silently missing on every read. Without
gcs_path the key has no slash, which is why this went unnoticed.

Wrap the object name with urllib.parse.quote(..., safe="") in get_cache
and async_get_cache. Apply the same encoding to the name= query
parameter in set_cache and async_set_cache so the key written matches
the key read back.

Adds regression tests asserting the GET path and SET query are encoded
(%2F) when gcs_path is set, for both sync and async paths; these fail on
the unpatched code.

Fixes BerriAI#30377

* chore: add soniox stt-async-v5 model (BerriAI#30672)

* fix(proxy): include model group aliases in v1 model info (BerriAI#30626)

* Include model group aliases in v1 model info

* Fix model info alias implementation

* removed extra blank line

* chore: rerun CI

* fix(lint): remove redundant noqa directive in proxy_cli.py

* fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme

* Revert "fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme"

This reverts commit 52c7a07.

* Revert "fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (BerriAI#30341)"

This reverts commit c9e8a17.

* Revert "fix(proxy): stop IAM-refresh engine restart from cascading reconnects (BerriAI#29176) (BerriAI#30183)"

This reverts commit 85828da.

* fix(proxy): stop IAM-refresh engine restart from cascading reconnects (BerriAI#29176) (BerriAI#30183)

An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the
running query-engine and spawns a new one. That planned kill was
indistinguishable from a crash, and three reconnect paths used two
uncoordinated locks, so a single refresh triggered a cascade of engine
kill/respawn cycles:

  1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old
     engine, spawn new one.
  2. The engine-death watcher sees that kill, assumes a crash, and calls
     `attempt_db_reconnect(force=True)` (a different lock,
     `_db_reconnect_lock`) -> recreate again -> kills the fresh engine.
  3. In-flight queries failing during the swap are classified as transport
     errors and trigger their own `attempt_db_reconnect` -> recreate again.

Fix coordinates planned restarts across the wrapper and the watcher:

  - PrismaWrapper records the old engine PID in `_expected_engine_deaths`
    before killing it; all four watcher death-detectors (waitpid thread,
    pidfd, already-dead probe, os.kill poll) consume that PID and skip the
    reconnect instead of treating it as a crash.
  - `recreate_prisma_client` now serializes through `_reconnection_lock` and
    bumps a monotonic `_engine_generation`. Callers pass `expected_generation`
    as an optimistic-lock token, so racing/cascading recreates collapse into a
    single restart (losers no-op). This closes the two-lock gap.
  - The direct reconnect path probes the writer with SELECT 1 before
    recreating; a healthy connection (e.g. engine already replaced by a
    refresh) skips the recreate entirely.
  - `_safe_refresh_token` coalesces: it skips when the current token still has
    more than the refresh buffer of runway, so stacked triggers (proactive
    loop + __getattr__ fallback) don't each restart the engine. An
    `on_engine_replaced` hook re-arms the watcher on the new PID.

RoutingPrismaWrapper forwards `expected_generation` and skips recreating the
reader when the writer recreate was skipped.

* fix(lint): modernize type annotations in IAM-refresh prisma client files (UP006/UP045)

* Revert "feat(proxy): show session-aggregate cost and duration in request logs (BerriAI#25708) (BerriAI#30507)"

This reverts commit f530b22.

* Revert "fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (BerriAI#30653)"

This reverts commit 4f58bd0.

* Revert "fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (BerriAI#30646)"

This reverts commit 50f34e0.

* Revert "fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (BerriAI#30366)"

This reverts commit 0544eed.

* fix(bedrock_mantle): restore BedrockMantleAuthMixin and constants removed by routing rewrite

* fix(key management): restore exact /key/list user_id & key_alias matching by default (BerriAI#30593)

Before substring search was added (commit 33bd570), /key/list matched user_id
and key_alias exactly. That change made admin-authenticated calls substring-match
by default, breaking the prior contract: a caller passing an exact user_id as an
access filter (e.g. an integration scoping to one user with an admin key) then
received other users' keys -- user_id="alice" also returned "alice2",
"alice-test", etc. This is a cross-user key disclosure.

Make substring matching opt-in via a new admin-only substring_matching=true query
param; default to exact, restoring the prior behavior. The dashboard search box
(keyListCall) passes the flag so partial search still works. Non-admins remain
exact and scoped to their own keys.

Updates the proxy-behavior key_alias test to opt in and adds an exact-by-default
guard; adds list_keys unit coverage for the opt-in gate.

---------

Co-authored-by: perseus <[email protected]>
Co-authored-by: Hannah Smith <[email protected]>
Co-authored-by: Charlie Patterson <[email protected]>
Co-authored-by: Matthew Lapointe <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Co-authored-by: KRISH SONI <[email protected]>
Co-authored-by: Yash Raj Pandey <[email protected]>
Co-authored-by: Nitish Agarwal <[email protected]>
Co-authored-by: hcl <[email protected]>
Co-authored-by: tushar8408 <[email protected]>
Co-authored-by: AD Mohanraj <[email protected]>
Co-authored-by: Fede Kamelhar <[email protected]>
Co-authored-by: Lavish Bansal <[email protected]>
Co-authored-by: max-amos <[email protected]>
Co-authored-by: inference_provider <[email protected]>
Co-authored-by: NK <[email protected]>
Co-authored-by: 安妮的心动录 <[email protected]>
Co-authored-by: Rick <[email protected]>
Co-authored-by: Bytechoreographer <[email protected]>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Burak Ömür <[email protected]>
Co-authored-by: Dan Lemon <[email protected]>
Co-authored-by: Vanika Dangi <[email protected]>
Co-authored-by: Jay Gowdy <[email protected]>