* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (#29693)

* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens

* test: scope local cost map env var with monkeypatch to avoid test pollution

* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (#30764)

* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold

_mask_value did partial reveal by showing the first visible_prefix and last
visible_suffix characters, but for a value whose length was at or below
visible_prefix + visible_suffix (8 by default) it returned the value verbatim.
A value of exactly 8 chars fell through the length guard and computed
masked_length == 0, reconstructing the original string with no mask characters;
anything shorter hit the early return. Either way short credentials were emitted
in plaintext.

mask_dict routes real secrets through this path, so an 8-char-or-shorter redis
password, api key, or token could be written to logs and the UI unmasked. The
sibling helper mask_sensitive_keys already guards this case; _mask_value now does
the same by fully masking any value at or below the threshold.

* fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers

Fully masking short values is the right default for secret masking, but
CooldownCache reuses the masker purely to truncate exception messages to the
first 50 characters, and it relies on short messages being returned readable.
Masking those blanked out short exception text and broke its tests.

Add a mask_short_values flag (default True, secure) and have CooldownCache pass
False so it keeps the truncation behavior, while every secret-masking caller
still gets short values fully masked.

* fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview

MCPDebug uses the masker to preview auth tokens in debug headers and documents
that values of 10 chars or fewer are shown unchanged so token types stay
distinguishable. Pass mask_short_values=False so that diagnostic behavior is
preserved while secret maskers keep masking short values.

* fix(mcp_debug): mask short auth values in debug headers instead of echoing them

Earlier this masker opted out of short-value masking to keep a token preview, but
that echoes short authorization and token values verbatim in debug response
headers, which is the same leak this change is meant to close. Auth material
should never be emitted in full, so mask short values here too; the first/last
character preview still applies to longer tokens. Only CooldownCache keeps the
opt-out, since it truncates exception text rather than masking secrets.

* test(mcp_debug): assert masked short value preserves length

* refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (#30917)

Fireworks AI deprecated audio inference on 2026-06-10
(https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation).
Live API testing confirms the endpoint is already non-functional: a valid
Fireworks API key receives HTTP 401 "Unauthorized" from
api.fireworks.ai/inference/v1/audio/transcriptions for every request,
regardless of payload. The audio-prod.api.fireworks.ai host referenced in
the test suite returns 401 for every path; the entire host is decommissioned.

Remove the dead FireworksAIAudioTranscriptionConfig class and every
reference to it across the codebase:

- Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line
  config class that inherited from OpenAIWhisperAudioTranscriptionConfig)
- Remove the Fireworks branch from
  ProviderConfigManager.get_provider_audio_transcription_config() in
  litellm/utils.py; update the stale comment in
  get_optional_params_transcription that referenced fireworks ai
- Remove the FireworksAIAudioTranscriptionConfig entries from
  LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in
  litellm/_lazy_imports_registry.py
- Remove the TYPE_CHECKING re-export in litellm/__init__.py
- Remove the transcription branch in the fireworks_ai case of
  get_supported_openai_params() in
  litellm/litellm_core_utils/get_supported_openai_params.py
- Remove the whisper-v3 and whisper-v3-turbo entries from
  model_prices_and_context_window.json and
  litellm/model_prices_and_context_window_backup.json (both had
  mode: audio_transcription and zero-cost pricing)
- Remove the TestFireworksAIAudioTranscription test class and its
  imports from tests/llm_translation/test_fireworks_ai_translation.py

No other provider is affected. The openai_compatible_providers list,
FireworksAIMixin, and the OpenAI Whisper transcription handler all stay
because they are shared with other Fireworks endpoints and other
providers. The provider_endpoints_support.json registry already had
audio_transcriptions set to false for fireworks_ai.

* feat: add darkbloom provider (#30876)

* feat: add darkbloom provider

* fix: document darkbloom provider endpoints

* fix: address darkbloom review feedback

* fix: update darkbloom tool metadata

* fix: fail fast for non-Postgres database URLs (#30883)

* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup

LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.

Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.

Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.

Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.

Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.

* fix: resolve CI failures and proxy DB URL typing issue

* fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging

* Validate DIRECT_URL alongside DATABASE_URL startup guards

* fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (#24608) (#30946)

* fix(bedrock): surface modeled HTTP status for mid-stream error events (#24608)

* test(bedrock): mid-stream server errors trigger streaming fallback (#24608)

* style(bedrock): black-format stream-error helper (#24608)

* fix(mcp): re-land native tool preservation with typed annotations (#30645)

* fix(mcp): preserve native tools in semantic filter hook with typed annotations

* fix(mcp): tighten _is_mcp_tool Chat Completions shape check

* fix(sambanova): return embeddings supported params instead of dropping them (#30937)

* fix(router): send fallback metadata when streaming (#30914)

When a streaming request triggers a fallback, there was previously no way to
know it happened. This commit addresses this in a few ways:

1. The response now correctly populates the fallback headers
    (`x-litellm-attempted-fallbacks`) so callers know a fallback happened.
2. The correct model ID is passed in the streaming chunks.
3. A streaming chunk with the fallback error can be optionally sent back
    to the client (opt-in) by passing `include_fallback_errors: true` in
    the request.

The format of the fallback errors while streaming is intentionally OpenAI
compatible to not break existing libraries that parse these events. It was
tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not
delieved unexpectedly to callers by default.

* fix(mistral): drop output-only reasoning fields from input messages (#30884)

LiteLLM attaches reasoning_content and thinking_blocks to assistant
responses. Replaying those assistant turns verbatim forwarded the fields
back to Mistral, whose input schema forbids unknown keys, so the whole
request failed with a 422 extra_forbidden and reasoning models became
unusable across multiple turns.

Strip both fields from assistant messages before the request is built, in
a spot that runs ahead of the image/file branch so it applies on every
path. Fixes #30835

Co-authored-by: Cursor <[email protected]>

* fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (#30652)

* fix(perplexity): bill search queries at the per-request price, not 1/1000

The fallback cost calculator divided search_context_cost_per_query by
1000, but that field stores the per-request price in USD: sonar is
{low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published
$5/$8/$12 per 1,000 requests expressed per request. The gemini cost
calculator reads the same field per request with no division (its
docstring calls it "the per-request cost").

The division understated search cost by 1000x on every Perplexity call
that falls back to manual calculation (i.e. when the API does not return
a pre-computed usage.cost). Use the value directly.

Update the tests that had encoded the /1000 factor in their expectations,
and drop an unused import flagged by ruff in the touched test file.

* test(perplexity): update integration test search-cost expectations to per-request

The integration tests still encoded the old /1000 search-cost factor, so
they failed once the fallback calculator was corrected to bill
search_context_cost_per_query per request. Update the four expected-cost
computations (and the high-volume dollar-value comments) to match.

* test(perplexity): drop unused mock imports flagged by ruff

* fix: include model_access_groups when expanding all-team-models in get_team_models (#30622)

* fix(fireworks_ai): return None for transcription in get_supported_openai_params

Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is
decommissioned. Without an explicit transcription branch, requests with
request_type='transcription' fell through to the else and returned
FireworksAIConfig chat-completion params. Return None instead to signal
the provider does not support transcription.

* fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting

Without an operator gate, any authenticated caller could set include_fallback_errors=True,
trigger a fallback, and read raw upstream exception messages from the
x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event.

Strip include_fallback_errors from request data in common_processing_pre_call_logic
when expose_fallback_errors_to_caller is not set, so the router never builds the
error list. Also gate _should_include_fallback_errors on the same setting as a
secondary check for the streaming SSE injection path.

* test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test

The operator gate added in e7ff3e1 means include_fallback_errors is only
honoured when general_settings.expose_fallback_errors_to_caller is True.
Set that flag via monkeypatch in the test that exercises the emit path.

* test(prompt_templates): make test_convert_url hermetic instead of hitting picsum.photos

test_convert_url called convert_url_to_base64 against a live picsum.photos
URL and asserted nothing, so it added no real signal and broke CI whenever
the host was unreachable (it was returning 522 and blocking this branch).
Replace the live call with a mocked HTTP client and assert the produced
base64 data URL, so the conversion path is exercised deterministically with
no network dependency. This suite runs under VCR, which is why a transport
level mock (respx) does not reliably intercept; mocking the client object
itself is robust regardless.

* fix(interactions): drop role from Interaction response to match Google spec

Google removed the output-only role field from the Interaction schema (it
now lives only on Turn), so the live OpenAPI compliance canary started
failing with 'role' not in spec. Reconcile our generated types by removing
role from Interaction, CreateModelInteractionParams, CreateAgentInteractionParams
and from the LiteLLM InteractionsAPIResponse/InteractionsAPIStreamingResponse,
stop stamping role=model in the responses-to-interactions transformation, and
update the compliance and integration tests accordingly. Turn.role is kept
since the spec still defines it.

* fix: align all-team-models sentinel access

* fix(router): forward include_fallback_errors through multi-hop fallbacks

run_async_fallback received include_fallback_errors as an explicit named
parameter, so it was bound out of **kwargs and never reached the nested
async_function_with_fallbacks call. Multi-hop fallback chains (a fallback
group that itself fails over) therefore stopped collecting fallback errors
beyond the first hop when a caller opted in. Re-inject the flag into kwargs
before the nested call so inner hops keep accumulating errors, which
add_fallback_headers_to_response already merges across levels.

---------

Co-authored-by: Srivatsa Kamballa <[email protected]>
Co-authored-by: Ahmad Shahzad <[email protected]>
Co-authored-by: Jeremy Chapeau <[email protected]>
Co-authored-by: KRISH SONI <[email protected]>
Co-authored-by: Kent <[email protected]>
Co-authored-by: Ayush Shekhar <[email protected]>
Co-authored-by: dav nguyxn <[email protected]>
Co-authored-by: Tal Marian <[email protected]>
Co-authored-by: Hemant K <[email protected]>
Co-authored-by: Cursor <[email protected]>
Co-authored-by: Yash Raj Pandey <[email protected]>
Co-authored-by: Zang Peiyu <[email protected]>
Co-authored-by: mateo-berri <[email protected]>

… config (#30624)

* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (#29693)

* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens

* test: scope local cost map env var with monkeypatch to avoid test pollution

* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (#30764)

* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold

_mask_value did partial reveal by showing the first visible_prefix and last
visible_suffix characters, but for a value whose length was at or below
visible_prefix + visible_suffix (8 by default) it returned the value verbatim.
A value of exactly 8 chars fell through the length guard and computed
masked_length == 0, reconstructing the original string with no mask characters;
anything shorter hit the early return. Either way short credentials were emitted
in plaintext.

mask_dict routes real secrets through this path, so an 8-char-or-shorter redis
password, api key, or token could be written to logs and the UI unmasked. The
sibling helper mask_sensitive_keys already guards this case; _mask_value now does
the same by fully masking any value at or below the threshold.

* fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers

Fully masking short values is the right default for secret masking, but
CooldownCache reuses the masker purely to truncate exception messages to the
first 50 characters, and it relies on short messages being returned readable.
Masking those blanked out short exception text and broke its tests.

Add a mask_short_values flag (default True, secure) and have CooldownCache pass
False so it keeps the truncation behavior, while every secret-masking caller
still gets short values fully masked.

* fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview

MCPDebug uses the masker to preview auth tokens in debug headers and documents
that values of 10 chars or fewer are shown unchanged so token types stay
distinguishable. Pass mask_short_values=False so that diagnostic behavior is
preserved while secret maskers keep masking short values.

* fix(mcp_debug): mask short auth values in debug headers instead of echoing them

Earlier this masker opted out of short-value masking to keep a token preview, but
that echoes short authorization and token values verbatim in debug response
headers, which is the same leak this change is meant to close. Auth material
should never be emitted in full, so mask short values here too; the first/last
character preview still applies to longer tokens. Only CooldownCache keeps the
opt-out, since it truncates exception text rather than masking secrets.

* test(mcp_debug): assert masked short value preserves length

* refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (#30917)

Fireworks AI deprecated audio inference on 2026-06-10
(https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation).
Live API testing confirms the endpoint is already non-functional: a valid
Fireworks API key receives HTTP 401 "Unauthorized" from
api.fireworks.ai/inference/v1/audio/transcriptions for every request,
regardless of payload. The audio-prod.api.fireworks.ai host referenced in
the test suite returns 401 for every path; the entire host is decommissioned.

Remove the dead FireworksAIAudioTranscriptionConfig class and every
reference to it across the codebase:

- Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line
  config class that inherited from OpenAIWhisperAudioTranscriptionConfig)
- Remove the Fireworks branch from
  ProviderConfigManager.get_provider_audio_transcription_config() in
  litellm/utils.py; update the stale comment in
  get_optional_params_transcription that referenced fireworks ai
- Remove the FireworksAIAudioTranscriptionConfig entries from
  LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in
  litellm/_lazy_imports_registry.py
- Remove the TYPE_CHECKING re-export in litellm/__init__.py
- Remove the transcription branch in the fireworks_ai case of
  get_supported_openai_params() in
  litellm/litellm_core_utils/get_supported_openai_params.py
- Remove the whisper-v3 and whisper-v3-turbo entries from
  model_prices_and_context_window.json and
  litellm/model_prices_and_context_window_backup.json (both had
  mode: audio_transcription and zero-cost pricing)
- Remove the TestFireworksAIAudioTranscription test class and its
  imports from tests/llm_translation/test_fireworks_ai_translation.py

No other provider is affected. The openai_compatible_providers list,
FireworksAIMixin, and the OpenAI Whisper transcription handler all stay
because they are shared with other Fireworks endpoints and other
providers. The provider_endpoints_support.json registry already had
audio_transcriptions set to false for fireworks_ai.

* feat: add darkbloom provider (#30876)

* feat: add darkbloom provider

* fix: document darkbloom provider endpoints

* fix: address darkbloom review feedback

* fix: update darkbloom tool metadata

* fix: fail fast for non-Postgres database URLs (#30883)

* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup

LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.

Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.

Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.

Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.

Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.

* fix: resolve CI failures and proxy DB URL typing issue

* fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging

* Validate DIRECT_URL alongside DATABASE_URL startup guards

* fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (#24608) (#30946)

* fix(bedrock): surface modeled HTTP status for mid-stream error events (#24608)

* test(bedrock): mid-stream server errors trigger streaming fallback (#24608)

* style(bedrock): black-format stream-error helper (#24608)

* fix(mcp): re-land native tool preservation with typed annotations (#30645)

* fix(mcp): preserve native tools in semantic filter hook with typed annotations

* fix(mcp): tighten _is_mcp_tool Chat Completions shape check

* fix(sambanova): return embeddings supported params instead of dropping them (#30937)

* fix(router): send fallback metadata when streaming (#30914)

When a streaming request triggers a fallback, there was previously no way to
know it happened. This commit addresses this in a few ways:

1. The response now correctly populates the fallback headers
    (`x-litellm-attempted-fallbacks`) so callers know a fallback happened.
2. The correct model ID is passed in the streaming chunks.
3. A streaming chunk with the fallback error can be optionally sent back
    to the client (opt-in) by passing `include_fallback_errors: true` in
    the request.

The format of the fallback errors while streaming is intentionally OpenAI
compatible to not break existing libraries that parse these events. It was
tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not
delieved unexpectedly to callers by default.

* fix(mistral): drop output-only reasoning fields from input messages (#30884)

LiteLLM attaches reasoning_content and thinking_blocks to assistant
responses. Replaying those assistant turns verbatim forwarded the fields
back to Mistral, whose input schema forbids unknown keys, so the whole
request failed with a 422 extra_forbidden and reasoning models became
unusable across multiple turns.

Strip both fields from assistant messages before the request is built, in
a spot that runs ahead of the image/file branch so it applies on every
path. Fixes #30835

Co-authored-by: Cursor <[email protected]>

* fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (#30652)

* fix(perplexity): bill search queries at the per-request price, not 1/1000

The fallback cost calculator divided search_context_cost_per_query by
1000, but that field stores the per-request price in USD: sonar is
{low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published
$5/$8/$12 per 1,000 requests expressed per request. The gemini cost
calculator reads the same field per request with no division (its
docstring calls it "the per-request cost").

The division understated search cost by 1000x on every Perplexity call
that falls back to manual calculation (i.e. when the API does not return
a pre-computed usage.cost). Use the value directly.

Update the tests that had encoded the /1000 factor in their expectations,
and drop an unused import flagged by ruff in the touched test file.

* test(perplexity): update integration test search-cost expectations to per-request

The integration tests still encoded the old /1000 search-cost factor, so
they failed once the fallback calculator was corrected to bill
search_context_cost_per_query per request. Update the four expected-cost
computations (and the high-volume dollar-value comments) to match.

* test(perplexity): drop unused mock imports flagged by ruff

* fix: include model_access_groups when expanding all-team-models in get_team_models (#30622)

* fix(fireworks_ai): return None for transcription in get_supported_openai_params

Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is
decommissioned. Without an explicit transcription branch, requests with
request_type='transcription' fell through to the else and returned
FireworksAIConfig chat-completion params. Return None instead to signal
the provider does not support transcription.

* fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting

Without an operator gate, any authenticated caller could set include_fallback_errors=True,
trigger a fallback, and read raw upstream exception messages from the
x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event.

Strip include_fallback_errors from request data in common_processing_pre_call_logic
when expose_fallback_errors_to_caller is not set, so the router never builds the
error list. Also gate _should_include_fallback_errors on the same setting as a
secondary check for the streaming SSE injection path.

* test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test

The operator gate added in e7ff3e1 means include_fallback_errors is only
honoured when general_settings.expose_fallback_errors_to_caller is True.
Set that flag via monkeypatch in the test that exercises the emit path.

* test(prompt_templates): make test_convert_url hermetic instead of hitting picsum.photos

test_convert_url called convert_url_to_base64 against a live picsum.photos
URL and asserted nothing, so it added no real signal and broke CI whenever
the host was unreachable (it was returning 522 and blocking this branch).
Replace the live call with a mocked HTTP client and assert the produced
base64 data URL, so the conversion path is exercised deterministically with
no network dependency. This suite runs under VCR, which is why a transport
level mock (respx) does not reliably intercept; mocking the client object
itself is robust regardless.

* fix(interactions): drop role from Interaction response to match Google spec

Google removed the output-only role field from the Interaction schema (it
now lives only on Turn), so the live OpenAPI compliance canary started
failing with 'role' not in spec. Reconcile our generated types by removing
role from Interaction, CreateModelInteractionParams, CreateAgentInteractionParams
and from the LiteLLM InteractionsAPIResponse/InteractionsAPIStreamingResponse,
stop stamping role=model in the responses-to-interactions transformation, and
update the compliance and integration tests accordingly. Turn.role is kept
since the spec still defines it.

* fix: align all-team-models sentinel access

* fix(router): forward include_fallback_errors through multi-hop fallbacks

run_async_fallback received include_fallback_errors as an explicit named
parameter, so it was bound out of **kwargs and never reached the nested
async_function_with_fallbacks call. Multi-hop fallback chains (a fallback
group that itself fails over) therefore stopped collecting fallback errors
beyond the first hop when a caller opted in. Re-inject the flag into kwargs
before the nested call so inner hops keep accumulating errors, which
add_fallback_headers_to_response already merges across levels.

* fix(router): stop fallback lookups from mutating the router fallbacks config

get_fallback_model_group resolved a bare-string fallback by popping it out
of the fallbacks list it was handed. That list is frequently the live
router.fallbacks config, so a single lookup permanently removed the entry and
the configured fallback stopped applying to later requests until restart. The
pop also ran inside enumerate(), shifting indices and skipping an adjacent
string fallback. Read the item instead of popping it, and add a regression
test that fails on the old mutating behavior

---------

Co-authored-by: Srivatsa Kamballa <[email protected]>
Co-authored-by: Ahmad Shahzad <[email protected]>
Co-authored-by: Jeremy Chapeau <[email protected]>
Co-authored-by: KRISH SONI <[email protected]>
Co-authored-by: Kent <[email protected]>
Co-authored-by: Ayush Shekhar <[email protected]>
Co-authored-by: dav nguyxn <[email protected]>
Co-authored-by: Tal Marian <[email protected]>
Co-authored-by: Hemant K <[email protected]>
Co-authored-by: Cursor <[email protected]>
Co-authored-by: Yash Raj Pandey <[email protected]>
Co-authored-by: Zang Peiyu <[email protected]>
Co-authored-by: Sameer Kankute <[email protected]>
Co-authored-by: mateo-berri <[email protected]>

* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (BerriAI#29693)

* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens

* test: scope local cost map env var with monkeypatch to avoid test pollution

* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (BerriAI#30764)

* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold

_mask_value did partial reveal by showing the first visible_prefix and last
visible_suffix characters, but for a value whose length was at or below
visible_prefix + visible_suffix (8 by default) it returned the value verbatim.
A value of exactly 8 chars fell through the length guard and computed
masked_length == 0, reconstructing the original string with no mask characters;
anything shorter hit the early return. Either way short credentials were emitted
in plaintext.

mask_dict routes real secrets through this path, so an 8-char-or-shorter redis
password, api key, or token could be written to logs and the UI unmasked. The
sibling helper mask_sensitive_keys already guards this case; _mask_value now does
the same by fully masking any value at or below the threshold.

* fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers

Fully masking short values is the right default for secret masking, but
CooldownCache reuses the masker purely to truncate exception messages to the
first 50 characters, and it relies on short messages being returned readable.
Masking those blanked out short exception text and broke its tests.

Add a mask_short_values flag (default True, secure) and have CooldownCache pass
False so it keeps the truncation behavior, while every secret-masking caller
still gets short values fully masked.

* fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview

MCPDebug uses the masker to preview auth tokens in debug headers and documents
that values of 10 chars or fewer are shown unchanged so token types stay
distinguishable. Pass mask_short_values=False so that diagnostic behavior is
preserved while secret maskers keep masking short values.

* fix(mcp_debug): mask short auth values in debug headers instead of echoing them

Earlier this masker opted out of short-value masking to keep a token preview, but
that echoes short authorization and token values verbatim in debug response
headers, which is the same leak this change is meant to close. Auth material
should never be emitted in full, so mask short values here too; the first/last
character preview still applies to longer tokens. Only CooldownCache keeps the
opt-out, since it truncates exception text rather than masking secrets.

* test(mcp_debug): assert masked short value preserves length

* refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (BerriAI#30917)

Fireworks AI deprecated audio inference on 2026-06-10
(https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation).
Live API testing confirms the endpoint is already non-functional: a valid
Fireworks API key receives HTTP 401 "Unauthorized" from
api.fireworks.ai/inference/v1/audio/transcriptions for every request,
regardless of payload. The audio-prod.api.fireworks.ai host referenced in
the test suite returns 401 for every path; the entire host is decommissioned.

Remove the dead FireworksAIAudioTranscriptionConfig class and every
reference to it across the codebase:

- Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line
  config class that inherited from OpenAIWhisperAudioTranscriptionConfig)
- Remove the Fireworks branch from
  ProviderConfigManager.get_provider_audio_transcription_config() in
  litellm/utils.py; update the stale comment in
  get_optional_params_transcription that referenced fireworks ai
- Remove the FireworksAIAudioTranscriptionConfig entries from
  LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in
  litellm/_lazy_imports_registry.py
- Remove the TYPE_CHECKING re-export in litellm/__init__.py
- Remove the transcription branch in the fireworks_ai case of
  get_supported_openai_params() in
  litellm/litellm_core_utils/get_supported_openai_params.py
- Remove the whisper-v3 and whisper-v3-turbo entries from
  model_prices_and_context_window.json and
  litellm/model_prices_and_context_window_backup.json (both had
  mode: audio_transcription and zero-cost pricing)
- Remove the TestFireworksAIAudioTranscription test class and its
  imports from tests/llm_translation/test_fireworks_ai_translation.py

No other provider is affected. The openai_compatible_providers list,
FireworksAIMixin, and the OpenAI Whisper transcription handler all stay
because they are shared with other Fireworks endpoints and other
providers. The provider_endpoints_support.json registry already had
audio_transcriptions set to false for fireworks_ai.

* feat: add darkbloom provider (BerriAI#30876)

* feat: add darkbloom provider

* fix: document darkbloom provider endpoints

* fix: address darkbloom review feedback

* fix: update darkbloom tool metadata

* fix: fail fast for non-Postgres database URLs (BerriAI#30883)

* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup

LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.

Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.

Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.

Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.

Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.

* fix: resolve CI failures and proxy DB URL typing issue

* fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging

* Validate DIRECT_URL alongside DATABASE_URL startup guards

* fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (BerriAI#24608) (BerriAI#30946)

* fix(bedrock): surface modeled HTTP status for mid-stream error events (BerriAI#24608)

* test(bedrock): mid-stream server errors trigger streaming fallback (BerriAI#24608)

* style(bedrock): black-format stream-error helper (BerriAI#24608)

* fix(mcp): re-land native tool preservation with typed annotations (BerriAI#30645)

* fix(mcp): preserve native tools in semantic filter hook with typed annotations

* fix(mcp): tighten _is_mcp_tool Chat Completions shape check

* fix(sambanova): return embeddings supported params instead of dropping them (BerriAI#30937)

* fix(router): send fallback metadata when streaming (BerriAI#30914)

When a streaming request triggers a fallback, there was previously no way to
know it happened. This commit addresses this in a few ways:

1. The response now correctly populates the fallback headers
    (`x-litellm-attempted-fallbacks`) so callers know a fallback happened.
2. The correct model ID is passed in the streaming chunks.
3. A streaming chunk with the fallback error can be optionally sent back
    to the client (opt-in) by passing `include_fallback_errors: true` in
    the request.

The format of the fallback errors while streaming is intentionally OpenAI
compatible to not break existing libraries that parse these events. It was
tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not
delieved unexpectedly to callers by default.

* fix(mistral): drop output-only reasoning fields from input messages (BerriAI#30884)

LiteLLM attaches reasoning_content and thinking_blocks to assistant
responses. Replaying those assistant turns verbatim forwarded the fields
back to Mistral, whose input schema forbids unknown keys, so the whole
request failed with a 422 extra_forbidden and reasoning models became
unusable across multiple turns.

Strip both fields from assistant messages before the request is built, in
a spot that runs ahead of the image/file branch so it applies on every
path. Fixes BerriAI#30835

Co-authored-by: Cursor <[email protected]>

* fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (BerriAI#30652)

* fix(perplexity): bill search queries at the per-request price, not 1/1000

The fallback cost calculator divided search_context_cost_per_query by
1000, but that field stores the per-request price in USD: sonar is
{low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published
$5/$8/$12 per 1,000 requests expressed per request. The gemini cost
calculator reads the same field per request with no division (its
docstring calls it "the per-request cost").

The division understated search cost by 1000x on every Perplexity call
that falls back to manual calculation (i.e. when the API does not return
a pre-computed usage.cost). Use the value directly.

Update the tests that had encoded the /1000 factor in their expectations,
and drop an unused import flagged by ruff in the touched test file.

* test(perplexity): update integration test search-cost expectations to per-request

The integration tests still encoded the old /1000 search-cost factor, so
they failed once the fallback calculator was corrected to bill
search_context_cost_per_query per request. Update the four expected-cost
computations (and the high-volume dollar-value comments) to match.

* test(perplexity): drop unused mock imports flagged by ruff

* fix: include model_access_groups when expanding all-team-models in get_team_models (BerriAI#30622)

* fix(fireworks_ai): return None for transcription in get_supported_openai_params

Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is
decommissioned. Without an explicit transcription branch, requests with
request_type='transcription' fell through to the else and returned
FireworksAIConfig chat-completion params. Return None instead to signal
the provider does not support transcription.

* fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting

Without an operator gate, any authenticated caller could set include_fallback_errors=True,
trigger a fallback, and read raw upstream exception messages from the
x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event.

Strip include_fallback_errors from request data in common_processing_pre_call_logic
when expose_fallback_errors_to_caller is not set, so the router never builds the
error list. Also gate _should_include_fallback_errors on the same setting as a
secondary check for the streaming SSE injection path.

* test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test

The operator gate added in e7ff3e1 means include_fallback_errors is only
honoured when general_settings.expose_fallback_errors_to_caller is True.
Set that flag via monkeypatch in the test that exercises the emit path.

* test(prompt_templates): make test_convert_url hermetic instead of hitting picsum.photos

test_convert_url called convert_url_to_base64 against a live picsum.photos
URL and asserted nothing, so it added no real signal and broke CI whenever
the host was unreachable (it was returning 522 and blocking this branch).
Replace the live call with a mocked HTTP client and assert the produced
base64 data URL, so the conversion path is exercised deterministically with
no network dependency. This suite runs under VCR, which is why a transport
level mock (respx) does not reliably intercept; mocking the client object
itself is robust regardless.

* fix(interactions): drop role from Interaction response to match Google spec

Google removed the output-only role field from the Interaction schema (it
now lives only on Turn), so the live OpenAPI compliance canary started
failing with 'role' not in spec. Reconcile our generated types by removing
role from Interaction, CreateModelInteractionParams, CreateAgentInteractionParams
and from the LiteLLM InteractionsAPIResponse/InteractionsAPIStreamingResponse,
stop stamping role=model in the responses-to-interactions transformation, and
update the compliance and integration tests accordingly. Turn.role is kept
since the spec still defines it.

* fix: align all-team-models sentinel access

* fix(router): forward include_fallback_errors through multi-hop fallbacks

run_async_fallback received include_fallback_errors as an explicit named
parameter, so it was bound out of **kwargs and never reached the nested
async_function_with_fallbacks call. Multi-hop fallback chains (a fallback
group that itself fails over) therefore stopped collecting fallback errors
beyond the first hop when a caller opted in. Re-inject the flag into kwargs
before the nested call so inner hops keep accumulating errors, which
add_fallback_headers_to_response already merges across levels.

---------

Co-authored-by: Srivatsa Kamballa <[email protected]>
Co-authored-by: Ahmad Shahzad <[email protected]>
Co-authored-by: Jeremy Chapeau <[email protected]>
Co-authored-by: KRISH SONI <[email protected]>
Co-authored-by: Kent <[email protected]>
Co-authored-by: Ayush Shekhar <[email protected]>
Co-authored-by: dav nguyxn <[email protected]>
Co-authored-by: Tal Marian <[email protected]>
Co-authored-by: Hemant K <[email protected]>
Co-authored-by: Cursor <[email protected]>
Co-authored-by: Yash Raj Pandey <[email protected]>
Co-authored-by: Zang Peiyu <[email protected]>
Co-authored-by: mateo-berri <[email protected]>

* fix(ui): widen Y-axis gutter on Usage charts so large token/request labels aren't clipped

The Total Tokens Over Time and Total Requests Over Time AreaCharts on the
Usage page used Tremor's default yAxisWidth (~56 px), which is too narrow
once totals pass the hundred-million mark — leading digits of labels like
"100.00M" / "4500.00M" got clipped against the chart edge. The requests
chart was worse: it formatted with toLocaleString(), so billion-scale
request counts produced "1,000,000,000" (13 chars) and overflowed
immediately.

Fix in two places so neither alone has to carry the whole margin:
- activity_metrics.tsx: add yAxisWidth={80} to both AreaCharts, and
  switch the requests chart to the shared valueFormatter so it uses the
  same compact k/M/B suffixes as the tokens chart.
- value_formatters.tsx: add a >= 1e9 branch to valueFormatter /
  valueFormatterSpend that emits a "B" suffix (4.50B, $4.50B), keeping
  every formatted label at most 7 chars.

Co-Authored-By: Claude Opus 4 (1M context) <[email protected]>

* Update ui/litellm-dashboard/src/components/UsagePage/utils/value_formatters.tsx

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* docs(readme): add Deploy on AWS/GCP with Terraform section

Adds a quickstart for the two published Terraform modules on the public
registry (BerriAI/litellm/aws and BerriAI/litellm/google). Copy-paste
main.tf for each cloud, the one-time GCP Artifact Registry remote-repo
command, and pointers to the registry pages for the full input surface.

Sits inside the Get Started section, between the gateway/SDK table and
Run in Developer Mode -- where someone scanning the README for "how do I
deploy this" will land.

Co-Authored-By: Claude Opus 4.7 <[email protected]>

* docs(readme): add 1-click deploy buttons for AWS + GCP

GCP gets the real 1-click: Open in Cloud Shell badge that clones the repo
and walks through `terraform apply` via the existing DeployStack
tutorial (already shipped at terraform/litellm/gcp/examples/default/
TUTORIAL.md). User just picks a project.

AWS gets a soft 1-click: a Launch in AWS CloudShell badge that opens an
in-browser, already-authenticated shell. User runs four commands
(clone + cd + cp tfvars + terraform apply) once inside. There's no
native AWS deeplink that pre-clones a repo + runs a tutorial -- CFN
"Launch Stack" + CodeBuild would be needed for that, and that's a
separate piece of work.

Co-Authored-By: Claude Opus 4.7 <[email protected]>

* docs(readme): move AWS + GCP deploy buttons next to Render button

* docs(readme): unify deploy button sizes and badge styles

* docs(readme): bump deploy button height to 48 to match Render/Railway

* docs(readme): bump AWS/GCP badge height to compensate for SVG padding

* docs(readme): bump AWS/GCP badge height to 72

* docs(readme): bump AWS/GCP badge height to 84

* fix(readme): make deploy buttons same height (48px)

https://claude.ai/code/session_01MxQRMHSDXbqJh74rF86UBc

* docs(readme): flag GCP project ID substitution in image_registry

* docs(readme): equalize deploy button heights and fix Cloud Shell button font

GitHub rewrites an image's height attribute to "height: auto; max-height: Npx", which only caps and never stretches, so each image renders at its intrinsic height. The AWS/GCP shields badges are intrinsically 28px while the Render/Railway buttons are 40px, leaving the row uneven regardless of the height="48" we set. Replace the two shields badges with committed 40px PNGs so all four header buttons render at the same 40px.

Also swap the Cloud Shell button from open-btn.svg to open-btn.png. The SVG renders its label as live text with font-family "Roboto, Sans" and no generic fallback; since neither font exists in GitHub's render environment, the text fell back to a serif (Times New Roman). The PNG bakes in the correct typeface.

* docs(readme): collapse Railway deploy anchor to a single line

The Railway button wrapped its img across indented lines, so the anchor contained leading and trailing whitespace. GitHub underlines link content, rendering that whitespace as a small blue underline beside the button. Put the anchor on one line like the other three buttons so there is no inner whitespace to underline.

* Add Claude Fable 5 cost map entries as a data-only hotfix

Backports only the model map changes from #30064 so deployments on
released litellm versions pick up Fable 5 pricing, context window, and
the adaptive thinking flag through the hosted cost map fetch without
upgrading. Includes the supports_sampling_params flag on the 28
Fable 5 / Opus 4.7 / Opus 4.8 entries (ignored by released code, read
by the gating that ships with the next release) and the matching
one-line schema declaration so the map validation test passes.

https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm

* fix: correct context window tokens for GPT-5 Pro and GPT-5.4 Mini/Nano

Three bugs in model_prices_and_context_window.json:

1. gpt-5-pro and gpt-5-pro-2025-10-06: max_input_tokens and max_tokens
   were SWAPPED. GPT-5 Pro has a 400K context window (input) with 128K
   max output, but the values were set as max_input=128000,
   max_tokens=272000. This caused token limit errors when sending
   prompts over 128K tokens to GPT-5 Pro.

2. gpt-5.4-mini and gpt-5.4-mini-2026-03-17: max_input_tokens was
   272000, but GPT-5.4 Mini shares the same 1,050,000 token context
   window as GPT-5.4. This was inconsistent with the azure/ variants
   which already correctly had 1,050,000.

3. gpt-5.4-nano and gpt-5.4-nano-2026-03-17: same issue as Mini,
   max_input_tokens was 272000 instead of 1,050,000.

Source: OpenAI model documentation and contextwindows.dev which
aggregates official context window sizes.

Fixes #30928 (partially — the issue incorrectly claims gpt-5/gpt-5-mini
should be 400K; their 272K values are correct per OpenAI docs)

* fix: also correct max_output_tokens for gpt-5-pro (272000→128000)

Per reviewer feedback, max_output_tokens was left at 272000 while
max_tokens was corrected to 128000, causing an internal inconsistency.
Both should be 128000 per OpenAI docs.

* fix(cost): price gpt-image generated output tokens as image tokens (#31147)

The OpenAI Images endpoints (/v1/images/generations, /v1/images/edits) return
usage with no output token breakdown — litellm's `ImageUsage` has no
`output_tokens_details` field — so generated-image OUTPUT tokens were priced at
the text rate (`output_cost_per_token`) instead of the image rate
(`output_cost_per_image_token`). For gpt-image-2 that is $10/1M vs $30/1M, a ~3x
undercount on the dominant cost component (image output is ~74% of spend). This
also affects azure gpt-image, which shares this calculator.

The OpenAI gpt-image cost calculator re-implemented usage handling instead of
reusing `calculate_image_response_cost_from_usage`, the shared helper that
azure_ai/gemini/vertex_ai already use. That helper classifies generated output
tokens as image tokens when the provider does not itemize output, and splits
text/image when it does.

Fix: route the ImageUsage path through `calculate_image_response_cost_from_usage`
(pre-transformed chat Usage objects are still costed directly). Adds a regression
test for the no-breakdown ImageUsage case (gpt-image-2).

* fix(bedrock): route application-inference-profile ARNs to converse (#18258) (#31098)

A bare application-inference-profile ARN passed as bedrock/arn:... fell
through to the invoke route, which cannot derive a provider from the
opaque profile id and raised 'Unknown provider=None'. The converse route
needs no provider, so detect these ARNs in get_bedrock_route and route
them to converse, matching the behavior of the already-documented
bedrock/converse/arn:... workaround.

Explicit invoke/ prefixes still win, and they remain a dead end for these
ARNs by design (no provider derivable). System-defined inference-profile
ARNs that embed a known model, and other opaque ARN types
(provisioned-model, imported-model, custom-model-deployment) that are
frequently invoke-only, are deliberately left on their current routes;
tests guard both boundaries.

* fix(moonshot): stop mutating caller messages on tool_choice='required' (#31060)

_add_tool_choice_required_message appended the "select a tool" prompt to
the caller's messages list in place, so transform_request corrupted the
caller's conversation history and appended a duplicate prompt on every
retry. Build and return a new list instead so the call stays idempotent.

Adds a regression test asserting the input messages list is unchanged
across repeated transform_request calls.

Co-authored-by: Wassbdr <[email protected]>

* fix(transcription): accept fractional usage.seconds in diarized_json responses (#30996)

gpt-4o-transcribe and compatible ASR backends return a diarized_json
response with usage={"type": "duration", "seconds": <float>}, e.g. 295.8.
TranscriptionUsageDurationObject typed seconds as int, so parsing the
response raised a pydantic ValidationError (int_from_float). That error
surfaces as an APIConnectionError which the router treats as retryable, so
it keeps re-calling the upstream (200 every time) until the upstream
rate-limits and returns 429 to the caller.

OpenAI specs this field as a float (see openai SDK UsageDuration.seconds),
so widen seconds to float. With the parse succeeding there is no exception
left to retry, which removes the loop.

Co-authored-by: Neimar Avila <[email protected]>

* fix(deepseek): drop non-function tools before chat completions call (#30910)

* fix(deepseek): drop non-function tools before chat completions call

DeepSeek's /chat/completions only accepts tools of type "function".
Requests bridged from /v1/responses can carry responses-API-native tool
types, for example a Codex CLI tool typed "namespace", which DeepSeek
rejects with "unknown variant 'namespace', expected 'function'" so the
whole request fails (issue #30722).

Filter unsupported tool types in the DeepSeek request transform so the
function tools still go through; when nothing callable remains, also drop
the now-dangling tool_choice and parallel_tool_calls

Fixes #30722

* test(deepseek): cover async tool filtering and document tool_choice assumption

Add an async_transform_request regression test so the sync and async tool
filtering paths cannot silently diverge, and document in _drop_unsupported_tools
that only non-function tools are dropped, so a function-named tool_choice always
references a surviving tool

* feat(catalog): add zai/glm-5.1, zai/glm-4.7-flash, openrouter/z-ai/glm-5.1 (#29840)

* feat(ui): surface team budget on key overview when key has no own budget (#30801)

* feat(ui): surface team budget on key overview when key has no own budget

* fix(ui): replace IIFE with derived variable and use find() for team budget display

* fix(anthropic): emit replayable streaming thinking blocks (#31022)

* feat(proxy): read cold-storage prompts back in the logs detail view (#30364)

* feat(proxy): read cold-storage prompts back in the logs detail view

When a deployment offloads prompts and responses to cold storage instead of
Postgres, the spend-log row holds only "{}" placeholders plus a
metadata.cold_storage_object_key pointer, so the UI logs detail drawer showed
nothing. The detail endpoint only read the placeholder columns and never
fetched the object back.

Resolve the payload per row based on actual content, not a config flag: if
Postgres has content, return it; otherwise read the exact stored object key and
fetch from the configured cold storage backend through ColdStorageHandler.
Reading the persisted key is a single GET. The key embeds a microsecond
timestamp that cannot be reconstructed from the millisecond-precision startTime
column, and listing the day's prefix to match on request_id would be too
expensive for this per-open path.

Also teach the detail drawer's pretty-view parser to accept a bare messages
array. The cold storage payload carries the prompt as a top-level messages list
with no proxy_server_request, so without this the output rendered while the
input stayed blank.

ColdStorageHandler gains an optional injected logger so the resolver can be unit
tested without monkeypatching. Postgres-stored prompts are unaffected: the fast
path returns the existing columns and the request-body object still renders the
same way.

* Update litellm/proxy/spend_tracking/spend_management_endpoints.py

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* test(proxy): cover ColdStorageHandler resolution paths and cold-storage fetch failure

Add unit tests for ColdStorageHandler (injected logger, graceful None when no
logger is configured, and resolution of a configured logger from the callback
registry) and a regression test asserting a cold storage backend exception
degrades to the Postgres values instead of surfacing a 500.

---------

Co-authored-by: Bytechoreographer <[email protected]>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* fix(mavvrik): advance metricsMarker after upload; fix scheduler startup (#31068)

* fix(mavvrik): advance metricsMarker after upload + fix scheduler startup

Two bugs fixed:

1. deliver() never called PATCH /metrics/agent/ai/{connectionId} after a
   successful GCS upload, so metricsMarker stayed at 0 and every daily run
   re-exported the same dates in an infinite catch-up loop.
   Fix: add _update_metrics_marker(date_epoch) called at the end of deliver()
   after _upload_to_gcs() succeeds. A 4xx warns but does not raise (the GCS
   file is already committed). A 410 raises consistent with the rest of the
   destination.

2. init_mavvrik_focus_background_job runs at proxy startup before any LLM call
   has triggered lazy instantiation of MavvrikFocusLogger, so it found no
   logger instance and silently skipped registering the daily export job.
   Fix: if no instance is found but "mavvrik" is in litellm.callbacks, call
   _init_custom_logger_compatible_class to force instantiation before
   the APScheduler job is registered.

* fix(mavvrik): catch up from earliest window when metricsMarker=0

When the connector is freshly registered, metricsMarker=0 parses to None.
The catch-up block was guarded by `if last_ingested and ...` which skipped
it entirely for None, so only yesterday was exported instead of the full
_MAX_CATCHUP_DAYS window.

Fix: treat None as being _MAX_CATCHUP_DAYS behind (start from earliest_catchup).
The existing > 7 day warning only fires for non-None markers that are old.

* fix(mavvrik): use now as end_time for yesterday's export window

LiteLLM_DailyUserSpend rows for a given date get their updated_at
bumped by the spend flush job throughout the next morning. The core
database query filters on updated_at, so capping end_time at midnight
(yesterday + 1 day) missed any spend rows flushed after midnight.

Fix: pass now (cron fire time) as end_time for the daily "yesterday"
window so all fully-settled rows are captured regardless of when the
flush job ran.

Verified: claude-3-5-sonnet BilledCost went from 0.0 to ~$2.40 per
row in the exported FOCUS CSV.

* fix(mavvrik): also use now as end_time for catch-up windows

* fix(mavvrik_focus): pass required args to _init_custom_logger_compatible_class

Calling it with only logging_integration raised TypeError at proxy startup
because internal_usage_cache and llm_router have no defaults. Also fix test
name to reflect the actual status code (5xx not 4xx) used in the mock.

* ci: retrigger CI run

* feat: pass through optional `instruction` field in the rerank API (vLLM/Qwen3-Reranker) (#30757)

* Add optional `instruction` passthrough to the rerank API

vLLM's /v1/rerank and /v1/score accept an optional top-level `instruction`
field (folded into the model's chat_template_kwargs and consumed by the
chat template — e.g. Qwen3-Reranker). LiteLLM's managed rerank route silently
dropped it: RerankRequest / OptionalRerankParams had no such field, so the
outgoing body was rebuilt without it.

Thread an opt-in `instruction: Optional[str]` through rerank()/arerank(),
get_optional_rerank_params, and the hosted_vllm transformation into the
request body, only when non-None. When callers omit it, model_dump(exclude_none)
drops the field and the outgoing request is byte-for-byte unchanged — fully
backward-compatible. (DeepInfra already forwards `instruction` via
non_default_params; this formalizes the field in the shared types.)

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* Address review: thread `instruction` as a typed param + cover rerank_utils

Per PR review (greptile P2 + codecov):

- Make `instruction` a typed, named argument on the rerank provider interface
  instead of recovering it from the opaque `non_default_params` blob. Adds
  `instruction: Optional[str] = None` to `BaseRerankConfig.map_cohere_rerank_params`
  and every provider override, and forwards it explicitly from
  `get_optional_rerank_params`. hosted_vllm now reads the named param directly.
  It is still also surfaced in `non_default_params` so providers that read it
  there (e.g. DeepInfra) keep working now that `rerank()` consumes `instruction`
  as a named param rather than leaving it in **kwargs.
- Add get_optional_rerank_params unit tests (present + absent) to cover the
  previously-uncovered threading line flagged by codecov.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* fix: scan rerank `instruction` through request guardrails

The rerank guardrail translation (CohereRerankHandler.process_input_messages)
only scanned `query`, so the newly added `instruction` field reached the
backend model unscanned. Since instruction-aware rerankers (hosted vLLM /
Qwen3-Reranker) fold `instruction` into the prompt, an authenticated caller
could place content there to bypass configured rerank request guardrails.

Generalize the handler to scan every user-controlled text field (`query` and
`instruction`) in one apply_guardrail call and write each sanitized value back
by index. Query-only requests are unchanged (single-element list at index 0);
non-string fields are left untouched. Adds tests covering instruction
scanning, PII masking write-back, and the non-string case.

Addresses the Veria AI security review on PR #30757.

* test: narrow Optional results before len() to satisfy basedpyright budget

The lint gate (basedpyright delta-vs-base budget) flagged one new
reportArgumentType: len(result.results) where results is
List[RerankResponseResult] | None. Assert results is not None first to
narrow the type before len()/indexing.

* fix: read rerank `instruction` from kwargs to satisfy basedpyright budget

The basedpyright delta-vs-base gate flagged one new reportArgumentType: the
Router forwards rerank calls via an untyped `**kwargs` unpack
(`litellm.arerank(**{**data, **kwargs})`), and declaring `instruction` as a
typed named param on the public `rerank`/`arerank` entrypoints made pyright
check that key against `str | None`, adding an error at router.py with no real
safety gain. Read `instruction` from kwargs in `rerank` instead.

It remains fully typed where it matters - threaded as a typed argument through
`get_optional_rerank_params` and each provider's `map_cohere_rerank_params`
(the original Greptile P2 ask). Whole-repo reportArgumentType is back to the
base count (net 0); rerank hosted_vllm + cohere guardrail suites pass; ruff clean.

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

* fix(github_copilot): synthesize empty choices at the provider seam (#30929)

Newer Copilot Claude models (opus-4.7, opus-4.8) return responses with
choices=[], either carrying Anthropic-native content blocks or, for the
max_tokens=1 probe Claude Code sends, no content at all. github_copilot
is dispatched through the OpenAI SDK handler, which calls
convert_to_model_response_object directly and never invokes
GithubCopilotConfig.transform_response, so the empty-choices guard there
surfaced as a 500

Instead of synthesizing choices inside the shared
convert_to_model_response_object (which would silently turn empty choices
into a fabricated success for every provider), add a no-op
transform_parsed_response_dict hook on BaseConfig. GithubCopilotConfig
overrides it to synthesize choices from Anthropic-native content, reusing
its existing parsing, and the OpenAI SDK handler routes its parsed
response through the hook before generic conversion. The core utility
keeps treating empty choices as an error for all other providers

Fixes: #30927

Signed-off-by: David J. M. Karlsen <[email protected]>

* fix(router): stop fallback lookups from mutating the router fallbacks config (#30624)

* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (#29693)

* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens

* test: scope local cost map env var with monkeypatch to avoid test pollution

* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (#30764)

* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold

_mask_value did partial reveal by showing the first visible_prefix and last
visible_suffix characters, but for a value whose length was at or below
visible_prefix + visible_suffix (8 by default) it returned the value verbatim.
A value of exactly 8 chars fell through the length guard and computed
masked_length == 0, reconstructing the original string with no mask characters;
anything shorter hit the early return. Either way short credentials were emitted
in plaintext.

mask_dict routes real secrets through this path, so an 8-char-or-shorter redis
password, api key, or token could be written to logs and the UI unmasked. The
sibling helper mask_sensitive_keys already guards this case; _mask_value now does
the same by fully masking any value at or below the threshold.

* fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers

Fully masking short values is the right default for secret masking, but
CooldownCache reuses the masker purely to truncate exception messages to the
first 50 characters, and it relies on short messages being returned readable.
Masking those blanked out short exception text and broke its tests.

Add a mask_short_values flag (default True, secure) and have CooldownCache pass
False so it keeps the truncation behavior, while every secret-masking caller
still gets short values fully masked.

* fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview

MCPDebug uses the masker to preview auth tokens in debug headers and documents
that values of 10 chars or fewer are shown unchanged so token types stay
distinguishable. Pass mask_short_values=False so that diagnostic behavior is
preserved while secret maskers keep masking short values.

* fix(mcp_debug): mask short auth values in debug headers instead of echoing them

Earlier this masker opted out of short-value masking to keep a token preview, but
that echoes short authorization and token values verbatim in debug response
headers, which is the same leak this change is meant to close. Auth material
should never be emitted in full, so mask short values here too; the first/last
character preview still applies to longer tokens. Only CooldownCache keeps the
opt-out, since it truncates exception text rather than masking secrets.

* test(mcp_debug): assert masked short value preserves length

* refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (#30917)

Fireworks AI deprecated audio inference on 2026-06-10
(https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation).
Live API testing confirms the endpoint is already non-functional: a valid
Fireworks API key receives HTTP 401 "Unauthorized" from
api.fireworks.ai/inference/v1/audio/transcriptions for every request,
regardless of payload. The audio-prod.api.fireworks.ai host referenced in
the test suite returns 401 for every path; the entire host is decommissioned.

Remove the dead FireworksAIAudioTranscriptionConfig class and every
reference to it across the codebase:

- Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line
  config class that inherited from OpenAIWhisperAudioTranscriptionConfig)
- Remove the Fireworks branch from
  ProviderConfigManager.get_provider_audio_transcription_config() in
  litellm/utils.py; update the stale comment in
  get_optional_params_transcription that referenced fireworks ai
- Remove the FireworksAIAudioTranscriptionConfig entries from
  LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in
  litellm/_lazy_imports_registry.py
- Remove the TYPE_CHECKING re-export in litellm/__init__.py
- Remove the transcription branch in the fireworks_ai case of
  get_supported_openai_params() in
  litellm/litellm_core_utils/get_supported_openai_params.py
- Remove the whisper-v3 and whisper-v3-turbo entries from
  model_prices_and_context_window.json and
  litellm/model_prices_and_context_window_backup.json (both had
  mode: audio_transcription and zero-cost pricing)
- Remove the TestFireworksAIAudioTranscription test class and its
  imports from tests/llm_translation/test_fireworks_ai_translation.py

No other provider is affected. The openai_compatible_providers list,
FireworksAIMixin, and the OpenAI Whisper transcription handler all stay
because they are shared with other Fireworks endpoints and other
providers. The provider_endpoints_support.json registry already had
audio_transcriptions set to false for fireworks_ai.

* feat: add darkbloom provider (#30876)

* feat: add darkbloom provider

* fix: document darkbloom provider endpoints

* fix: address darkbloom review feedback

* fix: update darkbloom tool metadata

* fix: fail fast for non-Postgres database URLs (#30883)

* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup

LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.

Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.

Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.

Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.

Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.

* fix: resolve CI failures and proxy DB URL typing issue

* fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging

* Validate DIRECT_URL alongside DATABASE_URL startup guards

* fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (#24608) (#30946)

* fix(bedrock): surface modeled HTTP status for mid-stream error events (#24608)

* test(bedrock): mid-stream server errors trigger streaming fallback (#24608)

* style(bedrock): black-format stream-error helper (#24608)

* fix(mcp): re-land native tool preservation with typed annotations (#30645)

* fix(mcp): preserve native tools in semantic filter hook with typed annotations

* fix(mcp): tighten _is_mcp_tool Chat Completions shape check

* fix(sambanova): return embeddings supported params instead of dropping them (#30937)

* fix(router): send fallback metadata when streaming (#30914)

When a streaming request triggers a fallback, there was previously no way to
know it happened. This commit addresses this in a few ways:

1. The response now correctly populates the fallback headers
    (`x-litellm-attempted-fallbacks`) so callers know a fallback happened.
2. The correct model ID is passed in the streaming chunks.
3. A streaming chunk with the fallback error can be optionally sent back
    to the client (opt-in) by passing `include_fallback_errors: true` in
    the request.

The format of the fallback errors while streaming is intentionally OpenAI
compatible to not break existing libraries that parse these events. It was
tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not
delieved unexpectedly to callers by default.

* fix(mistral): drop output-only reasoning fields from input messages (#30884)

LiteLLM attaches reasoning_content and thinking_blocks to assistant
responses. Replaying those assistant turns verbatim forwarded the fields
back to Mistral, whose input schema forbids unknown keys, so the whole
request failed with a 422 extra_forbidden and reasoning models became
unusable across multiple turns.

Strip both fields from assistant messages before the request is built, in
a spot that runs ahead of the image/file branch so it applies on every
path. Fixes #30835

Co-authored-by: Cursor <[email protected]>

* fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (#30652)

* fix(perplexity): bill search queries at the per-request price, not 1/1000

The fallback cost calculator divided search_context_cost_per_query by
1000, but that field stores the per-request price in USD: sonar is
{low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published
$5/$8/$12 per 1,000 requests expressed per request. The gemini cost
calculator reads the same field per request with no division (its
docstring calls it "the per-request cost").

The division understated search cost by 1000x on every Perplexity call
that falls back to manual calculation (i.e. when the API does not return
a pre-computed usage.cost). Use the value directly.

Update the tests that had encoded the /1000 factor in their expectations,
and drop an unused import flagged by ruff in the touched test file.

* test(perplexity): update integration test search-cost expectations to per-request

The integration tests still encoded the old /1000 search-cost factor, so
they failed once the fallback calculator was corrected to bill
search_context_cost_per_query per request. Update the four expected-cost
computations (and the high-volume dollar-value comments) to match.

* test(perplexity): drop unused mock imports flagged by ruff

* fix: include model_access_groups when expanding all-team-models in get_team_models (#30622)

* fix(fireworks_ai): return None for transcription in get_supported_openai_params

Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is
decommissioned. Without an explicit transcription branch, requests with
request_type='transcription' fell through to the else and returned
FireworksAIConfig chat-completion params. Return None instead to signal
the provider does not support transcription.

* fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting

Without an operator gate, any authenticated caller could set include_fallback_errors=True,
trigger a fallback, and read raw upstream exception messages from the
x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event.

Strip include_fallback_errors from request data in common_processing_pre_call_logic
when expose_fallback_errors_to_caller is not set, so the router never builds the
error list. Also gate _should_include_fallback_errors on the same setting as a
secondary check for the streaming SSE injection path.

* test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test

The operator gate added in e7ff3e1 means include_fallback_errors is only
honoured when general_settings.expose_fallback_errors_to_caller is True.
Set that flag via monkeypatch in the test that exercises the emit path.

* test(prompt_templates): make test_convert_url hermetic instead of hitting picsum.photos

test_convert_url called convert_url_to_base64 against a live picsum.photos
URL and asserted nothing, so it added no real signal and broke CI whenever
the host was unreachable (it was returning 522 and blocking this branch).
Replace the live call with a mocked HTTP client and assert the produced
base64 data URL, so the conversion path is exercised deterministically with
no network dependency. This suite runs under VCR, which is why a transport
level mock (respx) does not reliably intercept; mocking the client object
itself is robust regardless.

* fix(interactions): drop role from Interaction response to match Google spec

Google removed the output-only role field from the Interaction schema (it
now lives only on Turn), so the live OpenAPI compliance canary started
failing with 'role' not in spec. Reconcile our generated types by removing
role from Interaction, CreateModelInteractionParams, CreateAgentInteractionParams
and from the LiteLLM InteractionsAPIResponse/InteractionsAPIStreamingResponse,
stop stamping role=model in the responses-to-interactions transformation, and
update the compliance and integration tests accordingly. Turn.role is kept
since the spec still defines it.

* fix: align all-team-models sentinel access

* fix(router): forward include_fallback_errors through multi-hop fallbacks

run_async_fallback received include_fallback_errors as an explicit named
parameter, so it was bound out of **kwargs and never reached the nested
async_function_with_fallbacks call. Multi-hop fallback chains (a fallback
group that itself fails over) therefore stopped collecting fallback errors
beyond the first hop when a caller opted in. Re-inject the flag into kwargs
before the nested call so inner hops keep accumulating errors, which
add_fallback_headers_to_response already merges across levels.

* fix(router): stop fallback lookups from mutating the router fallbacks config

get_fallback_model_group resolved a bare-string fallback by popping it out
of the fallbacks list it was handed. That list is frequently the live
router.fallbacks config, so a single lookup permanently removed the entry and
the configured fallback stopped applying to later requests until restart. The
pop also ran inside enumerate(), shifting indices and skipping an adjacent
string fallback. Read the item instead of popping it, and add a regression
test that fails on the old mutating behavior

---------

Co-authored-by: Srivatsa Kamballa <[email protected]>
Co-authored-by: Ahmad Shahzad <[email protected]>
Co-authored-by: Jeremy Chapeau <[email protected]>
Co-authored-by: KRISH SONI <[email protected]>
Co-authored-by: Kent <[email protected]>
Co-authored-by: Ayush Shekhar <[email protected]>
Co-authored-by: dav nguyxn <[email protected]>
Co-authored-by: Tal Marian <[email protected]>
Co-authored-by: Hemant K <[email protected]>
Co-authored-by: Cursor <[email protected]>
Co-authored-by: Yash Raj Pandey <[email protected]>
Co-authored-by: Zang Peiyu <[email protected]>
Co-authored-by: Sameer Kankute <[email protected]>
Co-authored-by: mateo-berri <[email protected]>

* fix(sambanova): update pricing, deprecate retired models, and add missing models (#30016)

* feat(bedrock): add amazon.titan-embed-g1-text-02 embedding model support

- Add model to provider routing allowlist in embedding.py
- Add request transformation using AmazonTitanG1Config
- Add response transformation using AmazonTitanG1Config
- Add pricing metadata to model_prices_and_context_window.json
- Add unit tests for embedding and model info

Fixes missing cost tracking reported in #29786
Related to VANDRANKI/litellm PR #29790

* style: fix syntax error, trailing whitespace and missing newline

* style: apply black formatting to embedding.py

* style: apply black formatting to test_bedrock_embedding.py

* fix(sambanova): update pricing, fix context windows, add deprecation dates, and add missing models

* fix(sambanova): sync model_prices_and_context_window_backup.json with primary

* fix(sambanova): fix indentation on Meta-Llama-3.2-1B-Instruct deprecation_date

* fix(bedrock): add amazon.titan-embed-g1-text-02 to unmapped model error message

* style: apply black formatting to embedding.py

* fix(sambanova): correct indentation on DeepSeek-V3.2 entry

* fix(sambanova): replace gemma-3-12b-it with gemma-4-31B-it (verified pricing)

* fix(utils): preserve arbitrary above-threshold tiered pricing keys in get_model_info (#30880)

* fix(utils): preserve arbitrary above-threshold tiered pricing keys in get_model_info

get_model_info rebuilt ModelInfo by copying a fixed allow-list of
input/output_cost_per_token_above_<N>_tokens keys (128k/200k/272k/512k), so any other
threshold a user registered was dropped before reaching _get_token_base_cost, which already
reads an arbitrary threshold out of the key name. Custom tiers such as above_500k_tokens were
silently ignored and billing fell back to the base per-token rate. Carry over any
_above_<N>_tokens cost key present on the source cost-map entry that the fixed fields miss

Fixes #30344

* test(cost): keep suite hermetic by popping the temp tiered-pricing model

Wrap the regression body in try/finally so litellm.model_cost no longer
leaks the litellm-test-non-standard-tier entry into later tests that
iterate or reset the global cost map. Addresses Greptile review thread.

* fix: resolve UP045 lint violations (Optional[X] -> X | None)

Convert Optional[X] type annotations to X | None syntax across rerank
transformations, spend tracking, and other modules to satisfy ruff strict gate.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix: run black formatting on UP045-fixed files

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix: remove unused Optional imports after UP045 migration

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix: black format cold_storage_handler.py

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix(ci): correct OSS staging branch name in guard-main-branch errors

Co-authored-by: Cursor <[email protected]>

* fix: strip trailing zeros from M/B spend formatter

* fix: address focus and streaming edge cases

* feat: add LAR-1 semantic routing strategy

Optional router strategy that picks a deployment tier from
request_kwargs.metadata.lar1 (confidence, evidence, time). Deployments
are tagged with model_info.type (cloud-smart, cloud-fast, local, deep).
Thresholds are configurable via routing_strategy_args. Includes 30 unit
tests and an Ollama example config.

Co-authored-by: Cursor <[email protected]>

* fix(mavvrik): advance metricsMarker on empty-content deliver

When deliver() receives empty content (no spend data for a date), it now
registers with Mavvrik and PATCHes the metricsMarker before returning
instead of short-circuiting. Dates with zero spend no longer stall marker
advancement, preventing unnecessary catch-up API calls on subsequent runs.

* style: black format mavvrik_destination

* fix: handle empty mavvrik exports and lar1 reset

* test: add regression test for _reset_custom_routing_strategy

* fix(test): mock async destination.deliver in mavvrik export window test

* style: ruff format spend_management_endpoints after merge

* fix(router): apply LAR-1 strategy atomically so invalid thresholds don't leave partial state

apply_lar1_routing_strategy set router.routing_strategy to "lar1" before
constructing LAR1RoutingStrategy, whose __init__ validates thresholds via
_normalize_thresholds and raises on a misconfigured (out-of-order or
out-of-range) set. On a live update_settings call with bad thresholds the
router was left advertising routing_strategy="lar1" with no custom selector
bound, while the previous strategy's selectors stayed registered.

Build (and validate) the strategy before mutating any router state, so a
threshold error leaves the router exactly as it was. Add a regression test
that asserts a failed switch keeps the prior strategy intact.

---------

Signed-off-by: David J. M. Karlsen <[email protected]>
Co-authored-by: Bytechoreographer <[email protected]>
Co-authored-by: Claude Opus 4 (1M context) <[email protected]>
Co-authored-by: Rick <[email protected]>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: shin-berri <[email protected]>
Co-authored-by: yuneng-jiang <[email protected]>
Co-authored-by: Yassin Kortam <[email protected]>
Co-authored-by: mateo-berri <[email protected]>
Co-authored-by: Krrish Dholakia <[email protected]>
Co-authored-by: xbrxr03 <[email protected]>
Co-authored-by: hayden <[email protected]>
Co-authored-by: Kent <[email protected]>
Co-authored-by: Wassim Badraoui <[email protected]>
Co-authored-by: Wassbdr <[email protected]>
Co-authored-by: Neimar Avila <[email protected]>
Co-authored-by: Neimar Avila <[email protected]>
Co-authored-by: Jerry-Scintilla <[email protected]>
Co-authored-by: AlexBGoode <[email protected]>
Co-authored-by: Carsten Boloz <[email protected]>
Co-authored-by: jesco <[email protected]>
Co-authored-by: Praveen Ghuge <[email protected]>
Co-authored-by: Jim Smith <[email protected]>
Co-authored-by: David J. M. Karlsen <[email protected]>
Co-authored-by: Vedant Agarwal <[email protected]>
Co-authored-by: Srivatsa Kamballa <[email protected]>
Co-authored-by: Ahmad Shahzad <[email protected]>
Co-authored-by: Jeremy Chapeau <[email protected]>
Co-authored-by: KRISH SONI <[email protected]>
Co-authored-by: Ayush Shekhar <[email protected]>
Co-authored-by: dav nguyxn <[email protected]>
Co-authored-by: Tal Marian <[email protected]>
Co-authored-by: Hemant K <[email protected]>
Co-authored-by: Cursor <[email protected]>
Co-authored-by: Yash Raj Pandey <[email protected]>
Co-authored-by: Zang Peiyu <[email protected]>
Co-authored-by: bhumikadangayach <[email protected]>
Co-authored-by: Ewertonslv <[email protected]>
Co-authored-by: carlsonchik <[email protected]>

* test(interactions): drop role from Interaction output fields to match Google spec (#30986)

Google moved the role field off the top-level Interaction response schema onto the per-turn Turn schema, so the OpenAPI compliance canary test_interaction_response_fields started failing on main with "Output field 'role' not in spec". role on Turn is already asserted by test_turn_schema, so dropping it from the Interaction output_fields list realigns the test with the live spec without losing coverage.

* fix(mcp): stop exposing MCP server URLs on the AI Hub and public hub API (#30902)

The AI Hub MCP Hub listed each MCP server's upstream URL in a table column and in the server Details modal, on both the authenticated dashboard and the public hub. The unauthenticated GET /public/mcp_hub endpoint also returned the url field via MCPPublicServer, so the upstream address was readable by any client even with the column gone. These surfaces are for end users discovering available servers, so the gateway-internal endpoint should not be exposed there.

Drop the URL column from both MCP Hub tables and the URL field from both detail modals, and remove url from MCPPublicServer so /public/mcp_hub no longer serialises it; schema.d.ts is updated to match. The public hub MCPServerData interface no longer declares url since the response omits it. Admin surfaces that configure the endpoint (the MCP server management page, the submissions review tab, the make-public form) and the authenticated /v1/mcp/server endpoint are untouched.

publicMCPHubColumns is lifted to a module-level export so both hub column sets get a mutation-killing regression test, public_model_hub.test.tsx covers the details modal hiding the url, and test_public_endpoints.py asserts /public/mcp_hub never returns url even when the server has one.

* perf(otel): resolve LITELLM_OTEL_V2 flag once instead of rebuilding settings per call (#30989)

is_otel_v2_enabled() constructed a pydantic-settings model (_OTelV2Flag) on every
call, which re-scans the process environment and costs ~28us. The flag is read
multiple times along the proxy request hot path (auth, logging-callback setup,
proxy_server), so the cost compounded into a measurable per-request CPU overhead
and a throughput regression visible from v1.87.3 onward.

The flag is a process-level setting that is fixed at startup, so resolve it once
with lru_cache. Caching it alone restores throughput to the pre-regression
baseline in load tests. Tests that toggle the env now call cache_clear().

* fix(ui): stop per-model usage export from duplicating user spend across models (#30980)

The "day-by-day by user and model" usage export overcounted spend by
repeating each user-day total once per model. generateDailyWithModelsData
iterated day.breakdown.models crossed with the entity's own
api_key_breakdown and added the api key's full daily spend to every model,
leaving the per-model object (modelData) unused and never matching the api
key to the model. A user who called N models got N identical rows, so the
column summed to N times the real spend; one reported export totaled
$167,953 against a true $21,353 (7.87x).

Attribute each model only the spend of the keys that actually used it by
intersecting the entity's api keys with modelData.api_key_breakdown. This
mirrors the already-correct pattern in activity_metrics.tsx, so per-model
rows now sum back to the user-day total and models a user never called no
longer appear.

The existing tests passed only because their mocks omitted
models[*].api_key_breakdown and never asserted numeric values. The mocks
now match the real /user/daily/activity payload, and new tests assert that
per-model spend sums to the user-day total and that uncalled models are
omitted; both fail on the old code.

Resolves LIT-3866

* fix: reject model_list in proxy body and gate advisor client credentials (#30585)

* fix: validate proxy request body and nested fields

Ensure caller-supplied request fields cannot override server-side deployment
configuration, and apply request-body validation consistently to nested
structures. Adjusts router kwarg handling and client-side credential handling
for base-url overrides

* test: cover router strip ordering and advisor clientside credential gate

* fix: clear deployment credentials on client base-url override

When a request overrides api_base/base_url, recompute the deployment's
litellm_params (clearing the deployment's own api_key) and drop the cached
client built for the original endpoint, so the deployment credential is not
reused for the client-supplied endpoint. Adds regression tests that assert the
credentials actually forwarded to litellm.completion/acompletion.

* fix(proxy): require api_key alongside api_base override

A request that overrides api_base/base_url but supplies no api_key still
left the proxy carrying a server credential: once the override clears the
banned-param opt-in, the provider re-resolves a key from the environment
(api_key or get_secret("OPENAI_API_KEY") and ~30 sibling chains in
main.py) and forwards it to the caller-controlled URL. Popping the
deployment api_key only changed which server key leaked.

Gate is_request_body_safe so a permitted api_base/base_url override must
also carry a non-empty caller api_key; reject otherwise. The env
resolution in main.py is left as the provider boundary.

* fix(proxy): extend request-body banlist with five additional credential and session targeting fields

Yuneng's review found five deployment-owned request-body params still missing
from the denylist and the router strip set. Each lets a caller reach the
operator's provider credentials or retarget the outbound request:
aws_profile_name selects a local AWS profile, oci_compartment_id and oci_region
retarget the OCI request, litellm_credential_name selects any server-loaded
credential by name with no ownership check, and runtimeSessionId resumes a
Bedrock AgentCore runtime session (AWS does not enforce session-to-user
mapping, so this is a cross-tenant session-resume vector).

Add all five to _BANNED_REQUEST_BODY_PARAMS in auth_utils.py and to
_DEPLOYMENT_OWNED_CREDENTIAL_KWARGS in router.py. Deployment litellm_params and
SDK direct calls are unaffected: the banlist gates the request body only, and
the router strip drops caller kwargs, never deployment["litellm_params"].

* test: rename arbitrary canary values in security tests to neutral placeholders

* fix(proxy): apply api_key co-presence to nested base override and warn on Router credential strip

P1-A: is_request_body_safe descended into _NESTED_CONFIG_KEYS
(litellm_embedding_config, extra_body) for the banned-param check but not for
the api_key co-presence check, so a base override smuggled into one of those
nested dicts cleared the client-side-credentials opt-in without a paired
api_key and let the provider re-resolve a server credential from the
environment. Run _check_base_override_has_api_key on each nested config dict
too, so the requirement applies wherever a base override is permitted.

P1-B: the deployment-owned credential strip in the Router runs unconditionally
on every _completion/_acompletion, which is security-correct but silently
drops per-call api_version/vertex_project/etc. for SDK Router callers. Emit a
single warning (key names only, never values) when the strip removes a
non-empty value, so the backwards-incompatible behavior is visible without
gating the strip on a context flag that does not exist.

* fix(proxy): apply api_key co-presence to tool-entry base override

is_request_body_safe scans three surfaces (root, _NESTED_CONFIG_KEYS, and
tools[]); the previous commit extended the api_key co-presence rule to root
and nested config dicts but not to tool entries. With
allow_client_side_credentials enabled, a tool entry carrying api_base/base_url
and no paired api_key cleared the gate, letting a provider interceptor fall
back to a server-side credential for a caller-controlled URL. Add the same
_check_base_override_has_api_key call to each tool dict and its nested function
dict, mirroring the symmetry already applied to the nested config keys. The
rule is unchanged: api_key must live in the same dict as the base override it
accompanies.

* test(proxy/auth): require paired api_key under extra_body opt-in

* fix(router): gate deployment-owned kwarg strip on litellm.proxy_is_running

* fix(advisor): narrow proxy-import guard to ImportError-family

* fix(router): gate api_key clear on base override behind litellm.proxy_is_running

* test(proxy/auth): scope proxy_is_running flag to dynamic-params class with autouse fixture

* style: use built-in generics in PR-added type annotations

* revert: drop proxy_is_running flag and router-level credential strip; rely on proxy gate

* revert: scope PR to LIT-3828 + LIT-3834 only; drop LIT-3830/LIT-3833 changes

* style: black-format advisor orchestration test

* feat(scim): drive global proxy role from a SCIM admin group (#30895)

Adds an optional litellm_settings.scim_admin_group. When configured, the
global proxy role is recomputed from a user's resulting groups on every SCIM
write that can change membership: user create/PUT/PATCH and group
create/PUT/PATCH/DELETE, plus the existing-email upsert path. Membership in
the admin group grants PROXY_ADMIN and its absence demotes to the non-admin
default, enabling just-in-time elevation and automatic demotion without a
re-login. When the setting is unset the role is never touched, so current
behavior is preserved and a misconfigured IdP can never unexpectedly grant
admin.

* feat(scim): ingest enterprise extension attributes into user metadata (#30893)

Map the SCIM enterprise extension block
(urn:ietf:params:scim:schemas:extension:enterprise:2.0:User) onto SCIMUser
so create and PUT persist employeeNumber, costCenter, organization,
division, department, and manager into LiteLLM_UserTable.metadata under
scim_enterprise, and round-trip them back out on read. This lets financial
reporting group spend by fields like cost center and department.

The enterprise block holds directory-only HR attributes, so it is kept out
of the generic user management responses (/user/info, /v2/user/info, and
/user/list), which non-proxy-admin callers such as team and org admins can
use to read other users. The data still lands in metadata for reporting and
still round-trips through the SCIM read endpoints, which build their response
from the user row directly.

Resolves LIT-3617

* fix(ui): resolve user_id to email in Spend Per User usage chart (#30992)

The Usage dashboard "Spend Per User" chart rendered raw UUIDs (and
default_user_id) instead of emails. /user/daily/activity passed
entity_metadata_field=None, so every user entity in the breakdown
carried empty metadata; the chart could only fall back to the user_id.
The frontend resolved labels from a separately paginated user list, so
any spender not on a loaded page showed as a UUID.

Resolve the email/alias for the user_ids actually on the page (mirroring
how api key metadata is already resolved) and attach it to the entity
metadata, so the chart labels each spender with their email and falls
back to the UUID only when no email is on file. get_daily_activity gains
an optional resolve_entity_metadata hook so the user endpoint can do this
page-scoped lookup without loading the whole user table.

Resolves LIT-3889

* fix: prevent key-level metadata.tags from leaking into Bedrock passthrough body (#30985)

* fix: prevent key-level metadata.tags from leaking into Bedrock passthrough body

* test: cover bedrock key-tag litellm_metadata pre-seed in common_checks

Add a regression test asserting key-level tags on a bedrock passthrough
request land in litellm_metadata and never leak into the provider-facing
metadata field, which closes the codecov/patch gap on the auth_checks
pre-seed line. Also drop the now-stale comment that hardcoded
metadata["headers"]; the headers are written under whichever metadata
field _get_metadata_variable_name selects.

* refactor(auth): pre-seed litellm_metadata from LITELLM_METADATA_ROUTES

The auth-time pre-seed in common_checks hardcoded "bedrock", so any other
route later added to LITELLM_METADATA_ROUTES would reintroduce GH#30629 (key
tags leaking into the provider-facing metadata field) without a matching
update here. Key off the shared constant instead, and extend the regression
test to cover a non-bedrock metadata route so the route-agnostic behavior is
locked in.

* fix(auth): pre-seed litellm_metadata before header-tag merge

apply_client_tag_policy_pre_auth runs in user_api_key_auth.py before
common_checks, so it resolved get_metadata_variable_name_from_kwargs to
'metadata' (litellm_metadata was not yet present). common_checks then
pre-seeded litellm_metadata on LITELLM_METADATA_ROUTES, after which
apply_key_tags_pre_auth and _tag_max_budget_check both targeted
litellm_metadata, leaving header tags stranded in metadata and invisible
to per-tag budget enforcement on Bedrock and other matching routes.

Extract the pre-seed into LiteLLMProxyRequestSetup.pre_seed_litellm_metadata_for_route
and invoke it before apply_client_tag_policy_pre_auth so all tag merges
and the budget-check read agree on the same metadata key.

* test(auth): guard early litellm_metadata pre-seed wiring for header tags

Bugbot's autofix added a pre-seed of litellm_metadata in
_run_centralized_common_checks before apply_client_tag_policy_pre_auth, so
x-litellm-tags header tags land in litellm_metadata and stay visible to
_tag_max_budget_check on LITELLM_METADATA_ROUTES. Its test replayed that call
order in the test body, so removing the production call site still passed.

Add a wiring-level regression that drives the real _run_centralized_common_checks
and asserts header tags land in litellm_metadata (not metadata) for bedrock and
/v1/messages. Dropping the pre-seed call site now fails the test.

---------

Co-authored-by: Zang Peiyu <[email protected]>
Co-authored-by: Cursor Agent <[email protected]>

* fix(proxy): scope team BYOK models by key team_id in /model/info (#31009)

GET /model/info returned an empty list for a team key whose team only has team-scoped BYOK deployments, even though /v1/models and a master key both returned them. _get_caller_byok_team_scope resolved the caller's allowed teams only from user_api_key_dict.user_id and the bound user's team memberships. A team or service key has user_id=None, so the helper returned an empty set and _byok_row_outside_caller_teams then dropped every team BYOK row

This includes the key's own team_id in the allowed-team set across every non-admin branch, since a team key is authoritatively scoped to its team regardless of whether a bound user is resolvable or a formal member of that team

Completes the work in #30025, which aligned /v1/model/info with router deployments but missed the team-key case in the scope helper it introduced

* fix(bedrock): only expand config-sourced AWS credential references (#30867)

AWS auth parameters in the Bedrock and SageMaker path could be expanded against
the process environment when credentials were built. Config-sourced references
are already expanded at load time, so restrict expansion to that path: a
reference still present at request time is treated as caller-supplied input and
is left as-is, and the web-identity helper rejects environment-variable
references before resolving the token.

Also rework the ambient AWS_* fallback as a single pass that pairs each value
with its own env-var name, fixing a latent index misalignment that left
AWS_EXTERNAL_ID unresolved.

Adds regression tests covering the resolution behavior.

* feat(mcp): scope a key to zero MCP servers with no-mcp-servers sentinel (#31029)

* feat(mcp): scope a key to zero MCP servers with no-mcp-servers sentinel

A key under a team that has MCP servers had no way to opt out of them;
an empty list has always meant "inherit the team". This adds a
no-mcp-servers sentinel (mirroring no-default-models for models) so a key
can declare an explicit zero that overrides team inheritance, additive
grants, and allow_all_keys servers, surfaced as an exclusive "No MCP
Servers" option in the key create/edit UI.

* refactor(ui): centralize no-mcp-servers sentinel in a shared constant

The sentinel string was defined under two different local names and
inlined in two more files; a single exported constant removes the drift
risk flagged in review.

* fix(mcp): enforce no-mcp-servers sentinel on toolset-scoped routes

Toolset scoping replaced a key's mcp_servers with the toolset's servers,
dropping the no-mcp-servers sentinel, so a key opted out of all MCP could
still execute a granted toolset's tools via /toolset/{name}/mcp. Deny
toolset access when the key carries the sentinel, checked before the admin
branch to match get_allowed_mcp_servers.

* feat(ui): add Amazon Bedrock Mantle to the Add Model provider dropdown (#31034)

The Add Model provider dropdown is driven by provider_create_fields.json
(served at /public/providers/fields), and Bedrock Mantle had no entry, so it
could not be selected even though the backend provider, its models, and the UI
enum/logo mappings already existed.

Add a bedrock_mantle entry exposing the credential fields the provider actually
honors: an optional bearer api_key for BYOK, the AWS SigV4 chain, a region, and
an api_base override. Selecting it now populates the bedrock_mantle models from
the cost map via the existing getProviderModels filter.

Also resolve the provider logo when getProviderLogoAndName is given the enum key
(e.g. BedrockMantle) rather than the slug, which the dropdown passes; previously
only slugs that lowercase-matched their key (like bedrock) resolved a logo.

* feat(proxy): allow llm_api_routes virtual keys to list MCP tools via /v1/mcp/tools (#31031)

* feat(proxy): allow llm_api_routes virtual keys to list MCP tools via /v1/mcp/tools

GET /v1/mcp/tools returns the MCP tools available to the calling key, the same
data already exposed through /mcp/tools/list and /mcp-rest/tools/list, both of
which are in llm_api_routes. The /v1/mcp/tools path was in no route group, so
virtual keys created from the UI (which default to allowed_routes=["llm_api_routes"])
got a 403 listing tools one way but not the other.

Add it to mcp_inference_routes. Unlike /v1/mcp/server, this path has no
management write counterpart, so it does not need the method-aware carve-out
used for server discovery.

* test(proxy): parametrize MCP inference route check over the full endpoint set

* fix(ui): label request logs column "Key Alias" to match filter (#31037)

The request logs table column displayed "Key Name" while its accessor
(metadata.user_api_key_alias) and the corresponding filter both use the
"Key Alias" label; this aligns the column header with that naming.

* test(ui): scrub stale return-url cookie from e2e storageState (#30317)

The login flow stores a post-login return URL in the litellm_return_url
cookie (5 minute TTL). globalSetup snapshots cookies into the per-role
storageState that every spec reuses, so when the snapshot races ahead
of the app consuming that cookie, each test inheriting it gets
redirected to the stale URL (/ui/?login=success) mid-assertion the
first time it mounts a page. That one rogue navigation is behind the
recurring e2e failures whose call logs all show "navigated to
/ui/?login=success" while waiting for an element; which specs die
varies run to run with snapshot timing. Clear the cookie right before
saving the snapshot so no test starts with a pending redirect.

* docs: add MCP server change guidelines (#31038)

* docs: add MCP server change guidelines

* Add outbound_credentials directory and update AGENTS.md

Updated AGENTS.md to include new outbound_credentials directory and its files, along with additional comments on existing files.

---------

Co-authored-by: tin-berri <[email protected]>

* fix(passthrough,streaming): recover cost on interrupted and agentic Anthropic streams (#31035)

Streaming and pass-through requests could be logged with $0 cost or dropped from
SpendLogs entirely while the upstream provider still billed every token. This
closes the leak paths not already covered by #30160, #30787 and #30788.

- Catch a stream_chunk_builder raise in the core CustomStreamWrapper (sync and
  async). Large agentic tool-use / thinking streams can make assembly re-raise
  as APIError from inside the except-StopIteration handler, where the sibling
  except does not catch it, so it escaped __next__/__anext__ and dropped the
  request; recover best-effort usage from the raw chunks instead
- Add a usage-only fallback for Anthropic streaming pass-through: when
  stream_chunk_builder returns None or raises, rebuild usage from the
  message_start / message_delta SSE events via AnthropicConfig.calculate_usage so
  cache, web-search and geo tokens are priced instead of left at $0
- Decode buffered pass-through bytes with errors="replace" so a stream cut
  mid-multibyte-sequence still logs the usage events already received
- Record response_cost into model_call_details on the pass-through success path
  (it is read from there, not from kwargs), matching the gemini/cohere/openai
  handlers
- Name the key (alias + masked key) in the virtual-key BudgetExceededError so
  operators don't have to reverse-map spend back to a key

* fix(ui): clarify OpenAI-compatible provider dropdown labels (chat vs legacy completions) (#31046)

* fix(ui): clarify OpenAI-compatible provider dropdown labels (chat vs legacy completions)

* style: change labeling to be clearer

* fix(proxy): serialize team budget_limits to JSON in jsonify_team_object (#31045)

POST /team/new with any budget_limits returned 500 because
jsonify_team_object serialized members_with_roles but left budget_limits
as a raw Python list, which Prisma's Json column rejects. /team/update
and /key/generate worked only because each json.dumps the windows itself.
Serialize budget_limits in the shared helper, guarded by isinstance(list)
so the pre-serialized /team/update path is unaffected.

* fix(realtime): stop revalidating realtime events at the logging boundary (#31054)

Realtime websocket sessions emit events outside the OpenAIRealtimeEvents
union (e.g. rate_limits.updated, response.function_call_arguments.delta,
surfaced when logged_real_time_event_types="*"). Building
LiteLLMRealtimeStreamLoggingObject revalidated every stored event against
the 16-member union, producing thousands of ValidationErrors per session
(12,670 for a ~281-event session). That synchronous work blocked the
asyncio event loop, degrading realtime time-to-first-audio and dial latency
and tripping readiness probes, and the raised error discarded the session
usage so no cost was tracked.

Type results as SkipValidation[OpenAIRealtimeStreamList] and serialize the
events verbatim, so already-formed event dicts are not revalidated. The
flood drops from 12,670 errors to 0 and the combined usage survives to the
cost calculator.

Resolves LIT-3919
Resolves LIT-3920

* feat(mcp): scaffold outbound_credentials package with typed Result (#31047)

* feat(mcp): scaffold outbound_credentials package with typed Result

PR1 of the MCP v2 outbound-credential migration. Adds the
litellm/proxy/_experimental/mcp_server/outbound_credentials/ subpackage with a
hand-rolled Ok | Error Result union (pure stdlib + typing_extensions, no new
dependency) and its package surface. Nothing imports this on a live request path
yet, so production behavior is unchanged; later PRs add the typed config
vocabulary, the resolve_credentials dispatch, and the v1 graft.

* feat(mcp): add outbound_credentials typed vocabulary (#31049)

PR2 of the MCP v2 outbound-credential migration, stacked on the result.py
scaffolding. Adds types.py (the AuthConfig discriminated union over seven frozen
per-mode configs, CredError as an expression @tagged_union, Subject, ServerSpec,
and the parse_auth_spec_kind boundary parser) and httpx_auth.py (NoOpAuth,
StaticHeaderAuth). Pulls in expression>=5.6.0,<6.0 on the proxy extra for the
tagged union. Construction-time tests prove illegal mode/field combinations are
rejected. Nothing is wired onto a request path yet; the resolver lands next.

* fix(router): isolate all per-deployment pricing overrides from sibling deployments (#31021)

* fix(router): isolate all per-deployment pricing overrides from sibling deployments

CustomPricingLiteLLMParams is the authoritative set of per-deployment pricing
fields, used to strip overrides from the shared backend-alias key so one
deployment cannot pollute a sibling that shares the same backend model. It had
drifted from ModelInfoBase: tiered and per-unit cost fields such as
input_cost_per_token_above_272k_tokens, cache_read_input_token_cost_above_*,
output_vector_size, ocr_cost_per_*, and the regional uplift multipliers were
absent, so a deployment overriding any of them leaked the override into
litellm.model_cost under the shared key and every sibling read the wrong rate
via /model/info (LIT-3897).

Add the missing fields so the denylist covers every ModelInfoBase pricing
field, and guard against future drift with a test asserting the two stay in
sync, plus a regression test that a tiered override stays isolated to its own
deployment model_id key.

* chore(ui): regenerate schema.d.ts for custom pricing fields

* fix(mcp): stop auth failures on the /mcp path surfacing as cancelled tool calls (#31011)

user_api_key_auth raises ProxyException, not HTTPException, on an auth failure.
The streamable-HTTP and SSE MCP handlers only re-raised HTTPException to preserve
status and headers, so a ProxyException fell through to the catch-all and was
flattened to a generic 500, dropping the real status (for example 401) and any
WWW-Authenticate challenge. MCP clients render a 500 on the JSON-RPC POST as a
cancelled or terminated session, and an OAuth client never receives the 401 it
needs to re-authenticate. Because auth runs before server routing, one rejected
credential fails every targeted server at once.

Map ProxyException back to its real status and headers in both handlers
(handle_streamable_http_mcp, handle_sse_mcp) via a small
_proxy_exception_to_http_exception helper inserted before the generic
except Exception. A genuine auth failure now returns its real status; a key sent
without the documented Bearer prefix gets a clear 401 telling the caller to fix
the header rather than a cancelled session.

Regression tests assert that a ProxyException(401) raised during auth propagates
as a 401 with WWW-Authenticate from both the streamable-HTTP and SSE handlers,
and unit-test the converter for the 401/403/non-numeric-code cases.

* refactor(completion): extract provider dispatch into typed helpers so basedpyright can analyze it (#30813)

* chore: litellm oss staging (#30968)

* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens (#29693)

* fix: correct amazon.titan-embed-text-v2 input price to $0.02/1M tokens

* test: scope local cost map env var with monkeypatch to avoid test pollution

* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold (#30764)

* fix(sensitive_data_masker): fully mask secrets at or below the reveal threshold

_mask_value did partial reveal by showing the first visible_prefix and last
visible_suffix characters, but for a value whose length was at or below
visible_prefix + visible_suffix (8 by default) it returned the value verbatim.
A value of exactly 8 chars fell through the length guard and computed
masked_length == 0, reconstructing the original string with no mask characters;
anything shorter hit the early return. Either way short credentials were emitted
in plaintext.

mask_dict routes real secrets through this path, so an 8-char-or-shorter redis
password, api key, or token could be written to logs and the UI unmasked. The
sibling helper mask_sensitive_keys already guards this case; _mask_value now does
the same by fully masking any value at or below the threshold.

* fix(sensitive_data_masker): add mask_short_values opt-out for truncation callers

Fully masking short values is the right default for secret masking, but
CooldownCache reuses the masker purely to truncate exception messages to the
first 50 characters, and it relies on short messages being returned readable.
Masking those blanked out short exception text and broke its tests.

Add a mask_short_values flag (default True, secure) and have CooldownCache pass
False so it keeps the truncation behavior, while every secret-masking caller
still gets short values fully masked.

* fix(mcp_debug): opt out of short-value masking to keep diagnostic token preview

MCPDebug uses the masker to preview auth tokens in debug headers and documents
that values of 10 chars or fewer are shown unchanged so token types stay
distinguishable. Pass mask_short_values=False so that diagnostic behavior is
preserved while secret maskers keep masking short values.

* fix(mcp_debug): mask short auth values in debug headers instead of echoing them

Earlier this masker opted out of short-value masking to keep a token preview, but
that echoes short authorization and token values verbatim in debug response
headers, which is the same leak this change is meant to close. Auth material
should never be emitted in full, so mask short values here too; the first/last
character preview still applies to longer tokens. Only CooldownCache keeps the
opt-out, since it truncates exception text rather than masking secrets.

* test(mcp_debug): assert masked short value preserves length

* refactor(fireworks_ai): remove deprecated audio transcriptions endpoint (#30917)

Fireworks AI deprecated audio inference on 2026-06-10
(https://docs.fireworks.ai/updates/changelog#audio-inference-and-image-generation-deprecation).
Live API testing confirms the endpoint is already non-functional: a valid
Fireworks API key receives HTTP 401 "Unauthorized" from
api.fireworks.ai/inference/v1/audio/transcriptions for every request,
regardless of payload. The audio-prod.api.fireworks.ai host referenced in
the test suite returns 401 for every path; the entire host is decommissioned.

Remove the dead FireworksAIAudioTranscriptionConfig class and every
reference to it across the codebase:

- Delete litellm/llms/fireworks_ai/audio_transcription/ directory (17-line
  config class that inherited from OpenAIWhisperAudioTranscriptionConfig)
- Remove the Fireworks branch from
  ProviderConfigManager.get_provider_audio_transcription_config() in
  litellm/utils.py; update the stale comment in
  get_optional_params_transcription that referenced fireworks ai
- Remove the FireworksAIAudioTranscriptionConfig entries from
  LLM_CONFIG_NAMES and _LLM_CONFIGS_IMPORT_MAP in
  litellm/_lazy_imports_registry.py
- Remove the TYPE_CHECKING re-export in litellm/__init__.py
- Remove the transcription branch in the fireworks_ai case of
  get_supported_openai_params() in
  litellm/litellm_core_utils/get_supported_openai_params.py
- Remove the whisper-v3 and whisper-v3-turbo entries from
  model_prices_and_context_window.json and
  litellm/model_prices_and_context_window_backup.json (both had
  mode: audio_transcription and zero-cost pricing)
- Remove the TestFireworksAIAudioTranscription test class and its
  imports from tests/llm_translation/test_fireworks_ai_translation.py

No other provider is affected. The openai_compatible_providers list,
FireworksAIMixin, and the OpenAI Whisper transcription handler all stay
because they are shared with other Fireworks endpoints and other
providers. The provider_endpoints_support.json registry already had
audio_transcriptions set to false for fireworks_ai.

* feat: add darkbloom provider (#30876)

* feat: add darkbloom provider

* fix: document darkbloom provider endpoints

* fix: address darkbloom review feedback

* fix: update darkbloom tool metadata

* fix: fail fast for non-Postgres database URLs (#30883)

* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup

LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.

Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.

Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.

Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.

Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.

* fix: resolve CI failures and proxy DB URL typing issue

* fix(proxy): fail fast on non-PostgreSQL DATABASE_URLs with clear startup errors instead of hanging

* Validate DIRECT_URL alongside DATABASE_URL startup guards

* fix(bedrock): surface modeled HTTP status for mid-stream error events so 5xx is retryable (#24608) (#30946)

* fix(bedrock): surface modeled HTTP status for mid-stream error events (#24608)

* test(bedrock): mid-stream server errors trigger streaming fallback (#24608)

* style(bedrock): black-format stream-error helper (#24608)

* fix(mcp): re-land native tool preservation with typed annotations (#30645)

* fix(mcp): preserve native tools in semantic filter hook with typed annotations

* fix(mcp): tighten _is_mcp_tool Chat Completions shape check

* fix(sambanova): return embeddings supported params instead of dropping them (#30937)

* fix(router): send fallback metadata when streaming (#30914)

When a streaming request triggers a fallback, there was previously no way to
know it happened. This commit addresses this in a few ways:

1. The response now correctly populates the fallback headers
    (`x-litellm-attempted-fallbacks`) so callers know a fallback happened.
2. The correct model ID is passed in the streaming chunks.
3. A streaming chunk with the fallback error can be optionally sent back
    to the client (opt-in) by passing `include_fallback_errors: true` in
    the request.

The format of the fallback errors while streaming is intentionally OpenAI
compatible to not break existing libraries that parse these events. It was
tested with Vercel's AI SDK (ai-sdk.dev). It is also opt-in, so it is not
delieved unexpectedly to callers by default.

* fix(mistral): drop output-only reasoning fields from input messages (#30884)

LiteLLM attaches reasoning_content and thinking_blocks to assistant
responses. Replaying those assistant turns verbatim forwarded the fields
back to Mistral, whose input schema forbids unknown keys, so the whole
request failed with a 422 extra_forbidden and reasoning models became
unusable across multiple turns.

Strip both fields from assistant messages before the request is built, in
a spot that runs ahead of the image/file branch so it applies on every
path. Fixes #30835

Co-authored-by: Cursor <[email protected]>

* fix(perplexity): bill search queries at the per-request price, not 1/1000 of it (#30652)

* fix(perplexity): bill search queries at the per-request price, not 1/1000

The fallback cost calculator divided search_context_cost_per_query by
1000, but that field stores the per-request price in USD: sonar is
{low: 0.005, medium: 0.008, high: 0.012}, matching Perplexity's published
$5/$8/$12 per 1,000 requests expressed per request. The gemini cost
calculator reads the same field per request with no division (its
docstring calls it "the per-request cost").

The division understated search cost by 1000x on every Perplexity call
that falls back to manual calculation (i.e. when the API does not return
a pre-computed usage.cost). Use the value directly.

Update the tests that had encoded the /1000 factor in their expectations,
and drop an unused import flagged by ruff in the touched test file.

* test(perplexity): update integration test search-cost expectations to per-request

The integration tests still encoded the old /1000 search-cost factor, so
they failed once the fallback calculator was corrected to bill
search_context_cost_per_query per request. Update the four expected-cost
computations (and the high-volume dollar-value comments) to match.

* test(perplexity): drop unused mock imports flagged by ruff

* fix: include model_access_groups when expanding all-team-models in get_team_models (#30622)

* fix(fireworks_ai): return None for transcription in get_supported_openai_params

Fireworks AI deprecated audio inference on 2026-06-10; the endpoint is
decommissioned. Without an explicit transcription branch, requests with
request_type='transcription' fell through to the else and returned
FireworksAIConfig chat-completion params. Return None instead to signal
the provider does not support transcription.

* fix(proxy): gate include_fallback_errors behind expose_fallback_errors_to_caller setting

Without an operator gate, any authenticated caller could set include_fallback_errors=True,
trigger a fallback, and read raw upstream exception messages from the
x-litellm-fallback-errors header and the litellm-fallback-metadata SSE event.

Strip include_fallback_errors from request data in common_processing_pre_call_logic
when expose_fallback_errors_to_caller is not set, so the router never builds the
error list. Also gate _should_include_fallback_errors on the same setting as a
secondary check for the streaming SSE injection path.

* test(proxy): opt in to expose_fallback_errors_to_caller in streaming SSE test

The operator gate added in e7ff3e1 means include_fallback_errors is only
honoured when general_settings.expose_fallback_errors_to_caller is True.
Set that flag via monkeypatch in the test that exercises the emit path.

* test(prompt_templates): make test_convert_url hermetic instead of hitting picsum.photos

test_convert_url called convert_url_to_base64 against a live picsum.photos
URL and asserted nothing, so it added no real signal and broke CI whenever
the host was unreachable (it was returning 522 and blocking this branch).
Replace the live call with a mocked HTTP client and assert the produced
base64 data URL, so the conversion path is exercised deterministically with
no network dependency. This suite runs under VCR, which is why a transport
level mock (respx) does not reliably intercept; mocking the client object
itself is robust regardless.

* fix(interactions): drop role from Interaction response to match Google spec

Google removed the output-only role field from the Interaction schema (it
now lives only on Turn), so the live OpenAPI compliance canary started
failing with 'role' not in spec. Reconcile our generated types by removing
role from Interaction, CreateModelInteractionParams, CreateAgentInteractionParams
and from the LiteLLM InteractionsAPIResponse/InteractionsAPIStreamingResponse,
stop stamping role=model in the responses-to-interactions transformation, and
update the compliance and integration tests accordingly. Turn.role is kept
since the spec still defines it.

* fix: align all-team-models sentinel access

* fix(router): forward include_fallback_errors through multi-hop fallbacks

run_async_fallback received include_fallback_errors as an explicit named
parameter, so it was bound out of **kwargs and never reached the nested
async_function_with_fallbacks call. Multi-hop fallback chains (a fallback
group that itself fails over) therefore stopped collecting fallback errors
beyond the first hop when a caller opted in. Re-inject the flag into kwargs
before the nested call so inner hops keep accumulating errors, which
add_fallback_headers_to_response already merges across levels.

---------

Co-authored-by: Srivatsa Kamballa <[email protected]>
Co-authored-by: Ahmad Shahzad <[email protected]>
Co-authored-by: Jeremy Chapeau <[email protected]>
Co-authored-by: KRISH SONI <[email protected]>
Co-authored-by: Kent <[email protected]>
Co-authored-by: Ayush Shekhar <[email protected]>
Co-authored-by: dav nguyxn <[email protected]>
Co-authored-by: Tal Marian <[email protected]>
Co-authored-by: Hemant K <[email protected]>
Co-authored-by: Cursor <[email protected]>
Co-authored-by: Yash Raj Pandey <[email protected]>
Co-authored-by: Zang Peiyu <[email protected]>
Co-authored-by: mateo-berri <[email protected]>

* fix(typing): bring reportReturnType back under the basedpyright budget (#31103)

* fix(realtime): post-tool-call function_response id omission (#30446)

* feat: add opensandbox sandbox provider (#31024)

* feat: add opensandbox sandbox provider

* fix: harden opensandbox sandbox startup

* fix: address opensandbox review feedback

* fix: address opensandbox sandbox review feedback

* fix: address sandbox parser nits

* fix(ci): clear opensandbox gates

* fix(review): require opensandbox api base

* chore(ci): rerun pass-through check

* feat(mcp): add resolve_credentials dispatch skeleton (#31056)

PR3 of the MCP v2 outbound-credential migration, stacked on the typed vocabulary.
Adds resolver.py: UpstreamCredentialProvider.resolve_credentials dispatches on the
declared AuthConfig variant with one arm per mode, a wildcard-free match plus an
assert_never tail so a missing arm fails basedpyright's exhaustiveness gate. Every
arm is a not_implemented stub returning a typed CredError; each mode's real body and
seam land in follow-up PRs. Pure v2, no v1 imports, nothing wired onto a request path.

* fix(router): guard num_retries=None in async_function_with_retries (#30036)

When num_retries reaches async_function_with_retries as None - e.g. a caller
passes num_retries=None explicitly (dict.get() does not fall back on an
existing None value), an auto_router/complexity_router path does not propagate
it, or Router.update_settings(num_retries=None) is used - the comparison
`if num_retries > 0:` raised:

    TypeError: '>' not supported between instances of 'NoneType' and 'int'

This only surfaced when the underlying call failed with a retryable error
(rate limit / connection / 5xx), so the real upstream error was masked by a
confusing TypeError.

Normalise an explicit num_retries=None to the router default in
_update_kwargs_before_fallbacks (falling back to 0 when the router default is
itself None, and preserving an explicit 0), and keep the matching guard at the
single pop site in async_function_with_retries as the safety net for paths that
bypass the setter. Adds regression tests to
test_router_per_deployment_num_retries.py.

Relates to #23316, #25889, #23699, #28126

Co-authored-by: Cursor <[email protected]>
Co-authored-by: yuneng-jiang <[email protected]>

* fix(ui): keep team Organization optional for proxy admins in single-org setups (#30861)

The Create Team form auto-selected, disabled, and required the Organization
field whenever exactly one organization existed, regardless of role. For a
proxy admin the organization is optional, so single-org setups could not
create a standalone team even though the field is presented as optional.

Gate the single-org preselect, the disabled state, and the restrictive help
text on the org-admin role so they apply only to org admins, who must scope a
team to their organization. Proxy admins now keep an optional, clearable,
empty organization field regardless of how many organizations exist, matching
the multi-org behavior. The /team/new endpoint already accepts a null
organization, so this was a UI-only restriction.

* feat(cloudflare): add current Workers AI text-generation models to the cost map (#31051)

* feat(cloudflare): add current Workers AI text-generation models to the cost map

The Cloudflare Workers AI list in the model cost map was badly stale, holding
only 4 ancient entries (llama-2-7b, mistral-7b-v0.1, codellama). This adds the
26 current text-generation models from Cloudflare's live
/ai/models/search?task=Text Generation catalog (GLM 5.2, gpt-oss-120b/20b,
llama 3.x/4, qwen3, deepseek-r1-distill, kimi, nemotron, and more), with
pricing derived from the catalog's per-million USD rates, context windows,
supports_function_calling, supports_reasoning, and cache_read_input_token_cost
where Cloudflare publishes cached-input pricing.

The entries are merged identically into both the root
model_prices_and_context_window.json and the bundled
litellm/model_prices_and_context_window_backup.json so the two maps stay in
sync. A regression test pins the new entries and guards against the two files
drifting for the cloudflare namespace.

* fix(cloudflare): flag llama-3.2-11b-vision as vision-capable and tidy pricing precision

llama-3.2-11b-vision-instruct is multimodal but was added without supports_vision, so LiteLLM capability checks would not surface it for image inputs. This sets supports_vision: true in both the root and backup cost maps

It also rounds the newly added Workers AI per-token prices to their intended decimal values, dropping floating-point division artifacts like 4.839999999999999e-07 in favor of 4.84e-07, applied identically to both files so the cloudflare namespace stays in sync

* test(cloudflare): pin Workers AI models against the local cost map

test_glm_5_2_entry_is_present_and_well_formed and test_additional_current_models_are_present read litellm.model_cost, which defaults to the remote map fetched from main and therefore does not yet carry the entries this PR adds, so in the misc unit shard that lookup raised KeyError. The tests now load the bundled local map through an autouse fixture (LITELLM_LOCAL_MODEL_COST_MAP plus get_model_cost_map), matching the pattern used elsewhere in the suite, so they assert against the data this PR actually ships

It also adds a regression test that llama-3.2-11b-vision-instruct carries supports_vision, and skips the root/backup comparison when the root file is absent so the suite stays green on wheel installs

* ci: make the basedpyright budget gate delta-vs-base (#31106)

* ci: re-run absolute basedpyright budget gate on push to long-lived branches

The basedpyright budget gate counts codebase-wide errors per rule against a
committed ceiling, but it only ran on pull_request against each PR's own head.
Two PRs that each pass in isolation can together push a per-rule count over its
ceiling once both merge, and nothing re-evaluated the budget on the merge
commit, so the breach only surfaced on the next PR that happened to be checked
out after the count crossed the line.

Add a push trigger on the long-lived branches and a post-merge-budget job that
re-runs the absolute gate on the merged tree, catching the accumulation on the
merge commit itself. The existing pull_request jobs are guarded so their
delta-vs-base gates don't misfire on push, where no PR base SHA exists.

* ci: shallow-fetch the post-merge-budget checkout

The post-merge-budget job only runs basedpyright over the working tree and
the committed budget file; it never inspects git history, unlike the lint
job whose delta-vs-base gates need full history. Drop its checkout from
fetch-depth: 0 to fetch-depth: 1 to avoid cloning the whole repo history.

* ci: scope post-merge-budget push trigger to long-lived branches

On a push event the branches filter matches the branch being pushed to,
not the PR target. The litellm_** glob, correct for the pull_request
filter where it matches the target branch, therefore fired the
post-merge-budget basedpyright job on every short-lived feature branch
carrying the litellm_ prefix (litellm_dev_*, litellm_add_*, and so on),
duplicating the PR lint job and burning ~10 minutes of CI per push.

Restrict the push trigger to the long-lived branches PRs actually merge
into (main, litellm_internal_staging, litellm_oss_branch), where budget
accumulation happens. The pull_request filter keeps litellm_** so PRs
targeting any long-lived branch are still linted.

* ci: make the basedpyright budget gate delta-vs-base

The basedpyright gate counted absolute codebase-wide errors per rule against a
committed ceiling and ran only on each PR's own head. Two PRs that each pass in
isolation could together push a rule past its ceiling once both merged, and
because the gate had no comparison against the base, the next unrelated PR
branched off the now-over-ceiling tree inherited a red it did nothing to cause.

Give it the same shape as the ruff strict gate: a rule fails only when its total
is both over the ceiling and higher than the count on the merge-base it merges
into. Drift already in the base is never blamed on a bystander, while any change
that actually grows a rule past the cap still fails. Head counts come from the
existing stdin pipe; the base count is a second basedpyright pass over a detached
worktree at the merge-base, reusing the head environment so import resolution
matches and no second uv sync is needed.

This obsoletes the push-triggered post-merge-budget job (and its event guards),
which only detected accumulation after the fact; the delta check blocks it on the
PR instead. Slack for reportReturnType and reportUnnecessaryComparison is raised
to give real headroom under the cap.

* refactor(ci): give the base ref its own name in type_check_gate cmd_check

cmd_check took a parameter named base that held a git ref string, then
rebound the same name to the dict of base-tree error counts returned by
base_counts. Rename the parameter to base_ref so the ref and the counts
each keep a single name and type, matching the no-reassignment style used
elsewhere; behavior is unchanged.

---------

Co-authored-by: Claude <[email protected]>

* fix(cloudflare): route native Workers AI provider through OpenAI-compatible endpoint (#31053)

* fix(cloudflare): route native Workers AI provider through OpenAI-compatible endpoint

* fix(cloudflare): guard missing account id and migrate legacy /ai/run base

Centralize the OpenAI-compatible api_base default in get_complete_url so it
is built in one place instead of being duplicated in main.py. When neither
api_base nor CLOUDFLARE_ACCOUNT_ID is set the call now fails fast with a clear
error rather than sending a request to a URL containing the literal 'None'.

An api_base still pinned to the legacy Workers AI '/ai/run' path is rewritten
to the '/ai/v1' OpenAI-compatible endpoint with a deprecation warning, so
users who hardcoded the previous default migrate gracefully instead of hitting
a silently broken '/ai/run/chat/completions' URL.

* fix(cloudflare): treat empty api_base as unset when resolving URL

* fix(cloudflare): treat empty CLOUDFLARE_ACCOUNT_ID as unset

An empty or whitespace-only CLOUDFLARE_ACCOUNT_ID slipped past the None
guard and built .../accounts//ai/v1, producing the same confusing 404 the
PR set out to prevent. Normalize the secret with normalize_nonempty_secret_str
so blank values raise the explicit missing-account-id error instead.

* feat: add chat completions code interpreter loop (#31027)

* feat: add chat code interpreter loop

* fix: address code interpreter pr checks

* fix: satisfy strict lint budget

* test: cover chat no-op interception

* fix: address code interpreter review

* fix: clean up agentic loop helpers

* fix: preserve agentic loop controls

* fix: generalize agentic loop params

* fix: carry agentic state via metadata

* fix: restore litellm params helpers

* refactor: move chat code-interpreter loop out of provider code

Dispatch the chat-completions agentic loop from a provider-agnostic
helper (litellm/litellm_core_utils/chat_completion_agentic_loop.py)
called from main.acompletion, instead of from OpenAI provider files.
Register the agentic loop control fields in all_litellm_params so they
stay LiteLLM-level and never become provider payload, removing the need
for the OpenAIGPTConfig scrubber. No litellm/llms/ files are modified for
this feature.

* docs: explain chat agentic loop dispatch and litellm-level param registration

* style: drop Any annotations and use PEP585 generics to satisfy ruff strict budget

* docs: replace module docstring with one-line patch note

* fix(search): block server credential leak to caller-supplied api_base (#30682)

Search providers resolved the server-configured API key (e.g.
get_secret_str("SERPER_API_KEY")) in validate_environment whenever the
caller omitted api_key, while get_complete_url independently honored a
caller-supplied api_base. A caller who passes their own api_base and no
api_key therefore made the proxy send the operator's provider key to a
host they control; POST /search_tools/test_connection forwards
request-body api_base/api_key straight into asearch, so any authenticated
user could exfiltrate the server's search credentials.

Add a shared host-aware fallback in BaseSearchConfig.resolve_server_api_key
that only applies a server-managed secret when the caller-supplied
api_base is absent or resolves to a trusted host (the provider default or
the operator's own *_API_BASE env override); otherwise it refuses and asks
for an explicit api_key. The guard only triggers when a server secret
actually exists, so keyless and self-hosted providers (searxng, you.com
free tier) keep working. Every provider that carries a server secret is
migrated to the helper; dataforseo reuses the same guard for its
login:password basic-auth credentials.

This changes behavior for callers that previously passed a per-request
api_base while relying on a server-configured key: they must now pass an
explicit api_key, or the operator must configure the base via the
provider's *_API_BASE env var (which stays trusted).

* feat: add LiteLLM Rust workspace with Mistral OCR bridge (#31033)

* docs(readme): add Deploy on AWS/GCP with Terraform section

Adds a quickstart for the two published Terraform modules on the public
registry (BerriAI/litellm/aws and BerriAI/litellm/google). Copy-paste
main.tf for each cloud, the one-time GCP Artifact Registry remote-repo
command, and pointers to the registry pages for the full input surface.

Sits inside the Get Started section, between the gateway/SDK table and
Run in Developer Mode -- where someone scanning the README for "how do I
deploy this" will land.

Co-Authored-By: Claude Opus 4.7 <[email protected]>

* docs(readme): add 1-click deploy buttons for AWS + GCP

GCP gets the real 1-click: Open in Cloud Shell badge that clones the repo
and walks through `terraform apply` via the existing DeployStack
tutorial (already shipped at terraform/litellm/gcp/examples/default/
TUTORIAL.md). User just picks a project.

AWS gets a soft 1-click: a Launch in AWS CloudShell badge that opens an
in-browser, already-authenticated shell. User runs four commands
(clone + cd + cp tfvars + terraform apply) once inside. There's no
native AWS deeplink that pre-clones a repo + runs a tutorial -- CFN
"Launch Stack" + CodeBuild would be needed for that, and that's a
separate piece of work.

Co-Authored-By: Claude Opus 4.7 <[email protected]>

* docs(readme): move AWS + GCP deploy buttons next to Render button

* docs(readme): unify deploy button sizes and badge styles

* docs(readme): bump deploy button height to 48 to match Render/Railway

* docs(readme): bump AWS/GCP badge height to compensate for SVG padding

* docs(readme): bump AWS/GCP badge height to 72

* docs(readme): bump AWS/GCP badge height to 84

* fix(readme): make deploy buttons same height (48px)

https://claude.ai/code/session_01MxQRMHSDXbqJh74rF86UBc

* docs(readme): flag GCP project ID substitution in image_registry

* docs(readme): equalize deploy button heights and fix Cloud Shell button font

GitHub rewrites an image's height attribute to "height: auto; max-height: Npx", which only caps and never stretches, so each image renders at its intrinsic height. The AWS/GCP shields badges are intrinsically 28px while the Render/Railway buttons are 40px, leaving the row uneven regardless of the height="48" we set. Replace the two shields badges with committed 40px PNGs so all four header buttons render at the same 40px.

Also swap the Cloud Shell button from open-btn.svg to open-btn.png. The SVG renders its label as live text with font-family "Roboto, Sans" and no generic fallback; since neither font exists in GitHub's render environment, the text fell back to a serif (Times New Roman). The PNG bakes in the correct typeface.

* docs(readme): collapse Railway deploy anchor to a single line

The Railway button wrapped its img across indented lines, so the anchor contained leading and trailing whitespace. GitHub underlines link content, rendering that whitespace as a small blue underline beside the button. Put the anchor on one line like the other three buttons so there is no inner whitespace to underline.

* Add Claude Fable 5 cost map entries as a data-only hotfix

Backports only the model map changes from #30064 so deployments on
released litellm versions pick up Fable 5 pricing, context window, and
the adaptive thinking flag through the hosted cost map fetch without
upgrading. Includes the supports_sampling_params flag on the 28
Fable 5 / Opus 4.7 / Opus 4.8 entries (ignored by released code, read
by the gating that ships with the next release) and the matching
one-line schema declaration so the map validation test passes.

https://claude.ai/code/session_01MZarYYT3aS7DxaNjoax6Gm

* Add litellm rust workspace with mistral ocr bridge

* address greptile rust ocr feedback

* Simplify rust ocr entrypoint

* rust(core): add Auth/Http/Network error variants

* rust: add reqwest (rustls-tls) workspace dependency

* rust(providers): depend on reqwest

* rust(mistral): add complete_url + resolve_api_key helpers

* rust(providers): end-to-end run_ocr orchestrator with shared client + timeout

* rust(bridge): depend on litellm-core

* rust(bridge): add GIL release accounting

* rust(bridge): end-to-end ocr() + gil_stats(), GIL released for HTTP

* ocr: add minimal Rust bridge (use_litellm_rust + rust_ocr)

* ocr: route mistral to Rust when enabled; keep bare-str file rejection

* litellm: export use_litellm_rust()

* test(ocr): cover Rust OCR routing + toggle

* rust: stop ignoring Cargo.lock

* rust: commit Cargo.lock for reproducible builds

* ci(rust): build with --locked to enforce the lockfile

* Potential fix for pull request finding 'CodeQL / Module-level cyclic import'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* Potential fix for pull request finding 'CodeQL / Module-level cyclic import'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* ocr: lazily import rust bridge inside ocr() to break the import cycle the CodeQL autofix mangled

* ocr: guard OCRResponse under TYPE_CHECKING so the annotation resolves

* ocr: modernize rust_bridge typing (PEP 604, drop typing.Any/Dict) to satisfy strict-rule gate

* ci: re-trigger checks

* ci: re-trigger checks

* Potential fix for pull request finding 'CodeQL / Cyclic import'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* Potential fix for pull request finding 'CodeQL / Cyclic import'

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>

* ocr: make rust_bridge a leaf (return raw dict, no litellm import) so the CodeQL autofix stops re-breaking it

* ocr: wrap rust bridge dict into OCRResponse at the call site

* test(ocr): assert rust_ocr returns the raw bridge dict

* test(interactions): add budget_exceeded to expected status enum (Google updated the published spec)

* ocr: resolve mistral key via get_secret_str before the rust path (secret-manager parity)

* test(ocr): assert rust path resolves key via secret manager

* rust(mistral): document that secret-manager resolution happens on the Python side

* fix(ocr): honor timeout, logging, and missing-bridge fallback on Rust OCR path

- Forward the caller's timeout into the Rust bridge so the fixed 600s client
  ceiling no longer overrides shorter deadlines or the library default.
- Run update_from_kwargs and pre_call before invoking the Rust shortcut so
  observability, callbacks, and spend tracking match the Python path.
- Fall back to the Python OCR path when litellm_python_bridge isn't importable
  instead of raising ImportError to callers.
- Truncate upstream Mistral OCR error bodies before they cross the host
  boundary to avoid leaking document or prompt contents in CoreError::Http.

* fix(ocr): log resolved api_base and headers on Rust path

* refactor(ocr): inject the rust bridge via a typed seam, drop the importlib cycle dodge

The rust OCR path was reached through importlib.import_module both for the
bridge module and for probing the native extension, purely to keep CodeQL from
flagging a cyclic import. rust_bridge has no litellm imports, so it is a leaf
and main.py can import it statically without any cycle; the dance is gone

Bridge selection now goes through a typed RustOcr Protocol and a load_rust_ocr()
seam. use_litellm_rust() takes an optional injected bridge, so an embedder (or a
test) can supply an alternative without reaching into sys.modules. The rust-path
body moves into _run_rust_ocr(), which receives its dependencies (the bridge
callable, the logging object, the key resolver) as arguments and is unit-tested
by passing fakes in rather than monkeypatching class methods or module globals

The tests are rewritten around that injection: the bridge is provided via
use_litellm_rust(ocr=...), pre_call is observed through a spy logging object, and
the missing-extension fallback is covered by load_rust_ocr() returning None when
no wheel is built. Types were tightened along the way (a cast for the logging
object, OCRResponse.model_validate for the bridge result) so no basedpyright
per-rule count increases

Co-authored-by: Mateo Wang <[email protected]>

* fix(ocr): preserve injected rust bridge across toggle calls

use_litellm_rust() unconditionally assigned the keyword default of None to
_rust_ocr_impl, so any call without ocr= silently dropped a previously
injected bridge. Use a sentinel default so omission preserves the impl
while ocr=None still clears it explicitly.

* ci: run tests/test_litellm/ocr in the misc unit-test group

The OCR test directory was not wired into any CI test group, so its
coverage never uploaded to Codecov and patch coverage failed for new
OCR lines. Add it to the misc group.

* test(ocr): cover compiled-extension load and Python fallback paths

Adds two tests so the Rust bridge module hits 100% and the ocr()
fallback-to-Python branch is exercised:
- load_rust_ocr() returning the compiled extension's ocr callable
- ocr() degrading to the HTTP handler when no bridge is available

* style(ocr): use PEP 604 X | None annotations in rust_bridge

Converts Optional[X]/Union[...] to the X | None form so the new OCR
code stays under the UP045 strict-rule budget gate (lint job). Safe at
runtime — the module already has 'from __future__ import annotations'.

---------

Co-authored-by: shin-berri <[email protected]>
Co-authored-by: yuneng-jiang <[email protected]>
Co-authored-by: Yassin Kortam <[email protected]>
Co-authored-by: Claude Opus 4.7 <[email protected]>
Co-authored-by: mateo-berri <[email protected]>
Co-authored-by: Krrish Dholakia <[email protected]>
Co-authored-by: Ishaan Jaffer <[email protected]>
Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <[email protected]>
Co-authored-by: Mateo Wang <[email protected]>

* fix(router): honor litellm_settings.request_timeout as an independent per-attempt timeout (#31119)

request_timeout was shadowed by router_settings.timeout: Router stored a single
slot via `self.timeout = timeout or litellm.request_timeout`, so when a router
timeout was set the configured request_timeout was never used. Provider calls
with no per-model timeout (Bedrock especially) then fell back to the hardcoded
600s httpx client default. Mirrors PR #25701 and completes it on top of the
CompletionTimeout work already on this branch.

- Router: add an independent self.request_timeout and prefer it over
  router_settings.timeout in both _get_non_stream_timeout and _get_stream_timeout
- http_handler: cached default clients now fall back to request_timeout instead
  of a hardcoded 600s
- Replace the brittle `== 6000` default-detection heuristic with a single
  get_configured_request_timeout() resolver backed by an explicit
  request_timeout_explicitly_set sentinel (set from REQUEST_TIMEOUT env and
  litellm_settings), keeping the value-differs fallback for SDK assignment.
  This also fixes an explicit request_timeout of 6000 being coerced to 600
- CompletionTimeout no longer second-guesses the package default; the caller
  passes the explicitly-configured value or None

Regression for LIT-2369.

* fix: tighten role-based visibility of config and MCP fields (#30587)

* fix: redact config and MCP secrets in read-only admin views

GET /config/field/info and the MCP server list/detail endpoints returned
secret-bearing field…