fix(guardrails): stop re-initializing DB guardrails on every poll by yassin-berriai · Pull Request #30542 · BerriAI/litellm

yassin-berriai · 2026-06-16T16:33:16Z

Relevant issues

Fixes the guardrail re-initialization memory leak reported during performance testing.

Linear ticket

LIT-3606

Pre-Submission checklist

Please complete all items before asking a LiteLLM maintainer to review your PR

I have added meaningful tests
My PR passes all unit tests on make test-unit
My PR's scope is as isolated as possible; it only solves 1 specific problem
I have requested a Greptile review by commenting @greptileai and received a Confidence Score of at least 4/5 before requesting a maintainer review

Delays in PR merge?

If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).

CI (LiteLLM team)

Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:

Screenshots / Proof of Fix

Reproduced against a live proxy backed by a Dockerized Postgres, hitting the real OpenAI API (gpt-4o-mini). The DB row is seeded with a sparse litellm_params (only the keys a user actually set), which is what an older proxy version writes; this is the shape that triggers the comparison mismatch.

Setup:

docker run -d --name lit3606-pg -e POSTGRES_PASSWORD=postgres -e POSTGRES_DB=litellm -p 5434:5432 postgres:16

docker exec lit3606-pg psql -U postgres -d litellm -c \
  "INSERT INTO \"LiteLLM_GuardrailsTable\" (guardrail_id, guardrail_name, litellm_params, guardrail_info, status, created_at, updated_at) \
   VALUES (gen_random_uuid()::text, 'my-content-filter', \
   '{\"guardrail\":\"litellm_content_filter\",\"mode\":\"pre_call\",\"default_on\":true,\"blocked_words\":[{\"keyword\":\"secret\",\"action\":\"BLOCK\"}]}'::jsonb, \
   '{}'::jsonb, 'active', now(), now());"

python litellm/proxy/proxy_cli.py --config config.yaml --detailed_debug

for n in 1 2; do
  curl -s -X POST http://localhost:4000/v1/chat/completions \
    -H "Authorization: Bearer sk-1234" -H "Content-Type: application/json" \
    -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"say hello in 3 words"}]}' \
    -o /dev/null -w "request $n -> HTTP %{http_code}\n"
done
# request 1 -> HTTP 200
# request 2 -> HTTP 200

Then the proxy is left running across DB poll cycles (30s interval) and the re-initialization log lines are counted.

Before the fix, the DB-stored guardrail re-initializes on every single poll:

DB poll cycles that saw the guardrail : 4
re-initialization log lines           : 4
16:24:06 - guardrail_registry.py:752 - Guardrail 'my-content-filter' params changed, re-initializing...
16:24:36 - guardrail_registry.py:752 - Guardrail 'my-content-filter' params changed, re-initializing...
16:25:06 - guardrail_registry.py:752 - Guardrail 'my-content-filter' params changed, re-initializing...
16:25:36 - guardrail_registry.py:752 - Guardrail 'my-content-filter' params changed, re-initializing...

After the fix, it is initialized once when first loaded and never re-initialized again, even as polling continues:

DB poll cycles that saw the guardrail : 5
re-initialization log lines           : 1
16:26:12 - guardrail_registry.py:781 - Guardrail 'my-content-filter' params changed, re-initializing...

The single line after the fix is the initial load (the guardrail is not yet in memory); every subsequent poll sees the config as unchanged and leaves the existing instance in place.

Type

🐛 Bug Fix

Changes

DB-stored guardrails were re-initializing on every DB poll cycle. InMemoryGuardrailHandler._has_guardrail_params_changed compared the in-memory LitellmParams against the raw dict loaded from the database. The in-memory side is a LitellmParams whose model_dump() carries every field default and coerces enums, while the DB side is the raw stored dict that only holds the keys originally written (so a row written by an earlier proxy version is missing fields the current model fills in, and a value like version reads as 2 on one side and absent on the other). The two shapes never compared equal, so the handler treated the guardrail as changed on every poll and rebuilt it indefinitely.

Each rebuild created a fresh guardrail instance, but delete_in_memory_guardrail only removed the old callback from litellm.callbacks. Request handling promotes guardrail callbacks into the success, failure, and async callback lists, so the previous instance stayed referenced in those lists after every rebuild and instances accumulated until the pod was OOM-killed.

The comparison now normalizes both sides through LitellmParams(...).model_dump() before diffing, so an unchanged config compares equal and the guardrail is rebuilt only on a genuine change. If a stored row fails LitellmParams validation the normalizer catches that specific ValidationError, logs a warning so the offending row is diagnosable, and treats the guardrail as changed rather than swallowing every error or crashing the poll cycle. delete_in_memory_guardrail now purges the callback from every callback list, so a rebuild leaves nothing stranded.

Tests in tests/test_litellm/proxy/guardrails/test_guardrail_registry.py cover the comparison (unchanged DB params no longer register as changed, a genuine change still does), the delete path (the callback is removed from every list), and an end-to-end check that repeated DB syncs keep exactly one live guardrail instance across all callback lists.

InMemoryGuardrailHandler._has_guardrail_params_changed compared the in-memory LitellmParams against the raw dict loaded from the DB. The in-memory side carries every field default and coerces enums via model_dump(), while the DB side only holds the keys originally stored, so the two shapes never compared equal and the guardrail was rebuilt on every poll cycle. Each rebuild created a fresh instance, but delete_in_memory_guardrail only removed the old callback from litellm.callbacks. Request handling promotes guardrail callbacks into the success/failure/async lists, so the previous instance stayed referenced there and instances accumulated. Normalize both sides through LitellmParams(...).model_dump() before diffing, and purge the callback from every callback list on delete.

CLAassistant · 2026-06-16T16:33:26Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

codecov · 2026-06-16T16:37:22Z

Codecov Report

❌ Patch coverage is 90.90909% with 2 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
litellm/proxy/guardrails/guardrail_registry.py	89.47%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!

greptile-apps · 2026-06-16T16:55:53Z

Greptile Summary

This PR fixes a memory leak where DB-backed guardrails were re-initialized on every poll cycle, accumulating stale callback instances in the success/failure/async lists until the pod was OOM-killed. Two independent bugs are addressed together.

Comparison mismatch fixed: _has_guardrail_params_changed now normalizes both the in-memory LitellmParams and the raw DB dict through LitellmParams(**params).model_dump() before diffing, so sparse DB rows (missing defaults) and fully-populated in-memory objects compare equal when the config is unchanged. Invalid DB rows fall back to "treat as changed" with a warning log.
Stale callbacks fixed: delete_in_memory_guardrail now calls remove_callback_from_all_lists instead of only removing from litellm.callbacks, ensuring re-initialization leaves nothing stranded in the success/failure/async lists that request handling promotes callbacks into. Five new unit tests cover each fix path, including an end-to-end accumulation regression test.

Confidence Score: 5/5

Safe to merge — the changes are narrowly scoped to the DB poll comparison and callback cleanup path, both of which are exercised by new unit tests that reproduce the original OOM scenario.

The normalization logic is correct: LitellmParams inherits extra="allow" from BaseLitellmParams, so both sides of the comparison go through the same Pydantic round-trip and produce identical model_dump() output for unchanged configs. The delete path change is additive — it removes from more lists than before, not fewer. The ValidationError fallback explicitly logs and treats malformed rows as changed, which is intentional and no longer causes accumulation because the delete is now thorough. Tests directly cover every new branch, including the end-to-end accumulation scenario.

No files require special attention.

Important Files Changed

Filename	Overview
litellm/proxy/guardrails/guardrail_registry.py	Adds _normalize_litellm_params_for_comparison to canonicalize both the in-memory LitellmParams and the raw DB dict through model_dump() before diffing; updates delete_in_memory_guardrail to purge callbacks from all five lists. Logic is correct and well-documented.
litellm/litellm_core_utils/logging_callback_manager.py	Adds remove_callback_from_all_lists that iterates over all five callback lists and delegates to the existing remove_callback_from_list_by_object. Straightforward and consistent with existing patterns.
tests/test_litellm/proxy/guardrails/test_guardrail_registry.py	Adds five new unit tests covering: unchanged DB params not triggering re-init, genuine changes still detected, malformed DB rows handled gracefully, delete purging from all lists, and an end-to-end accumulation regression test. All tests use mocks only — no real network calls.
tests/litellm_utils_tests/test_logging_callback_manager.py	Adds test_remove_callback_from_all_lists verifying the new method removes from every callback list. Correct use of test setup/teardown pattern.

_{Reviews (4): Last reviewed commit: "refactor(callbacks): add remove_callback..." | Re-trigger Greptile}

yassin-berriai · 2026-06-16T17:05:06Z

@greptileai

…ionError The comparison normalizer caught a bare Exception and silently fell back to the raw dict, which hid the cause and quietly degraded the affected guardrail back to re-initializing on every poll. Catch only the ValidationError that LitellmParams construction can raise, log a warning so the offending row is diagnosable, and let any other error surface instead of being swallowed.

yassin-berriai · 2026-06-16T17:08:49Z

@greptileai

…ager Move the knowledge of which callback lists a callback can be promoted into out of the guardrail registry and into LoggingCallbackManager, where the rest of the callback-list bookkeeping already lives. delete_in_memory_guardrail now delegates to the new helper instead of iterating the lists itself.

yassin-berriai · 2026-06-16T18:01:52Z

@greptileai

yassin-berriai · 2026-06-16T18:14:46Z

+        for callback_list in (
+            litellm.callbacks,
+            litellm.success_callback,
+            litellm.failure_callback,
+            litellm._async_success_callback,
+            litellm._async_failure_callback,


_get_all_callbacks returns a flattened concatenation (litellm.callbacks + litellm.success_callback + ...), which is a fresh list object, so removing from its result would mutate that temporary list and leave the underlying lists untouched.

@OverRide

…30554) * chore(codecov): add Batches, Videos, and Realtime components (#30517) * chore(codecov): add Batches, Videos, and Realtime components Define per-feature Codecov components so PR comments track coverage for batch API, video generation, and realtime streaming paths. Co-authored-by: Cursor <[email protected]> * chore(codecov): use wildcard path for Batches proxy component Align batches_endpoints glob with Videos, Realtime, and Proxy_Authentication. Co-authored-by: Cursor <[email protected]> --------- Co-authored-by: Cursor <[email protected]> * test(batches): move orphan tests into tests/test_litellm for CI coverage (#30510) Four batch-related tests lived under tests/litellm/ and were never picked up by GitHub Actions. Relocate them and fix gemini multimodal e2e to use the batchEmbedContents path expected for gemini/ provider. Co-authored-by: Cursor <[email protected]> * fix(guardrails): run pre_call hook once for model-level guardrails (#30543) * fix(guardrails): run pre_call hook once for model-level guardrails A CustomGuardrail attached to a deployment via litellm_params.guardrails gets its async_pre_call_hook invoked twice per request: once by the proxy pre-call loop and again by async_pre_call_deployment_hook after the router spreads the model-level guardrails into the top-level request kwargs. Record in request metadata that the proxy pre-call loop already ran a given guardrail, and have the deployment hook skip it when the marker is present. Direct-SDK usage never runs the proxy loop, so the deployment hook stays the sole invocation there and still fires exactly once. The marker key is stripped from untrusted caller metadata so a request body cannot suppress a model-only guardrail by pre-seeding it. * fix(guardrails): mark pre_call dedup on the post-hook request data Record the exactly-once marker after async_pre_call_hook runs, on the data object that flows downstream, rather than before it. A guardrail whose hook returns a brand-new request dict (instead of mutating or spreading the one it received) would otherwise discard the marker, letting the deployment hook re-run the guardrail a second time. * fix(guardrails): stop re-initializing DB guardrails on every poll (#30542) * fix(guardrails): stop re-initializing DB guardrails on every poll InMemoryGuardrailHandler._has_guardrail_params_changed compared the in-memory LitellmParams against the raw dict loaded from the DB. The in-memory side carries every field default and coerces enums via model_dump(), while the DB side only holds the keys originally stored, so the two shapes never compared equal and the guardrail was rebuilt on every poll cycle. Each rebuild created a fresh instance, but delete_in_memory_guardrail only removed the old callback from litellm.callbacks. Request handling promotes guardrail callbacks into the success/failure/async lists, so the previous instance stayed referenced there and instances accumulated. Normalize both sides through LitellmParams(...).model_dump() before diffing, and purge the callback from every callback list on delete. * refactor(guardrails): narrow params-normalization fallback to ValidationError The comparison normalizer caught a bare Exception and silently fell back to the raw dict, which hid the cause and quietly degraded the affected guardrail back to re-initializing on every poll. Catch only the ValidationError that LitellmParams construction can raise, log a warning so the offending row is diagnosable, and let any other error surface instead of being swallowed. * refactor(callbacks): add remove_callback_from_all_lists helper to manager Move the knowledge of which callback lists a callback can be promoted into out of the guardrail registry and into LoggingCallbackManager, where the rest of the callback-list bookkeeping already lives. delete_in_memory_guardrail now delegates to the new helper instead of iterating the lists itself. * chore(oss): litellm oss staging 150626 (#30463) * fix(pricing): add GitHub Copilot MAI Code Flash pricing (#30415) * fix(pricing): add GitHub Copilot MAI Code Flash pricing Add GitHub Copilot pricing entries for MAI-Code-1-Flash and the internal Copilot CLI model name so cost calculation can price input, cached input, and output tokens. Co-authored-by: Copilot <[email protected]> * test(pricing): cover GitHub Copilot MAI Code Flash pricing Add regression coverage for both GitHub Copilot MAI-Code-1-Flash model names, including cached input pricing, chat endpoint metadata, and cost_per_token arithmetic. Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: Copilot <[email protected]> * fix(router/proxy): propagate completed_response through FallbackResponsesStreamWrapper for streaming /v1/responses container ownership (#30210) (#30213) * fix(router/proxy): propagate completed_response through FallbackResponsesStreamWrapper for streaming /v1/responses container ownership (#30210) #28990 added ownership recording for streaming /v1/responses via _wrap_responses_stream_for_container_ownership, which reads `getattr(stream_response, 'completed_response', None)` to extract the ResponsesAPIResponse. The unit test bypassed the Router, so it never exercised the production wrapping path. Through the Router (every proxy deployment), the stream is wrapped by FallbackResponsesStreamWrapper (router.py:2527). Its __init__ set `self.completed_response = None` and __anext__ only forwarded chunks — the inner source iterator's terminal event never bubbled up to the attribute the ownership hook reads, so the hook silently recorded nothing and every follow-up /v1/containers/<id>/files call returned 403 for non-admin keys. This commit: - router.py: pre-resolves the responses-API terminal event tuple (response.completed / .incomplete / .failed) once per _aresponses_streaming_iterator call, and has the wrapper's __anext__ sniff each forwarded chunk's .type. First terminal event hit gets stored on the wrapper's completed_response. Iterator-agnostic — works for source_iterator AND any future wrapper. - common_request_processing.py: when _extract_completed_responses_response returns None we now warn instead of silently skipping. Reporter on #30210 lost a day to this exact silent skip; the warning surfaces future regressions of the same shape directly in operator logs. Fixes #30210 * fix(router): type-ignore wrapper getattr-defaults; broaden ownership-skip warning CI lint (mypy) flagged the three pre-existing getattr(..., None) assignments in FallbackResponsesStreamWrapper.__init__: router.py:2564 self.response = getattr(source_iterator, 'response', None) router.py:2565 self.model = getattr(source_iterator, 'model', None) router.py:2566 self.logging_obj = getattr(..., None) Those lines also exist on litellm_internal_staging and pass mypy there. Adding the typed terminal-event tuple above the class made the function body more narrowable, which surfaced the pre-existing mismatch — base class declares non-Optional types but the bridge path (LiteLLMCompletionStreamingIterator) legitimately omits these. Keep the None fallback and silence with type: ignore[assignment]. Greptile 4/5 note: the ownership-skip warning hard-named code_interpreter which misleads operators when a non-code_interpreter stream aborts. Generalize to 'any tool container (e.g. code_interpreter)'. * fix(register_model): drop synthesized zero costs to preserve sparse entries (#30198) (#30201) * fix(register_model): drop synthesized zero costs to preserve sparse entries (#30198) get_model_info synthesizes input_cost_per_token / output_cost_per_token = 0 when they are absent from the raw entry (the price-unknown and free cases share the same representation). register_model then merges that result back into litellm.model_cost, which flips a sparse entry from 'no cost keys' (priced via model name) to 'cost keys = 0' (free). That defeats _is_cost_explicitly_configured (#24949) on re-registration: _is_model_cost_zero returns True, common_checks skips every tag / key / team / user / org budget check for the group, and over-budget traffic keeps returning 200. Spend keeps recording because cost calc still resolves by model name, so the symptom is silent and only triggers on the second register_model pass (router rebuild, /model/update, config sync). Mirror the existing litellm_provider-None guard one block above and pop the cost fields from the synthesized result when they are absent from the raw entry and not in the caller's value. Caller-provided zeros (genuinely free models, BYOK overrides) are preserved. Fixes #30198 * fix(register_model): switch _raw_entry to is-None checks + drop dead test assertion Greptile #30201 review notes: - the `or`-chain in the raw-entry lookup treated an empty dict (a key with no fields) as falsy and fell through to the second arm — replace with explicit `is None` checks so a present-but-empty entry is still taken at face value. - the first assertion in `test_router_double_init_keeps_db_model_entry_sparse` used `in (None, 0)` which passes under the bug condition (cost = 0 matches the tuple); the strong follow-up assertion already covers every shape, so drop the dead branch. * fix(bedrock mantle): use unique function-call id for responses->chat tool calls (#30426) * fix(bedrock mantle): use unique function-call id for responses->chat tool calls ... * fix(bedrock mantle): scope unique tool-call id fallback to degenerate call_id The previous revision preferred the Responses item id for every tool call, which broke providers (and existing tests) where call_id is a unique, canonical correlation key. Restrict the fallback to the degenerate index-based call_id that Bedrock Mantle returns (call_0, call_1, ... resetting per response) and keep call_id otherwise. Revert the change to the OUTPUT_ITEM_DONE streaming handler, whose tool_call_chunk is never emitted (dead code, per review). Extend the regression tests to assert a normal call_id is preserved. * fix(router): preserve azure_ad_token through CredentialLiteLLMParams for /v1/files + batches (#30235) (#30241) * fix(router): preserve azure_ad_token through CredentialLiteLLMParams for /v1/files + batches (#30235) Router.get_deployment_credentials_with_provider re-validates a deployment's litellm_params through CredentialLiteLLMParams before handing them to file/batch/passthrough callers: return CredentialLiteLLMParams( **deployment.litellm_params.model_dump(exclude_none=True) ).model_dump(exclude_none=True) Any field NOT declared on CredentialLiteLLMParams gets silently dropped on the way through. azure_ad_token was undeclared, so Azure deployments using OAuth/M2M (azure_ad_token instead of a static api_key) silently lost their token at the files endpoint and the proxy returned: Missing credentials. Please pass one of api_key, azure_ad_token, azure_ad_token_provider, ... Declare azure_ad_token on CredentialLiteLLMParams alongside api_key / api_base / api_version so it rides through the round-trip. Static-key deployments stay unaffected (Optional, default None, dropped by exclude_none=True). Provider-callable (azure_ad_token_provider) is a separate concern and out of scope here. Fixes #30235 * fix(ui-types): regenerate schema.d.ts for new azure_ad_token field CI's 'Verify schema.d.ts matches the proxy OpenAPI spec' check auto-detected the new field and emitted the exact diff to apply. Two schemas had `aws_secret_access_key` from CredentialLiteLLMParams, both get the new azure_ad_token marker next to it. * fix(proxy): org_admin with own user_id now sees all org teams on /v2/team/list (#30247) When the UI sends the callers own user_id (as it does for non-Admin global roles), _enforce_list_team_v2_access now nulls it out for org admins so _build_team_list_where_conditions scopes by organization_id only -- matching the legacy /team/list behavior and the documented intent. Fixes #30215 Co-authored-by: Claude Opus 4.6 <[email protected]> * test(vertex_ai): multi-region regression coverage for cachedContents host (#29571) (#29707) litellm_internal_staging already routes the cachedContents URL through get_vertex_base_url, fixing the multi-region 404 reported in #29571 — but carries no test coverage for the actual regression scenario (eu/us must resolve to the REP host aiplatform.{geo}.rep.googleapis.com). Add TestContextCachingMultiRegionUrls: parametrized eu/us REP-host assertions (including absence of the old broken {geo}-aiplatform host), plus regional (us-central1) and global no-regression checks. * fix(proxy): close upstream LLM stream when client disconnects mid-stream (#30245) * fix(proxy): close upstream LLM stream when client disconnects mid-stream When a streaming client disconnects, Starlette abandons the response body iterator without calling aclose(), so the proxy's connection to the upstream backend stays open until garbage collection, which may never come. The backend (e.g. vLLM) keeps generating into a dead pipe: small responses drain invisibly into TCP buffers while large ones block the backend on a full send buffer indefinitely (observed via lsof as an ESTABLISHED proxy->backend connection minutes after the client left) create_response now returns a StreamingResponse subclass that closes both its body iterator and the wrapped upstream-facing generator in a shielded finally. The upstream generator is closed directly rather than through a cascade because aclose() on a never-started generator skips its body, which would make the cascade a no-op when the client disconnects before the first chunk is sent. async_streaming_data_generator also gains the same shielded finally-aclose that async_data_generator in proxy_server.py already had, covering the Anthropic and Google SSE paths With this, killing a streaming client causes the backend to observe the abort within about a second and free its slot, while completed streams are unaffected. No flag is needed, unlike the non-streaming opt-in cancel in #30223: this only releases resources after the client is already gone and does not change any response a client can observe Fixes #30244 * fix(proxy): close upstream even when body iterator aclose raises BaseException Addresses the Greptile finding on #30245: the cleanup loop caught only Exception while the generator-level cleanup catches BaseException, so a CancelledError or GeneratorExit escaping body_iterator.aclose() would skip closing the upstream generator. Both sites now use the same scope and a regression test pins that the upstream is closed even when the body iterator explodes with a BaseException * fix(llms): expose aclose on BaseModelResponseIterator so stream close reaches the provider connection The response-level close added for #30244 only worked for SDK-based providers (e.g. openai), whose streams expose aclose all the way down. Providers served by base_llm_http_handler (hosted_vllm and most modern transformation-based providers) wrap a bare response.aiter_lines() generator in BaseModelResponseIterator, which had no aclose or close at all, and nothing retained the httpx response object; so CustomStreamWrapper.aclose() silently did nothing and the upstream connection stayed open. Verified with a vLLM-style mock: with hosted_vllm/ the backend streamed all 100 chunks to completion after the client disconnected, while openai/ aborted at chunk 6 BaseModelResponseIterator now carries an optional http_response and an aclose() that closes it; make_async_call_stream_helper attaches the response after building the iterator. With this, hosted_vllm aborts the backend within ~1.6s of the client dropping, and completed streams are unaffected --------- Co-authored-by: kursad <[email protected]> * feat(anthropic): surface compaction usage iterations data (#27065) * feat(anthropic): surface compaction usage iterations data * style: apply black formatting to fix lint checks * fix(usage): correct calculate usage with cached tokens when use ChatCompletionUsageBlock (#30422) * fix(usage): correct calculate usage with cached tokens when use ChatCompletionUsageBlock * fix(usage): optimize test imports * feat: add fastCRW search provider (#30434) * feat(provider): add LibertAI as a JSON-configured OpenAI-compatible provider (#30203) * feat(provider): add LibertAI as a JSON-configured OpenAI-compatible provider * libertai: update served endpoints backup + add mode/matrix tests Addresses review feedback: - Add libertai to litellm/provider_endpoints_support_backup.json, the file actually served by GET /public/supported_endpoints (the root provider_endpoints_support.json already had it). - Add tests asserting bge-m3 normalizes to mode='embedding' and that the served matrix lists libertai. embeddings stays false: the JSON-configured provider path only wires chat routing (OpenAILike embedding handler is reached only for literal openai_like/llamafile/lm_studio), matching the llamagate precedent; bge-m3 remains in the cost map for metadata. --------- Co-authored-by: Moshe Malawach <[email protected]> * feat(provider): add ModelScope as an OpenAI-compatible provider (#28460) * add ModelScope API support * add modelscope api support * update modelscope model list * add image-genetation support * update test and multimodal * fix: address PR review feedback for modelscope provider * update README * fix(customer_endpoints): restrict /customer/daily/activity to admin-only (#28849) * fix(customer_endpoints): restrict /customer/daily/activity to admin-only * fix(customer_endpoints): check role before prisma_client guard * fix(custom_guardrail): key disable_global_guardrails takes precedence over team guardrail list (#28563) * fix(fallbacks): preserve fallback model in SDK fallback responses (#28260) * fix(fallbacks): preserve fallback model in response when using SDK-level fallbacks * fix(fallbacks): gate x-litellm-* passthrough to trusted callers only The previous patch unconditionally let `x-litellm-*` keys bypass the `llm_provider-` prefix in `process_response_headers`. That function is also called on raw upstream-provider response headers (e.g. from `llm_http_handler.py`), so a malicious provider could return `x-litellm-attempted-fallbacks` and spoof a LiteLLM-internal marker, bypassing the proxy model-override guard. Add a `preserve_litellm_internal_headers` flag (default False). Only `response_metadata.py`, which re-processes the already-built `_hidden_params["additional_headers"]` dict (LiteLLM-owned), passes True. Raw provider header callsites keep the default False, so upstream `x-litellm-*` still gets the `llm_provider-` prefix. Adds a regression test for the spoofing case and renames the existing preserve test to make the trusted-path semantics explicit. * fix(fallbacks): ignore preserve_litellm_internal_headers for raw httpx.Headers inputs * style(core_helpers): apply black formatting * fix(lint): remove banned typing.List/Dict/Any imports and suppress PLR0913 on interface overrides Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(lint): apply black formatting to modelscope chat transformation Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(lint): replace noqa with proper fixes — use **kwargs and Awaitable instead of Any/List Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(lint): remove unused AllMessageValues import Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * revert: restore base_model_iterator.py to original PR state Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(lint): restore full method signatures for MyPy compatibility; bump PLR0913 budget for new provider files Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(lint): use @OverRide to suppress PLR0913 on inherited signatures instead of bumping budget The overrides keep their full base-class signatures for MyPy compatibility, but those signatures carry more than five parameters, which tripped PLR0913 on each subclass redeclaration. Since the arity is dictated by the base class and cannot be reduced, decorate the overrides with typing_extensions.override; ruff treats that as the intended signal that the parameter count is not under the author's control and skips PLR0913. This restores the PLR0913 baseline to 1813. * fix(lint): add @OverRide to modelscope image generation overrides Apply the same typing_extensions.override treatment to the image generation config so its inherited-signature overrides do not count against PLR0913. --------- Co-authored-by: Joel Tony <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: hcl <[email protected]> Co-authored-by: ztko <[email protected]> Co-authored-by: Nahrin <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: Humphrey <[email protected]> Co-authored-by: kursadlacin <[email protected]> Co-authored-by: kursad <[email protected]> Co-authored-by: Dushyant Acharya <[email protected]> Co-authored-by: Yuriy <[email protected]> Co-authored-by: Recep S <[email protected]> Co-authored-by: Moshe Malawach <[email protected]> Co-authored-by: Moshe Malawach <[email protected]> Co-authored-by: Rongkun Yan <[email protected]> Co-authored-by: Varshith <[email protected]> Co-authored-by: Mateo Wang <[email protected]> * ci(lint): add blanket-noqa, dataclass-default, and unused-noqa Ruff rules (#30516) * ci(lint): enforce blanket-noqa, dataclass-default, and unused-noqa rules Enable PGH004 (blanket-noqa), RUF008 (mutable-dataclass-default), RUF009 (function-call-in-dataclass-default-argument), and RUF100 (unused-noqa) in ruff.toml, and clean up every resulting violation. RUF008/RUF009 were already clean. PGH004/RUF100 surfaced ~335 stale or blanket noqas: blanket `# noqa` are now scoped to the rule they actually suppress (mostly T201), dead directives are removed, and inapplicable codes are trimmed (e.g. F401 dropped from `import *`). lint.external lists rules enforced outside this config (the strict-rule gate via ruff-strict.toml and upstream litellm's own ruff config) so RUF100 keeps the noqa directives that protect them instead of stripping coverage this config can't see. * ci(lint): trim RUF100 external list to load-bearing codes only Drop the 9 precautionary strict-gate codes (ANN001/002/003/401, B006, PLR0913, PLW0603, RUF012, TID251) that have zero `# noqa` references in the gated source. Keep only the 11 codes with live suppressions so RUF100 doesn't flag them as unused. Future strict-gate suppressions can re-add codes here (or fix the underlying issue) as needed. * ci: ratchet lint and type-check gates (ruff preview, ANN, mypy, basedpyright) (#30379) * ci: enable ruff preview rules under the budgeted strict gate Turn on ruff preview in the strict-budget lane (ruff-strict.toml) only, leaving the clean gate (ruff.toml) untouched so make lint-ruff stays at zero. Enumerate the 118 firing codes explicitly with explicit-preview-rules so the gate is deterministic and stable across ruff upgrades rather than depending on preview auto-selecting the broad catalog. Grandfather the existing 58438 violations into ruff-strict-budget.json as per-rule baselines with headroom, so only net-new violations fail CI. The existing ten rules keep their hand-tuned slack; the new rules get slack 10 when the baseline is 50 or more and 3 otherwise. * ci: add ANN return-type rules to the budgeted strict gate Add ANN201/202/204/205/206 (missing return annotations) to the strict lane and grandfather the existing counts into ruff-strict-budget.json so the codebase ratchets toward explicit return types without breaking CI. * ci: add mypy (disallow_untyped_defs) and basedpyright strict gates with baselines Add two type-check gates, each grandfathering the current tree so only net-new violations fail CI, matching the ruff strict-budget ratchet. mypy gains disallow_untyped_defs in litellm/mypy.ini (the config the CI invocation actually reads; the root [tool.mypy] is not picked up from the litellm/ working dir). The 4885 existing missing-annotation errors are captured in litellm/.mypy-baseline.txt and the run is piped through mypy-baseline filter so new untyped defs are rejected. basedpyright runs in strict mode over litellm/, with enableTypeIgnoreComments disabled so it only honors '# pyright: ignore' and never polices mypy's '# type: ignore'. The existing strict diagnostics are grandfathered into .basedpyright/baseline.json. Both tools are pinned in the dev group and uv.lock; the lint workflow and Makefile run them filtered through their baselines, with lint-mypy-baseline-update and lint-basedpyright-baseline-update to ratchet. * ci: raise lint job timeout to 15m for the basedpyright strict pass * ci: pin pythonVersion 3.12 and regenerate baselines against merged base Merge litellm_internal_staging so the baselines cover code the CI merge includes (e.g. the cisco_ai_defense guardrail), which otherwise tripped the mypy gate with 3 ungrandfathered no-untyped-def errors. Pin pythonVersion 3.12 in pyrightconfig so basedpyright's strict analysis is reproducible across interpreter versions (CI runs 3.12). * ci: regenerate basedpyright baseline against the frozen lint env The previous baseline was generated with optional provider deps (azure, google, anthropic, mcp, numpydoc, google-genai) installed locally, so CI's dev-only env surfaced ~3500 reportUnknown*/reportMissingTypeStubs errors not in the baseline. Regenerate after uv sync --frozen so the baseline reflects the same dependency set the lint job sees. * ci: regenerate basedpyright baseline on python 3.12 frozen env The prior baseline still carried proxy-dev packages (e.g. prisma) that the lint job's dev-only, python 3.12 env lacks, leaving 2 unresolved-import errors ungrandfathered. Regenerate in a python 3.12 venv synced to the frozen lock with default groups only, so the baseline matches exactly what CI sees. * ci: replace type-check baselines with per-file count budgets The mypy and basedpyright baselines were position-sensitive (and the basedpyright one was a 27MB file), so ordinary line shifts churned them. Replace both with a per-file count gate: scripts/type_check_gate.py reduces each tool's output to errors-per-file and checks it against a committed {file: max} budget, ignoring line and column numbers. A file fails only when it gains more errors than its ceiling; debt can't be shuffled between files because each file has its own cap and new files default to zero. Budgets (mypy-file-budget.json 48K, basedpyright-file-budget.json 96K) are generated in the python 3.12 frozen lint env so they match CI. Drops the mypy-baseline dependency; basedpyright runs without its native baseline. ratchet via make lint-mypy-budget-update / lint-basedpyright-budget-update. * ci: add a small per-file slack to the type-check gate Allow each file to drift PER_FILE_SLACK (5) errors past its recorded count before failing, so a basedpyright inference ripple in an unrelated file doesn't break the build over a couple of errors. Budgets still record exact counts; the tolerance is applied at check time. * ci: move type-check slack into the budget json and trim lint timeout Make slack declarative: the budget is now {"slack": N, "files": {path: count}} so the tolerance is tuned in JSON without editing the script, mirroring how ruff-strict-budget.json carries its slack. --update preserves the existing slack. Also drop the lint job timeout from 15m to 10m; the mypy and basedpyright passes add ~2m, leaving the job around 4-5m, so 10m is a comfortable margin. * ci: collapse fully-adopted ruff categories and drop inert preview flag ANN (all nine non-removed rules) and BLE (its only rule) were spelled out code-by-code; replace each with its category selector, which is exactly equivalent in 0.15.3 (the removed ANN101/ANN102 are skipped by a category selector and error when named explicitly). explicit-preview-rules was inert: every selected rule is stable and nothing is selected by category, so the flag had nothing to gate. Verified the strict-rule counts are identical before and after (62379 each, zero per-rule drift), so no budget change. * ci: drop redundant pyright dev dependency Nothing invokes bare pyright in the Makefile, the linting workflow, or scripts; the basedpyright gate added on this branch is the only type checker that runs. basedpyright is a superset fork that reads the same pyrightconfig.json and honors the same "# pyright: ignore" comments, so pyright==1.1.408 in the ci group was dead weight. Regenerated uv.lock under the same exclude-newer cutoff so the only change is removing pyright and its package stanza * ci: un-weaken mypy and error on Any in basedpyright mypy: enable warn_return_any, drop the valid-type silencer, and stop globally ignoring missing first-party imports via [mypy-litellm.*] ignore_missing_imports = False, which surfaced eight real broken litellm.* imports the blanket ignore was hiding; third-party imports stay ignored. The per-file budget moves 4888 -> 5799 (902 no-any-return, 1 valid-type, 8 import-not-found), all grandfathered so only net-new errors fail and the ceilings ratchet down basedpyright: error on reportExplicitAny and reportAny. The per-file budget moves 117033 -> 148946 (6931 explicit-Any, 24954 Any-typed expressions), grandfathered the same way * ci: add Any-discipline gate on changed lines under litellm/ Add scripts/check_any_discipline.py, a type-aware gate that fails when a changed line holds a value typed Any -- including the X | Any unions that mypy --strict / basedpyright accept (e.g. re.Match.group() -> str | Any, json.loads() -> Any, bare dict -> dict[Any, Any]). It reuses the repo's mypyc-compiled mypy 1.19 via a custom generic AST walker (mypyc precludes subclassing TraverserVisitor), loads litellm/mypy.ini for parity with lint-mypy, and uses a dedicated incremental cache (.mypy_cache_any) with mtime+hash invalidation to force re-checks. Scope is changed-lines-only so editing a legacy file never forces cleaning its existing Any debt; suppress a genuine typed/untyped boundary with # any-ok: <reason> (ANY002 requires the reason). Wire it into the Makefile (lint-any, lint, lint-dev), a parallel any-discipline CI job with its own actions/cache, .gitignore, and the CLAUDE.md / CONTRIBUTING.md docs. * ci: move Any-gate codes into the shared LIT namespace Renumber the Any-discipline checker into the LIT*** scheme owned by scripts/check_type_discipline.py (PR #30500) so the two checkers share one rule namespace and suppression convention: ANY001 -> LIT002 (Any-typed value; LIT002 was the retired/free slot) ANY002 -> LIT005 (any-ok without a reason; the shared suppression-reason code) ANY000 -> LIT000 (setup/build/read error; the shared error code) Messages and behavior are unchanged; LIT005's text already matches the "<token> requires a reason" shape used for cast-ok/guard-ok. * ci: gate mypy and basedpyright per error rule, not per file Switch the mypy/basedpyright budget gate from per-file error counts to per-rule-code totals, mirroring the {rule: {baseline, slack}} shape of ruff-strict-budget.json. A rule fails when its codebase-wide error count exceeds baseline + slack, so violations are tracked by category rather than by file location. scripts/type_check_gate.py now parses mypy from its text output (trailing [code]) and basedpyright from --outputjson (the JSON `rule` field), since basedpyright's wrapped text diagnostics mis-attribute the rule on continuation lines. Replace the *-file-budget.json files with freshly captured *-code-budget.json baselines and update the Makefile, CI, and CLAUDE.md accordingly. * docs: prefer Pydantic validation over any-ok suppression Point the Any-discipline guidance at validating Any with Pydantic (a model or TypeAdapter that returns a typed value or raises) and frame # any-ok as a last resort that should ideally never be used. * chore: remove extraneous comment * chore: make the CLAUDE.md more concise * chore: clean up bloated CONTRIBUTING.md additions * chore: make Makefile more concise * ci: add the lint-budget-update target CLAUDE.md references CLAUDE.md tells contributors to run make lint-budget-update, but the target was never defined. Add it as an aggregate that re-captures the ruff, mypy, and basedpyright budgets in one shot. * ci: recapture mypy and basedpyright budgets in the lint env The per-rule baselines were captured in a richer dependency env than the CI lint job's uv sync --frozen, so CI resolved fewer types and reported more errors than the budgets allowed (no-any-return 902 over cap 900, plus several basedpyright reportUnknown* rules). Regenerate both in the frozen env so they grandfather the true CI debt: mypy 5786 -> 5799 (no-any-return 890 -> 902, valid-type 1 restored), basedpyright 146213 -> 148942. * ci: check out PR head sha in lint and any-discipline jobs The default pull_request checkout uses refs/pull/N/merge, which folds the latest base commits into HEAD. The diff-based gates (ruff delta, Any discipline) then diff against the event's older base.sha and blame base's own new commits on this branch; staging's otel-v2 and streaming changes (#30326, #30485) tripped the Any gate on files this branch never touched. Checking out the PR head sha makes the gates diff the real branch tip against base, and pins the tree the mypy/basedpyright budgets were captured against so their counts stay deterministic as the base advances. * ci(lint): renumber Any-typed-value rule LIT002 -> LIT009 Free up LIT002 for the sibling type-discipline gate (check_type_discipline.py, #30500), which groups its mutable-collection family at LIT001 (annotation) and LIT002 (construction). This gate's Any-typed-value rule moves to LIT009 so the shared LIT namespace stays contiguous with no holes; LIT000 and LIT005 are unchanged. * style: rename lint-strict-budget -> lint-ruff-budget * ci: harden type-check gates against silent passes (greptile review) type_check_gate.py: refuse to certify a vacuous run. The CI pipe swallows the tool's exit code ('tool || true'), so a crashed mypy/basedpyright that emits nothing would parse to zero errors, breach no ceiling, and pass. is_vacuous_run() now fails when nothing was parsed but the budget expects errors. Also wrap basedpyright's json.loads in a JSONDecodeError handler that prints the offending output instead of dumping a raw traceback. check_any_discipline.py: ALL_LINES was None, which dict.get() also returns for a path absent from the line map, so a path-normalisation mismatch could let a violation on an unchanged file pass the scope filter. Make ALL_LINES a distinct sentinel object so 'whole file' and 'path missing' are unambiguous. Adds tests for all three. --------- Co-authored-by: Sameer Kankute <[email protected]> Co-authored-by: Cursor <[email protected]> Co-authored-by: Yassin Kortam <[email protected]> Co-authored-by: Joel Tony <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: hcl <[email protected]> Co-authored-by: ztko <[email protected]> Co-authored-by: Nahrin <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: Humphrey <[email protected]> Co-authored-by: kursadlacin <[email protected]> Co-authored-by: kursad <[email protected]> Co-authored-by: Dushyant Acharya <[email protected]> Co-authored-by: Yuriy <[email protected]> Co-authored-by: Recep S <[email protected]> Co-authored-by: Moshe Malawach <[email protected]> Co-authored-by: Moshe Malawach <[email protected]> Co-authored-by: Rongkun Yan <[email protected]> Co-authored-by: Varshith <[email protected]>

* feat(ui): gate "Default Credentials" hint on /ui/login behind env flag (#30234) Adds LITELLM_HIDE_DEFAULT_CREDENTIALS_HINT (and an equivalent general_settings.hide_default_credentials_hint) that suppresses the "By default, Username is admin and Password is your set LiteLLM Proxy MASTER_KEY" info card rendered on /ui/login and /fallback/login. Motivation: in production deployments operators set UI_USERNAME / UI_PASSWORD (or SSO), and the hardcoded hint becomes factually incorrect and is flagged by security scanners (Tenable WAS plugin 114625) as information disclosure. There is currently no way to suppress it without forking the dashboard. Behaviour: - Default is unchanged (hint shown), so existing deployments are unaffected. - New field hide_default_credentials_hint on the well-known UI config endpoint, populated from the env var or general_settings. - LoginPage.tsx conditionally renders the Alert based on the flag. Refs: BerriAI/litellm#30232 * fix(router): clean pattern_router state on upsert/delete (#29601) * fix(router): clean pattern_router state on upsert/delete PatternMatchRouter.add_pattern was append-only, and neither Router.upsert_deployment nor Router.delete_deployment removed the existing entry. Rotated-out api_keys stayed in the routing rotation for wildcard deployments (model_name with `*`) until proxy restart, silently defeating key rotation as an admin operation. The same leak applied to provider_default_deployment_ids and per-team pattern routers, and the patterns list grew unboundedly on every edit * test(router): direct unit tests for _remove_deployment_from_wildcard_state router_code_coverage.py greps test files for AST Call nodes and flagged the helper as untested because the existing coverage only exercised it transitively through upsert/delete. Adds two direct tests that pin the helper's contract (cleans across global pattern router, per-team routers with empty-router pop, and provider_default_deployment_ids; noop on falsy model_id) * fix(router): address Greptile review on pattern_router cleanup Widen PatternMatchRouter.remove_deployment annotation to Optional[str]; the implementation already handles None via the falsy guard and the unit test exercises it directly. Move _remove_deployment_from_wildcard_state up one level in upsert_deployment so it runs whenever the prior deployment is on the router, not only when the model_id is present in the fast-mapping index. The scenario is currently unreachable (get_deployment shares the same index), but the cleanup is idempotent so this is defensive against any future divergence between those code paths. * fix(router): widen _remove_deployment_from_wildcard_state to Optional[str] Moving the call out of the inner `deployment_id in deployment_fast_mapping` block in the previous commit lost mypy's narrowing of `deployment_id` from Optional[str] to str, tripping the lint CI. The helper already handles None via its falsy guard, so widening the annotation matches the actual contract. * fix(router): make delete_deployment wildcard cleanup symmetric with upsert After the previous commit moved _remove_deployment_from_wildcard_state out of the inner index-map guard in upsert_deployment, delete_deployment was still calling it only inside `if deployment_idx is not None`. Greptile flagged the asymmetry: under a desynced index_map, delete would silently leave the stale wildcard credential in pattern_router. Moves the cleanup call to the top of the try block, mirroring the upsert path. Cleanup is idempotent so the change is a no-op on the happy path. Adds a regression test that simulates the desync by removing the entry from model_id_to_deployment_index_map and asserts delete still clears pattern_router. * fix(pricing): add 1h cache-write cost for Anthropic Sonnet 4.5/4.6 (#30474) The native anthropic claude-sonnet-4-5/4-6 price-map entries were missing cache_creation_input_token_cost_above_1hr (and the >200K long-context sub-tier for 4.5), so 1-hour-TTL cache writes were costed at the 5-minute rate. Adds 6e-06 regular (and 1.2e-05 long-context) = 2x base input, matching the vertex_ai/azure_ai/bedrock siblings and the older claude-sonnet-4-20250514 entry. Adds a regression test. * fix(proxy): cancel upstream gemini request and release httpx connection on client disconnect (#30075) * fix(proxy): cancel upstream gemini request and release httpx connection on client disconnect - add _check_request_disconnection to common_request_processing; wrap llm_call as asyncio.Task so it can be cancelled; catch CancelledError and raise HTTPException(499) when client disconnects before LLM responds (non-streaming path) - pass raw httpx.Response into ModelResponseIterator in make_call/make_sync_call so the iterator holds a reference to the underlying connection - implement ModelResponseIterator.aclose() and .close(): close the line iterator then explicitly call response.aclose()/response.close() to release the httpx connection when the client drops mid-stream; errors are debug-logged, not raised - add tests for _check_request_disconnection (cancels task, graceful on exception, does not cancel when client stays connected) and base_process_llm_request 499 behavior; add TestModelResponseIteratorCleanup verifying aclose/close propagation through CustomStreamWrapper * fix(proxy): record 499 on streaming disconnect and cancel orphaned gather tasks Wire streaming generator cleanup to log client_disconnected with error_code 499 in spend logs, cancel pending during_call_hook tasks when the LLM call is cancelled on disconnect, and align the 600s poll limit comment with proxy_server. * fix: extract client disconnect logging helper to satisfy PLR0915 * fix: resolve mypy and code-quality CI failures for client disconnect logging Cast client disconnect error_information for mypy, only await pending gather tasks to avoid masking LLM errors, and add tests for the new logging helper and gather cleanup. * fix(proxy): harden gather cleanup so finally cannot mask LLM errors * fix(proxy): shield streaming disconnect logging and strip spoofable metadata Move streaming disconnect recording into a shielded cancel scope, add gather cleanup regression coverage for guardrail-converted cancels, and strip client_disconnected/error_information from user metadata at the proxy boundary. * fix(proxy): only map CancelledError to 499 for client disconnect Track when the disconnect poller cancels the LLM task and re-raise other CancelledError paths so graceful shutdown is not reported as HTTP 499. * fix(proxy): remove dead _check_request_disconnection helper Non-streaming client disconnect is handled by staging's cancel_on_disconnect path via _await_llm_call_cancelling_on_disconnect. Drop the unused is_disconnected poller and its unit tests; rename the remaining integration tests to TestDisconnectGatherCleanup. * feat(mistral): add mistral-medium-3-5 to model_prices_and_context_wind.. (#29303) * feat(mistral): add mistral-medium-3-5 to model_prices_and_context_window.json Mistral's docs page lists mistral-medium-3-5 as a new model offering. Pricing/specs sourced from Mistral's published model metadata: - input: $1.50 / 1M tokens - output: $7.50 / 1M tokens - context: 262,144 tokens - capabilities: vision, function calling, structured outputs, assistant prefill Adds entry: `mistral/mistral-medium-3-5`, mirroring the pattern used for the rest of the Mistral family. test(mistral): add model_info test for mistral-medium-3-5 + sync backup cost map - Mirror mistral/mistral-medium-3-5 entries into litellm/model_prices_and_context_window_backup.json so the bundled model cost map matches the canonical model_prices_and_context_window.json. - Add tests/test_litellm/test_mistral_medium_3_5_model_metadata.py covering pricing tiers, capability flags, context window, provider routing, and parity between the main and backup cost maps. - Point 'source' at the live Mistral models documentation page. * fix(ui): three small UI fixes — Gemini api_base + credential form reset + Mode badge (#30419) * fix(ui): three small UI fixes — Gemini api_base field + credential form reset + Mode badge Three independent fixes; bundled because they all touch the credential-form / logging-callbacks area. 1. expose api_base field on Google AI Studio credential form The runtime gemini provider supports custom api_base via `vertex_llm_base._check_custom_proxy`; the UI just needs to expose the field. Adds api_base to the Google_AI_Studio credential form ordered before api_key (matching OpenAI/Anthropic conventions). Default value matches the canonical Google AI Studio endpoint that LiteLLM's gemini provider talks to when api_base is unset, so leaving the default in the form behaves identically to leaving it blank. 2. reset credential form state when switching providers Switching the Provider select in AddCredentialModal / EditCredentialModal left the previous provider's field values populated. The form then submitted a mixed payload (e.g. Azure deployment fields under an OpenAI credential), producing confusing failures. Extract `getProviderFieldDefaults` helper and reset the form to it on provider change. Unit-tested via the extracted helper because Antd Select's portal/dropdown behaviour is unreliable in jsdom. 3. logging callbacks table reads backend `type` for Mode badge (#35) The `/get_callbacks` proxy endpoint returns each callback as `{name, type, variables}` where `type` is `"success"` or `"failure"`. The same callback name can appear twice (one per event class) and the two entries fire on disjoint events. `LoggingCallbacksTable` ignored `type` and read `record.mode` (always undefined), so every row fell back to the "Success" badge. A `generic_api` callback registered for both classes showed up as two identical "Success" rows + React duplicate-key warning. Read `record.type` first (fall back to `record.mode` for newly- added not-yet-server-acknowledged rows). Composite rowKey `${name}-${type ?? mode ?? 'success'}`. Removed leftover debug `console.log`. * fix(ui): drop api_base default_value to preserve Gemini v1alpha auto-routing Greptile P2 (PR #30419, threads on lines 1255-1256 of provider_create_fields.json): the api_base field's `default_value` was hard-coded to "https://generativelanguage.googleapis.com/v1beta". This: 1. Bakes v1beta into every credential record saved through the form, even when the user never touched the field. If LiteLLM's internal gemini default URL ever changes, those persisted credentials keep hitting the stale path. 2. Bypasses `_get_gemini_url`'s automatic version routing for Gemini 3+ models. That helper picks v1alpha for Gemini 3+ and v1beta for older models when api_base is unset. With the default pre-filled (and `_check_custom_proxy` then taking over because api_base is non-empty), Gemini 3+ requests get pinned to v1beta and may fail or behave unexpectedly — purely because the user accepted the visible default. Fix: set `default_value` to `null` and move the canonical URL guidance into the `placeholder` (visible to the user, never persisted) and an expanded tooltip. UX is unchanged — the URL is still shown in the greyed-out input — but the auto-version-routing path stays default. Updated test_google_ai_studio_provider_fields_expose_api_base to assert the new contract (`default_value is None`, `placeholder` carries the canonical URL), with a comment pointing at the Greptile threads as the rationale so future contributors don't accidentally re-introduce the default. 26/26 tests in the file pass. JSON validates (`json.load` clean). * feat(azure_ai): add gpt-5.5 to model cost map (#30428) * feat(azure_ai): add gpt-5.5 to model cost map Adds azure_ai/gpt-5.5 and its dated snapshot azure_ai/gpt-5.5-2026-04-23 to both the canonical and bundled cost maps. gpt-5.5 is generally available on Azure AI Foundry; pricing mirrors the openai gpt-5.5 entry, matching the established azure_ai convention (verified identical for gpt-5.4), in the azure tier structure (base / above-272k / priority). supports_minimal_ reasoning_effort is false, the capability that changed from gpt-5.4. Fixes #30306 * Update tests/test_litellm/test_gpt_5_5_model_metadata.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: guard check_and_fix_namespace against None key (#30435) * fix: guard check_and_fix_namespace against None key When user_id is None, the cache key can be None, causing AttributeError: 'NoneType' object has no attribute 'startswith' in check_and_fix_namespace. Add an early return for None key to prevent the error and the ERROR-level log noise it produces on every unauthenticated request. Fixes #30424 * fix: update type annotations for check_and_fix_namespace - key: str -> Optional[str] (now handles None input) - return: str -> Optional[str] (returns None when input is None) Addresses Greptile review concern about type signature mismatch. * fix: revert check_and_fix_namespace type signature to str to fix MyPy downstream errors * fix: update type annotations for check_and_fix_namespace - Change signature from str -> str to Optional[str] -> Optional[str] - Remove type: ignore comment on None return - Add None guard in async_set_cache_sadd before passing to helper Addresses review feedback from Sameerlite on type mismatch. * Revert "fix: update type annotations for check_and_fix_namespace" This reverts commit 5272920fa0daab676f5ad46dcadd8cd537cfc96f. --------- Co-authored-by: michaelxer <[email protected]> * fix(cost): apply service_tier suffix to above-threshold cache rates and expose priority+threshold keys in ModelInfo (#30450) * fix(cost): apply service_tier suffix to above-threshold cache rates and expose priority+threshold keys in ModelInfo Models that publish both a service_tier (e.g. priority) rate and an above-threshold tier (e.g. _above_200k_tokens) currently bill cached tokens at the standard above-threshold rate rather than the priority above-threshold rate. Affected entries in the live pricing JSON include gemini-3-pro-preview, gemini-3.1-pro-preview and their vertex_ai/ and gemini/ variants, plus azure/gpt-5.4 and azure_ai/gpt-5.4. For a 250K-token priority request with 200K cached tokens against gemini-3-pro-preview, the leak is about 44 percent of the prompt cost. Two stacked defects caused this. First, ModelInfoBase (and the ModelInfo pydantic class) and the get_model_info construction in litellm/utils.py omit the priority+above-threshold cost keys, so even if the calculator asked for them they would never reach it. Second, in _get_token_base_cost the cache_creation/cache_read tiered keys never get wrapped with _get_service_tier_cost_key, while the input/output tiered keys above and below do. The change here surfaces six new keys (input, output and cache_read at both 200k and 272k priority variants) and wraps the three cache tiered keys in _get_token_base_cost the same way input/output already are. _get_cost_per_unit's existing service_tier-to-base fallback covers models that ship the standard above-threshold rate without a priority variant. Adds one regression test in tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py that drives the actual generic_cost_per_token path for gemini-3-pro-preview at 200K cached + 50K text under priority and asserts the priority above_200k rates are picked. Verified the test fails on litellm_internal_staging without these changes and passes with them. * fix(cost): drop guard on cache tiered keys so service_tier fallback can reach standard above-threshold rate Addresses Greptile P1 on PR 30450. The previous commit wrapped cache_creation_tiered_key, cache_creation_1hr_tiered_key, and cache_read_tiered_key with _get_service_tier_cost_key (matching how the sibling input and output tiered keys are wrapped) but kept the surrounding 'if key in model_info' guards. For models that publish a standard above-threshold cache rate but no priority variant (gpt-5.4-pro, gpt-5.5-pro and their dated siblings, plus vertex_ai/claude-sonnet-4-5 for cache_creation), the guard short-circuits before _get_cost_per_unit's existing service_tier-to-base fallback can strip _priority and find the standard above-threshold key. The result on priority requests over the threshold was that those models silently dropped from the above-threshold rate back to the priority-base rate. Dropping the guard and calling _get_cost_per_unit unconditionally (mirroring how tiered_input_key and tiered_output_key are already handled) restores correct billing for that class of models while keeping the new priority+above-threshold behaviour for gemini-3-pro-preview and friends. Adds a second regression test that pins generic_cost_per_token for vertex_ai/claude-sonnet-4-5 priority + above_200k with cached and cache_creation tokens to the expected standard above-threshold rates, so the guard cannot be silently reintroduced for either the cache_read or cache_creation path. * fix(presidio): skip pre-call masking when guardrail is logging_only (#30461) The Presidio pre-call hook masked the live request unconditionally, ignoring the configured event hook. With mode: logging_only the masked request reached the model, so its response echoed anonymization tokens (e.g. <PERSON>) instead of the real output. Gate async_pre_call_hook on should_run_guardrail, matching every other guardrail; logging_only masking still happens via async_logging_hook. * fix(router): resolve list unhashable crash on model alias (#30464) * fix(router): resolve list unhashable crash on model alias Fixes the fallback parsing logic that mistakenly categorized standard array fallback definitions as override dictionaries when a deployment alias matches the literal string 'model'. Closes https://github.com/BerriAI/litellm/issues/30459 * fix(router): address greptile review for fallback parsing edge cases - Resolves ambiguity in standard vs override fallback dictionaries by iterating over all items and validating that no mapped litellm param resolves to a non-list type. - Adds regression tests in test_router_order_fallback.py to prevent unhashable type crash from silently re-entering the codebase. * chore(router): format code with black to pass CI * fix(hosted_vllm): remove thinking_blocks and convert list content to strings (#30475) * fix: hosted_vllm remove thinking_blocks and convert list content to strings vLLM endpoints reject assistant messages with thinking_blocks converted to content list blocks. This change removes thinking_blocks entirely and converts any list content back to strings. This fixes BadRequestError when using Claude Code with hosted_vllm models that pass thinking_blocks in messages. * fix(hosted_vllm): address Greptile review feedback - Join multiple text blocks with newline instead of empty string - Always set content to string (never None) to avoid vLLM validation errors * fix(hosted_vllm): update chat transformation to clean assistant messages * fix: re-raise exception instead of silently dropping MCP team permissions (#30477) * fix: re-raise exception instead of silently dropping MCP team permissions When MCPRequestHandler.get_allowed_mcp_servers raises, the broad except was swallowing the error and returning only allow_all_server_ids, silently discarding all team-level object_permission grants. Fixes #30476 * fix: log full traceback when MCP permission lookup fails Uses verbose_logger.exception() instead of warning() so operators can see the full traceback when team-level object_permission grants are dropped due to an internal error in get_allowed_mcp_servers. Fixes #30476 * fix: remove timezone date expansion in daily-activity aggregation (#29569) * fix: remove timezone date expansion in daily-activity aggregation Single-day spend queries from non-UTC timezones over-counted by ~2x because the previous implementation widened the SQL date range by a full UTC day on whichever side the offset pointed. Spend is bucketed in whole-UTC-day rows in LiteLLM_DailyUserSpend, so the expansion pulled an extra 24h of unrelated bucket data per boundary. Concretely on IST (UTC+5:30, offset -330): a single-day query for 2026-05-29 was rewritten to date >= 2026-05-28 AND date <= 2026-05-29 and returned spend across both UTC days. Sums of single-day queries across a 5-day window then exceeded the equivalent multi-day aggregate by ~50%, which is mathematically impossible. Treat the local date range as the UTC date range. The aggregation table has no hour-level granularity, so any conversion using only date arithmetic must round to whole UTC days; the previous fix turned that boundary slop into systematic over-counting. Pass-through trades a small one-time slop at each end of the range for correct, monotonic, additive results across single-day and multi-day queries. Repro from production: bedrock/global.anthropic.claude-opus-4-8 over 2026-05-29 to 2026-06-02, IST timezone: - 5-day aggregate: $701.39 / 1,831 reqs - Sum of 5 single-day queries: $1,070.94 / 2,755 reqs - Excess (was 1.527x): now matches within boundary slop Adds regression tests in TestAdjustDatesForTimezone and TestBuildAggregatedSqlQuery that pin the pass-through behavior and the additivity invariant for any future implementation. * ci: rerun checks on litellm_oss_branch base --------- Co-authored-by: Sameer Kankute <[email protected]> * fix: buffer native gemini sse frames (#30225) * fix: buffer native gemini sse frames * fix: scope native gemini sse buffering * fix: check raw sse residual buffer size * feat: updated openrouter provider to map max level to xhigh (#28881) * feat(proxy): allow use_redis_transaction_buffer without redis cache (#28764) * feat(proxy): allow use_redis_transaction_buffer without redis cache * fix(proxy): require host or url for standalone buffer redis * fix(mcp): fail closed when scope filter resolves to no servers (#30353) `_get_allowed_mcp_servers_from_mcp_server_names` returned the caller's full allowed-server set when the requested `mcp_servers` list (path- or header-derived) resolved to nothing. URL/header namespacing therefore appeared to work even when the requested name was unknown or the caller had no grant — `/mcp/<typo>/` silently exposed every server the key could reach. Fail closed instead: when `mcp_servers` is explicitly provided but nothing resolves, return an empty list. The `mcp_servers=None` path (no scope requested) keeps its existing behavior. Co-authored-by: Claude Opus 4.7 <[email protected]> * fix(token-counter): handle Anthropic tool_reference blocks to stop dropped spend logs (#30302) * fix(token-counter): handle Anthropic tool_reference blocks to stop dropped spend logs `token_counter` did not know about Anthropic tool-search `tool_reference` content blocks, a lightweight pointer to a deferred tool that shows up as `{"type": "tool_reference", "tool_name": ...}`. When such a block appeared in message content, `_count_content_list` fell through to its catch-all branch and raised `Invalid content item type: tool_reference`. On the streaming `anthropic_messages` proxy path that exception nulls `response_cost`, which makes the proxy drop the entire SpendLogs row. The result is a silent cost undercount on any tool-search traffic; the request succeeds for the caller but the spend is never recorded. This adds a `tool_reference` branch that counts the referenced `tool_name` (the full tool definition is already counted via the `tools` param, so only the name is added here) and handles an empty/missing name gracefully. The catch-all error message is updated to list `tool_reference` among the expected types. A regression test asserts that a message containing a `tool_reference` block no longer raises and returns a positive token count, and that an empty `tool_name` is handled without error. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * fix(token-counter): collapse explicit None tool_name to empty string In _count_content_list, c.get("tool_name", "") returns None when the key is present with an explicit None value, and str(None) == "None" which is truthy, causing a spurious token to be counted. Use c.get("tool_name") or "" so both a missing key and an explicit None collapse to an empty string and are skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * test(token-counter): cover catch-all for unknown content block type Adds a regression test that calls `_count_content_list` with an unrecognized content block type and asserts it raises `ValueError` whose message names the offending type and lists `tool_reference` among the supported types. This exercises the previously uncovered catch-all branch (codecov patch gap) and pins the error contract. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * test(token-counter): cover tool_reference on the spend/cost and streaming paths Adds end-to-end regression tests that exercise the real public entry points (`completion_cost` and `stream_chunk_builder`), not just the private `_count_content_list` helper, for Anthropic tool-search `tool_reference` content blocks. These pin the actual bug the fix addresses: before the fix the `tool_reference` block raised out of `completion_cost` -> the proxy logging layer nulled `response_cost` and the spend callback dropped the SpendLogs row (silent cost undercount on all tool-search traffic); and `stream_chunk_builder` swallowed the same raise and collapsed prompt_tokens to 0. With the fix, cost is positive and prompt_tokens are counted. Verified: 3 fail without the fix, 3 pass with it. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> * feat(cost): add cost mapping for deepseek-v4-flash and deepseek-v4-pro (#27056) * feat(cost): add cost mapping for deepseek-v4-flash and deepseek-v4-pro Adds pricing entries for the two new DeepSeek V4 models released on 2026-04-24, for both bare model names and the deepseek/ provider prefix. Prices sourced from https://api-docs.deepseek.com/quick_start/pricing: - deepseek-v4-flash: $0.14/M input, $0.28/M output - deepseek-v4-pro: $1.74/M input, $3.48/M output Cache hit price set to 1/10 of input (per DeepSeek docs). Context window: 1M tokens for both models. Closes #26709 * fix(cost): update backup registry for deepseek-v4 * style: remove print statement from deepseek-v4 test * feat(cost): add cost mapping for deepseek-v4-flash and deepseek-v4-pro Adds pricing entries for the two new DeepSeek V4 models released on 2026-04-24, for both bare model names and the deepseek/ provider prefix. Prices sourced from https://api-docs.deepseek.com/quick_start/pricing: - deepseek-v4-flash: $0.14/M input, $0.28/M output - deepseek-v4-pro: $1.74/M input, $3.48/M output Cache hit price set to 1/10 of input (per DeepSeek docs). Context window: 1M tokens for both models. Closes #26709 * fix: update deepseek-v4 prices to active discounted rates * test: update deepseek-v4 prices in tests to match active discounted rates * fix(deepseek): remove duplicate entries and update backup registry to active discounted rates * fix: update max_output_tokens to 384K for deepseek-v4 * fix: correctly restore upstream models accidentally dropped during merge * fix(tests): resolve failing claude-fable-5 and reasoning tests by safely updating cost map - Pulled the latest cost map from upstream staging - Safely appended deepseek-v4 mapping without deleting duplicate keys or formatting via json.dump * fix(tests): correct deepseek model cache prices and update JSON schema - Appended both prefixed and bare deepseek-v4 models to satisfy test assertions - Corrected deepseek-v4-pro expected cache hit and token prices based on latest review updates - Added missing realtime endpoint to test_utils.py INTENDED_SCHEMA * fix: remove accidental azure/gpt-realtime-whisper addition --------- Co-authored-by: Dushyant Acharya <[email protected]> * feat(key/info): expose per-model budget usage in /key/info response (#30394) * feat(key/info): expose per-model budget usage in /key/info response Add model_max_budget_usage to /key/info and /v2/key/info responses. For each model in model_max_budget, reads current-period spend from the same DualCache used by the budget enforcer and returns it alongside the limit and time period so callers can see how much of each model budget has been consumed in the active window. * test(key/info): add coverage for model_max_budget_usage in v1 and v2 endpoints Add tests for the model_max_budget_usage enrichment in both info_key_fn and info_key_fn_v2, covering the budget-present path, the empty-budget path, and the v2 batch endpoint. * fix(key/info): source model_max_budget current_spend from SpendLogs instead of DualCache The DualCache used for enforcement is ephemeral and only populated when budget metadata is present at request time. Fall back to a direct LiteLLM_SpendLogs DB aggregation using the budget period window (budget_reset_at - budget_duration) for accurate reporting. Also fall back to litellm_budget_table.model_max_budget when the key's top-level field is empty, and round current_spend to 4 decimal places. * test(key/info): cover remaining branches in model_max_budget_usage helpers Add unit tests for: prisma_client=None early return, DB query exception swallowing, invalid budget_duration handled by _compute_budget_period_start, budget_reset_at received as a datetime object (Prisma native type), max_seconds=0 early return, and skipping models that lack a budget_duration. Also remove an unreachable except branch where fromisoformat would fail after _compute_budget_period_start already validated the same value. * test(key/info): cover except path for unparseable per-model budget_duration * fix(key/info): compute per-model rolling windows in model_max_budget_usage Each model in model_max_budget now gets its own time window derived from its own budget_duration, rather than sharing a single window computed as the max (or the budget table's reset_at). This matches what the DualCache enforcer actually tracks and prevents current_spend from being inflated for models with shorter windows. _query_model_spend_for_period is refactored to accept a model filter (handling provider-prefix variants in SQL) and return a float directly. _compute_budget_period_start and the budget_table window path are removed as they are no longer needed. * refactor(model_max_budget_limiter): remove dead get_current_period_spend method * refactor(key/info): strip synthetic formatter noise from PR diff Restore key_management_endpoints.py and test_key_management_endpoints.py to origin/litellm_internal_staging, then re-apply only the intentional additions: _query_model_spend_for_period, _build_model_max_budget_usage, the two endpoint patches (info_key_fn / info_key_fn_v2), and the new test suite. The previous commits had reformatted ~300 pre-existing lines across both files, making the functional diff unreadable. * test(key/info): cover empty-rows path in _query_model_spend_for_period * fix(model_max_budget_limiter): guard BudgetConfig construction inside try/except A malformed model entry in the DB (e.g. non-numeric max_budget from a manually edited or migrated row) caused BudgetConfig(**budget_info) to raise a Pydantic ValidationError outside any exception guard, surfacing as a 500 for the entire /key/info or /v2/key/info call. Merging both try/except blocks into one ensures bad entries are silently skipped, consistent with the existing duration_in_seconds guard. * fix: don't stack provider prefix on wildcard models with a custom prefix (#30360) * fix: don't stack provider prefix on wildcard models with a custom prefix get_known_models_from_wildcard expanded provider-prefixed model ids (e.g. "ollama/gemma3:1b" from get_provider_models) by prepending the wildcard's prefix whenever the id did not already start with it. With a custom wildcard prefix such as "ollama_server1/*" (used to distinguish multiple Ollama instances), this produced "ollama_server1/ollama/gemma3:1b", which is uncallable and breaks /v1/models. When the expanded id already carries a provider prefix, replace it with the wildcard's prefix instead of stacking both. Matching-prefix and bare-model cases are unchanged. Fixes #30358 * fix: only strip a known provider prefix when expanding custom wildcard prefixes The wildcard expansion replaced the leading slash segment of every expanded id with the wildcard prefix whenever the id did not already start with it. For ids whose first segment is an org rather than a litellm provider (for example a provider returning "meta-llama/Llama-3-8B" with no outer provider prefix), that dropped the org and produced an uncallable id Only strip the leading segment when it is a recognized provider (membership in LlmProviders); otherwise keep it and just prepend the wildcard prefix. Provider-prefixed ids like "ollama/gemma3:1b" still have their prefix replaced, so the original fix is unchanged for known providers * address greptile review feedback: log dropped non-text vLLM assistant content blocks (greploop iteration 1) * fix(ci): format credential_form_helpers test + regenerate dashboard schema.d.ts * fix(proxy): raise litellm.BadRequestError for missing model param When no model is passed, route_request now raises a litellm.BadRequestError ('Missing model parameter') instead of falling through to ProxyModelNotFoundError. This keeps the missing-param error clear and independent of router wildcard state. Unknown (non-empty) model names still raise ProxyModelNotFoundError. * Revert "fix(proxy): raise litellm.BadRequestError for missing model param" This reverts commit 9240da403c0432a80473d6c4677ddb7e2bad7420. * Revert "fix(router): clean pattern_router state on upsert/delete (#29601)" This reverts commit ad4e6e2395620ea6d2fe38089a54cde160720de2. * fix: correct streaming and key budget usage reporting * fix(hosted_vllm): type assistant tool_calls to satisfy mypy * feat: aws secret manager cross region replication (#30368) * feat(aws-secret-manager): add replica_regions cross-region replication after CreateSecret When store_virtual_keys is enabled, async_write_secret() only wrote secrets to the primary AWS region. Multi-region proxy deployments had no built-in way to synchronize virtual key secrets across regions through LiteLLM, requiring external replication mechanisms. Add replica_regions support to AWSSecretsManagerV2: - New replica_regions field in KeyManagementSettings (types/secret_managers/main.py) - New async_replicate_secret() method that calls ReplicateSecretToRegions API - async_write_secret() calls replication after successful CreateSecret - Replication failure is logged as a warning but does NOT fail key creation - load_aws_secret_manager() forwards replica_regions from key_management_settings Configuration example: key_management_settings: store_virtual_keys: true replica_regions: - us-west-2 - eu-west-1 When replica_regions is omitted or empty, behavior is unchanged. * test(aws-secret-manager): restore litellm.secret_manager_client after test to prevent state pollution * test(aws-secret-manager): add coverage for HTTP error and replication exception paths * fix: restore litellm.secret_manager_client global state in test; add replication log proof - Global state in test_load_aws_secret_manager_passes_replica_regions was already guarded with try/finally (committed in previous pass); no further change needed for Fix 1. - Fix 2: add verbose_logger.info("ReplicateSecretToRegions called …") inside async_replicate_secret so callers get an observable INFO log line whenever replication fires. - Add test_replication_fires_on_create: calls async_replicate_secret directly with caplog.at_level(INFO, logger="LiteLLM") and asserts "ReplicateSecretToRegions" appears in the captured log output, proving the code path executes. * fix: pass request to streaming generators * fix(hosted-vllm): preserve assistant structured content * fix(hosted_vllm): satisfy mypy on preserved structured content assignment * chore: resolve litellm_internal_staging merge conflicts for #30527 (#30554) * chore(codecov): add Batches, Videos, and Realtime components (#30517) * chore(codecov): add Batches, Videos, and Realtime components Define per-feature Codecov components so PR comments track coverage for batch API, video generation, and realtime streaming paths. Co-authored-by: Cursor <[email protected]> * chore(codecov): use wildcard path for Batches proxy component Align batches_endpoints glob with Videos, Realtime, and Proxy_Authentication. Co-authored-by: Cursor <[email protected]> --------- Co-authored-by: Cursor <[email protected]> * test(batches): move orphan tests into tests/test_litellm for CI coverage (#30510) Four batch-related tests lived under tests/litellm/ and were never picked up by GitHub Actions. Relocate them and fix gemini multimodal e2e to use the batchEmbedContents path expected for gemini/ provider. Co-authored-by: Cursor <[email protected]> * fix(guardrails): run pre_call hook once for model-level guardrails (#30543) * fix(guardrails): run pre_call hook once for model-level guardrails A CustomGuardrail attached to a deployment via litellm_params.guardrails gets its async_pre_call_hook invoked twice per request: once by the proxy pre-call loop and again by async_pre_call_deployment_hook after the router spreads the model-level guardrails into the top-level request kwargs. Record in request metadata that the proxy pre-call loop already ran a given guardrail, and have the deployment hook skip it when the marker is present. Direct-SDK usage never runs the proxy loop, so the deployment hook stays the sole invocation there and still fires exactly once. The marker key is stripped from untrusted caller metadata so a request body cannot suppress a model-only guardrail by pre-seeding it. * fix(guardrails): mark pre_call dedup on the post-hook request data Record the exactly-once marker after async_pre_call_hook runs, on the data object that flows downstream, rather than before it. A guardrail whose hook returns a brand-new request dict (instead of mutating or spreading the one it received) would otherwise discard the marker, letting the deployment hook re-run the guardrail a second time. * fix(guardrails): stop re-initializing DB guardrails on every poll (#30542) * fix(guardrails): stop re-initializing DB guardrails on every poll InMemoryGuardrailHandler._has_guardrail_params_changed compared the in-memory LitellmParams against the raw dict loaded from the DB. The in-memory side carries every field default and coerces enums via model_dump(), while the DB side only holds the keys originally stored, so the two shapes never compared equal and the guardrail was rebuilt on every poll cycle. Each rebuild created a fresh instance, but delete_in_memory_guardrail only removed the old callback from litellm.callbacks. Request handling promotes guardrail callbacks into the success/failure/async lists, so the previous instance stayed referenced there and instances accumulated. Normalize both sides through LitellmParams(...).model_dump() before diffing, and purge the callback from every callback list on delete. * refactor(guardrails): narrow params-normalization fallback to ValidationError The comparison normalizer caught a bare Exception and silently fell back to the raw dict, which hid the cause and quietly degraded the affected guardrail back to re-initializing on every poll. Catch only the ValidationError that LitellmParams construction can raise, log a warning so the offending row is diagnosable, and let any other error surface instead of being swallowed. * refactor(callbacks): add remove_callback_from_all_lists helper to manager Move the knowledge of which callback lists a callback can be promoted into out of the guardrail registry and into LoggingCallbackManager, where the rest of the callback-list bookkeeping already lives. delete_in_memory_guardrail now delegates to the new helper instead of iterating the lists itself. * chore(oss): litellm oss staging 150626 (#30463) * fix(pricing): add GitHub Copilot MAI Code Flash pricing (#30415) * fix(pricing): add GitHub Copilot MAI Code Flash pricing Add GitHub Copilot pricing entries for MAI-Code-1-Flash and the internal Copilot CLI model name so cost calculation can price input, cached input, and output tokens. Co-authored-by: Copilot <[email protected]> * test(pricing): cover GitHub Copilot MAI Code Flash pricing Add regression coverage for both GitHub Copilot MAI-Code-1-Flash model names, including cached input pricing, chat endpoint metadata, and cost_per_token arithmetic. Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: Copilot <[email protected]> * fix(router/proxy): propagate completed_response through FallbackResponsesStreamWrapper for streaming /v1/responses container ownership (#30210) (#30213) * fix(router/proxy): propagate completed_response through FallbackResponsesStreamWrapper for streaming /v1/responses container ownership (#30210) #28990 added ownership recording for streaming /v1/responses via _wrap_responses_stream_for_container_ownership, which reads `getattr(stream_response, 'completed_response', None)` to extract the ResponsesAPIResponse. The unit test bypassed the Router, so it never exercised the production wrapping path. Through the Router (every proxy deployment), the stream is wrapped by FallbackResponsesStreamWrapper (router.py:2527). Its __init__ set `self.completed_response = None` and __anext__ only forwarded chunks — the inner source iterator's terminal event never bubbled up to the attribute the ownership hook reads, so the hook silently recorded nothing and every follow-up /v1/containers/<id>/files call returned 403 for non-admin keys. This commit: - router.py: pre-resolves the responses-API terminal event tuple (response.completed / .incomplete / .failed) once per _aresponses_streaming_iterator call, and has the wrapper's __anext__ sniff each forwarded chunk's .type. First terminal event hit gets stored on the wrapper's completed_response. Iterator-agnostic — works for source_iterator AND any future wrapper. - common_request_processing.py: when _extract_completed_responses_response returns None we now warn instead of silently skipping. Reporter on #30210 lost a day to this exact silent skip; the warning surfaces future regressions of the same shape directly in operator logs. Fixes #30210 * fix(router): type-ignore wrapper getattr-defaults; broaden ownership-skip warning CI lint (mypy) flagged the three pre-existing getattr(..., None) assignments in FallbackResponsesStreamWrapper.__init__: router.py:2564 self.response = getattr(source_iterator, 'response', None) router.py:2565 self.model = getattr(source_iterator, 'model', None) router.py:2566 self.logging_obj = getattr(..., None) Those lines also exist on litellm_internal_staging and pass mypy there. Adding the typed terminal-event tuple above the class made the function body more narrowable, which surfaced the pre-existing mismatch — base class declares non-Optional types but the bridge path (LiteLLMCompletionStreamingIterator) legitimately omits these. Keep the None fallback and silence with type: ignore[assignment]. Greptile 4/5 note: the ownership-skip warning hard-named code_interpreter which misleads operators when a non-code_interpreter stream aborts. Generalize to 'any tool container (e.g. code_interpreter)'. * fix(register_model): drop synthesized zero costs to preserve sparse entries (#30198) (#30201) * fix(register_model): drop synthesized zero costs to preserve sparse entries (#30198) get_model_info synthesizes input_cost_per_token / output_cost_per_token = 0 when they are absent from the raw entry (the price-unknown and free cases share the same representation). register_model then merges that result back into litellm.model_cost, which flips a sparse entry from 'no cost keys' (priced via model name) to 'cost keys = 0' (free). That defeats _is_cost_explicitly_configured (#24949) on re-registration: _is_model_cost_zero returns True, common_checks skips every tag / key / team / user / org budget check for the group, and over-budget traffic keeps returning 200. Spend keeps recording because cost calc still resolves by model name, so the symptom is silent and only triggers on the second register_model pass (router rebuild, /model/update, config sync). Mirror the existing litellm_provider-None guard one block above and pop the cost fields from the synthesized result when they are absent from the raw entry and not in the caller's value. Caller-provided zeros (genuinely free models, BYOK overrides) are preserved. Fixes #30198 * fix(register_model): switch _raw_entry to is-None checks + drop dead test assertion Greptile #30201 review notes: - the `or`-chain in the raw-entry lookup treated an empty dict (a key with no fields) as falsy and fell through to the second arm — replace with explicit `is None` checks so a present-but-empty entry is still taken at face value. - the first assertion in `test_router_double_init_keeps_db_model_entry_sparse` used `in (None, 0)` which passes under the bug condition (cost = 0 matches the tuple); the strong follow-up assertion already covers every shape, so drop the dead branch. * fix(bedrock mantle): use unique function-call id for responses->chat tool calls (#30426) * fix(bedrock mantle): use unique function-call id for responses->chat tool calls ... * fix(bedrock mantle): scope unique tool-call id fallback to degenerate call_id The previous revision preferred the Responses item id for every tool call, which broke providers (and existing tests) where call_id is a unique, canonical correlation key. Restrict the fallback to the degenerate index-based call_id that Bedrock Mantle returns (call_0, call_1, ... resetting per response) and keep call_id otherwise. Revert the change to the OUTPUT_ITEM_DONE streaming handler, whose tool_call_chunk is never emitted (dead code, per review). Extend the regression tests to assert a normal call_id is preserved. * fix(router): preserve azure_ad_token through CredentialLiteLLMParams for /v1/files + batches (#30235) (#30241) * fix(router): preserve azure_ad_token through CredentialLiteLLMParams for /v1/files + batches (#30235) Router.get_deployment_credentials_with_provider re-validates a deployment's litellm_params through CredentialLiteLLMParams before handing them to file/batch/passthrough callers: return CredentialLiteLLMParams( **deployment.litellm_params.model_dump(exclude_none=True) ).model_dump(exclude_none=True) Any field NOT declared on CredentialLiteLLMParams gets silently dropped on the way through. azure_ad_token was undeclared, so Azure deployments using OAuth/M2M (azure_ad_token instead of a static api_key) silently lost their token at the files endpoint and the proxy returned: Missing credentials. Please pass one of api_key, azure_ad_token, azure_ad_token_provider, ... Declare azure_ad_token on CredentialLiteLLMParams alongside api_key / api_base / api_version so it rides through the round-trip. Static-key deployments stay unaffected (Optional, default None, dropped by exclude_none=True). Provider-callable (azure_ad_token_provider) is a separate concern and out of scope here. Fixes #30235 * fix(ui-types): regenerate schema.d.ts for new azure_ad_token field CI's 'Verify schema.d.ts matches the proxy OpenAPI spec' check auto-detected the new field and emitted the exact diff to apply. Two schemas had `aws_secret_access_key` from CredentialLiteLLMParams, both get the new azure_ad_token marker next to it. * fix(proxy): org_admin with own user_id now sees all org teams on /v2/team/list (#30247) When the UI sends the callers own user_id (as it does for non-Admin global roles), _enforce_list_team_v2_access now nulls it out for org admins so _build_team_list_where_conditions scopes by organization_id only -- matching the legacy /team/list behavior and the documented intent. Fixes #30215 Co-authored-by: Claude Opus 4.6 <[email protected]> * test(vertex_ai): multi-region regression coverage for cachedContents host (#29571) (#29707) litellm_internal_staging already routes the cachedContents URL through get_vertex_base_url, fixing the multi-region 404 reported in #29571 — but carries no test coverage for the actual regression scenario (eu/us must resolve to the REP host aiplatform.{geo}.rep.googleapis.com). Add TestContextCachingMultiRegionUrls: parametrized eu/us REP-host assertions (including absence of the old broken {geo}-aiplatform host), plus regional (us-central1) and global no-regression checks. * fix(proxy): close upstream LLM stream when client disconnects mid-stream (#30245) * fix(proxy): close upstream LLM stream when client disconnects mid-stream When a streaming client disconnects, Starlette abandons the response body iterator without calling aclose(), so the proxy's connection to the upstream backend stays open until garbage collection, which may never come. The backend (e.g. vLLM) keeps generating into a dead pipe: small responses drain invisibly into TCP buffers while large ones block the backend on a full send buffer indefinitely (observed via lsof as an ESTABLISHED proxy->backend connection minutes after the client left) create_response now returns a StreamingResponse subclass that closes both its body iterator and the wrapped upstream-facing generator in a shielded finally. The upstream generator is closed directly rather than through a cascade because aclose() on a never-started generator skips its body, which would make the cascade a no-op when the client disconnects before the first chunk is sent. async_streaming_data_generator also gains the same shielded finally-aclose that async_data_generator in proxy_server.py already had, covering the Anthropic and Google SSE paths With this, killing a streaming client causes the backend to observe the abort within about a second and free its slot, while completed streams are unaffected. No flag is needed, unlike the non-streaming opt-in cancel in #30223: this only releases resources after the client is already gone and does not change any response a client can observe Fixes #30244 * fix(proxy): close upstream even when body iterator aclose raises BaseException Addresses the Greptile finding on #30245: the cleanup loop caught only Exception while the generator-level cleanup catches BaseException, so a CancelledError or GeneratorExit escaping body_iterator.aclose() would skip closing the upstream generator. Both sites now use the same scope and a regression test pins that the upstream is closed even when the body iterator explodes with a BaseException * fix(llms): expose aclose on BaseModelResponseIterator so stream close reaches the provider connection The response-level close added for #30244 only worked for SDK-based providers (e.g. openai), whose streams expose aclose all the way down. Providers served by base_llm_http_handler (hosted_vllm and most modern transformation-based providers) wrap a bare response.aiter_lines() generator in BaseModelResponseIterator, which had no aclose or close at all, and nothing retained the httpx response object; so CustomStreamWrapper.aclose() silently did nothing and the upstream connection stayed open. Verified with a vLLM-style mock: with hosted_vllm/ the backend streamed all 100 chunks to completion after the client disconnected, while openai/ aborted at chunk 6 BaseModelResponseIterator now carries an optional http_response and an aclose() that closes it; make_async_call_stream_helper attaches the response after building the iterator. With this, hosted_vllm aborts the backend within ~1.6s of the client dropping, and completed streams are unaffected --------- Co-authored-by: kursad <[email protected]> * feat(anthropic): surface compaction usage iterations data (#27065) * feat(anthropic): surface compaction usage iterations data * style: apply black formatting to fix lint checks * fix(usage): correct calculate usage with cached tokens when use ChatCompletionUsageBlock (#30422) * fix(usage): correct calculate usage with cached tokens when use ChatCompletionUsageBlock * fix(usage): optimize test imports * feat: add fastCRW search provider (#30434) * feat(provider): add LibertAI as a JSON-configured OpenAI-compatible provider (#30203) * feat(provider): add LibertAI as a JSON-configured OpenAI-compatible provider * libertai: update served endpoints backup + add mode/matrix tests Addresses review feedback: - Add libertai to litellm/provider_endpoints_support_backup.json, the file actually served by GET /public/supported_endpoints (the root provider_endpoints_support.json already had it). - Add tests asserting bge-m3 normalizes to mode='embedding' and that the served matrix lists libertai. embeddings stays false: the JSON-configured provider path only wires chat routing (OpenAILike embedding handler is reached only for literal openai_like/llamafile/lm_studio), matching the llamagate precedent; bge-m3 remains in the cost map for metadata. --------- Co-authored-by: Moshe Malawach <[email protected]> * feat(provider): add ModelScope as an OpenAI-compatible provider (#28460) * add ModelScope API support * add modelscope api support * update modelscope model list * add image-genetation support * update test and multimodal * fix: address PR review feedback for modelscope provider * update README * fix(customer_endpoints): restrict /customer/daily/activity to admin-only (#28849) * fix(customer_endpoints): restrict /customer/daily/activity to admin-only * fix(customer_endpoints): check role before prisma_client guard * fix(custom_guardrail): key disable_global_guardrails takes precedence over team guardrail list (#28563) * fix(fallbacks): preserve fallback model in SDK fallback responses (#28260) * fix(fallbacks): preserve fallback model in response when using SDK-level fallbacks * fix(fallbacks): gate x-litellm-* passthrough to trusted callers only The previous patch unconditionally let `x-litellm-*` keys bypass the `llm_provider-` prefix in `process_response_headers`. That function is also called on raw upstream-provider response headers (e.g. from `llm_http_handler.py`), so a malicious provider could return `x-litellm-attempted-fallbacks` and spoof a LiteLLM-internal marker, bypassing the proxy model-override guard. Add a `preserve_litellm_internal_headers` flag (default False). Only `response_metadata.py`, which re-processes the already-built `_hidden_params["additional_headers"]` dict (LiteLLM-owned), passes True. Raw provider header callsites keep the default False, so upstream `x-litellm-*` still gets the `llm_provider-` prefix. Adds a regression test for the spoofing case and renames the existing preserve test to make the trusted-path semantics explicit. * fix(fallbacks): ignore preserve_litellm_internal_headers for raw httpx.Headers inputs * style(core_helpers): apply black formatting * fix(lint): remove banned typing.List/Dict/Any imports and suppress PLR0913 on interface overrides Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(lint): apply black formatting to modelscope chat transformation Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(lint): replace noqa with proper fixes — use **kwargs and Awaitable instead of Any/List Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(lint): remove unused AllMessageValues import Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * revert: restore base_model_iterator.py to original PR state Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(lint): restore full method signatures for MyPy compatibility; bump PLR0913 budget for new provider files Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(lint): use @override to suppress PLR0913 on inherited signatures instead of bumping budget The overrides keep their full base-class signatures for MyPy compatibility, but those signatures carry more than five parameters, which tripped PLR0913 on each subclass redeclaration. Since the arity is dictated by the base class and cannot be reduced, decorate the overrides with typing_extensions.override; ruff treats that as the intended signal that the parameter count is not under the author's control and skips PLR0913. This restores the PLR0913 baseline to 1813. * fix(lint): add @override to modelscope image generation overrides Apply the same typing_extensions.override treatment to the image generation config so its inherited-signature overrides do not count against PLR0913. --------- Co-authored-by: Joel Tony <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: hcl <[email protected]> Co-authored-by: ztko <[email protected]> Co-authored-by: Nahrin <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: Humphrey <[email protected]> Co-authored-by: kursadlacin <[email protected]> Co-authored-by: kursad <[email protected]> Co-authored-by: Dushyant Acharya <[email protected]> Co-authored-by: Yuriy <[email protected]> Co-authored-by: Recep S <[email protected]> Co-authored-by: Moshe Malawach <[email protected]> Co-authored-by: Moshe Malawach <[email protected]> Co-authored-by: Rongkun Yan <[email protected]> Co-authored-by: Varshith <[email protected]> Co-authored-by: Mateo Wang <[email protected]> * ci(lint): add blanket-noqa, dataclass-default, and unused-noqa Ruff rules (#30516) * ci(lint): enforce blanket-noqa, dataclass-default, and unused-noqa rules Enable PGH004 (blanket-noqa), RUF008 (mutable-dataclass-default), RUF009 (function-call-in-dataclass-default-argument), and RUF100 (unused-noqa) in ruff.toml, and clean up every resulting violation. RUF008/RUF009 were already clean. PGH004/RUF100 surfaced ~335 stale or blanket noqas: blanket `# noqa` are now scoped to the rule they actually suppress (mostly T201), dead directives are removed, and inapplicable codes are trimmed (e.g. F401 dropped from `import *`). lint.external lists rules enforced outside this config (the strict-rule gate via ruff-strict.toml and upstream litellm's own ruff config) so RUF100 keeps the noqa directives that protect them instead of stripping coverage this config can't see. * ci(lint): trim RUF100 external list to load-bearing codes only Drop the 9 precautionary strict-gate codes (ANN001/002/003/401, B006, PLR0913, PLW0603, RUF012, TID251) that have zero `# noqa` references in the gated source. Keep only the 11 codes with live suppressions so RUF100 doesn't flag them as unused. Future strict-gate suppressions can re-add codes here (or fix the underlying issue) as needed. * ci: ratchet lint and type-check gates (ruff preview, ANN, mypy, basedpyright) (#30379) * ci: enable ruff preview rules under the budgeted strict gate Turn on ruff preview in the strict-budget lane (ruff-strict.toml) only, leaving the clean gate (ruff.toml) untouched so make lint-ruff stays at zero. Enumerate the 118 firing codes explicitly with explicit-preview-rules so the gate is deterministic and stable across ruff upgrades rather than depending on preview auto-selecting the broad catalog. Grandfather the existing 58438 violations into ruff-strict-budget.json as per-rule baselines with headroom, so only net-new violations fail CI. The existing ten rules keep their hand-tuned slack; the new rules get slack 10 when the baseline is 50 or more and 3 otherwise. * ci: add ANN return-type rules to the budgeted strict gate Add ANN201/202/204/205/206 (missing return annotations) to the strict lane and grandfather the existing counts into ruff-strict-budget.json so the codebase ratchets toward explicit return types without breaking CI. * ci: add mypy (disallow_untyped_defs) and basedpyright strict gates with baselines Add two type-check gates, each grandfathering the current tree so only net-new violations fail CI, matching the ruff strict-budget ratchet. mypy gains disallow_untyped_defs in litellm/mypy.ini (the config the CI invocation actually reads; the root [tool.mypy] is not picked up from the litellm/ working dir). The 4885 existing missing-annotation errors are captured in litellm/.mypy-baseline.txt and the run is piped through mypy-baseline filter so new untyped defs are rejected. basedpyright runs in strict mode over litellm/, with enableTypeIgnoreComments disabled so it only honors '# pyright: ignore' and never polices mypy's '# type: ignore'. The existing strict diagnostics are grandfathered into .basedpyright/baseline.json. Both tools are pinned in the dev group and uv.lock; the lint workflow and Makefile run them filtered through their baselines, with lint-mypy-baseline-update and lint-basedpyright-baseline-update to ratchet. * ci: raise lint job timeout to 15m for the basedpyright strict pass * ci: pin pythonVersion 3.12 and regenerate baselines against merged base Merge litellm_internal_staging so the baselines cover code the CI merge includes (e.g. the cisco_ai_defense guardrail), which otherwise tripped the mypy gate with 3 ungrandfathered no-untyped-def errors. Pin pythonVersion 3.12 in pyrightconfig so basedpyright's strict analysis is reproducible across interpreter versions (CI runs 3.12). * ci: regenerate basedpyright baseline against the frozen lint env The previous baseline was generated with optional provider deps (azure, google, anthropic, mcp, numpydoc, google-genai) installed locally, so CI's dev-only env surfaced ~3500 reportUnknown*/reportMissingTypeStubs errors not in the baseline. Regenerate after uv sync --frozen so the baseline reflects the same dependency set the lint job sees. * ci: regenerate basedpyright baseline on python 3.12 frozen env The prior baseline still carried proxy-dev packages (e.g. prisma) that the lint job's dev-only, python 3.12 env lacks, leaving 2 unresolved-import errors ungrandfathered. Regenerate in a python 3.12 venv synced to the frozen lock with default groups only, so the baseline matches exactly what CI sees. * ci: replace type-check baselines with per-file count budgets The mypy and basedpyright baselines were position-sensitive (and the basedpyright one was a 27MB file), so ordinary line shifts churned them. Replace both with a per-file count gate: scripts/type_check_gate.py reduces each tool's output to errors-per-file and checks it against a committed {file: max} budget, ignoring line and column numbers. A file fails only when it gains more errors than its ceiling; debt can't be shuffled between files because each file has its own cap and new files default to zero. Budgets (mypy-file-budget.json 48K, basedpyright-file-budget.json 96K) are generated in the python 3.12 frozen lint env so they match CI. Drops the mypy-baseline dependency; basedpyright runs without its native baseline. ratchet via make lint-mypy-budget-update / lint-basedpyright-budget-update. * ci: add a small per-file slack to the type-check gate Allow each file to drift PER_FILE_SLACK (5) errors past its recorded count before failing, so a basedpyright inference ripple in an unrelated file doesn't break the build over a couple of errors. Budgets still record exact counts; the tolerance is applied at check time. * ci: move type-check slack into the budget json and trim lint timeout Make slack declarative: the budget is now {"slack": N, "files": {path: count}} so the tolerance is tuned in JSON without editing the script, mirroring how ruff-strict-budget.json carries its slack. --update preserves the existing slack. Also drop the lint job timeout from 15m to 10m; the mypy and basedpyright passes add ~2m, leaving the job around 4-5m, so 10m is a comfortable margin. * ci: collapse fully-adopted ruff categories and drop inert preview flag ANN (all nine non-removed rules) and BLE (its only rule) were spelled out code-by-code; replace each with its category selector, which is exactly equivalent in 0.15.3 (the removed ANN101/ANN102 are skipped by a category selector and error when named explicitly). explicit-preview-rules was inert: every selected rule is stable and nothing is selected by category, so the flag had nothing to gate. Verified the strict-rule counts are identical before and after (62379 each, zero per-rule drift), so no budget change. * ci: drop redundant pyright dev dependency Nothing invokes bare pyright in the Makefile, the linting workflow, or scripts; the basedpyright gate added on this branch is the only type checker that runs. based…

…rriAI#30542) * fix(guardrails): stop re-initializing DB guardrails on every poll InMemoryGuardrailHandler._has_guardrail_params_changed compared the in-memory LitellmParams against the raw dict loaded from the DB. The in-memory side carries every field default and coerces enums via model_dump(), while the DB side only holds the keys originally stored, so the two shapes never compared equal and the guardrail was rebuilt on every poll cycle. Each rebuild created a fresh instance, but delete_in_memory_guardrail only removed the old callback from litellm.callbacks. Request handling promotes guardrail callbacks into the success/failure/async lists, so the previous instance stayed referenced there and instances accumulated. Normalize both sides through LitellmParams(...).model_dump() before diffing, and purge the callback from every callback list on delete. * refactor(guardrails): narrow params-normalization fallback to ValidationError The comparison normalizer caught a bare Exception and silently fell back to the raw dict, which hid the cause and quietly degraded the affected guardrail back to re-initializing on every poll. Catch only the ValidationError that LitellmParams construction can raise, log a warning so the offending row is diagnosable, and let any other error surface instead of being swallowed. * refactor(callbacks): add remove_callback_from_all_lists helper to manager Move the knowledge of which callback lists a callback can be promoted into out of the guardrail registry and into LoggingCallbackManager, where the rest of the callback-list bookkeeping already lives. delete_in_memory_guardrail now delegates to the new helper instead of iterating the lists itself.

* feat(ui): gate "Default Credentials" hint on /ui/login behind env flag (#30234) Adds LITELLM_HIDE_DEFAULT_CREDENTIALS_HINT (and an equivalent general_settings.hide_default_credentials_hint) that suppresses the "By default, Username is admin and Password is your set LiteLLM Proxy MASTER_KEY" info card rendered on /ui/login and /fallback/login. Motivation: in production deployments operators set UI_USERNAME / UI_PASSWORD (or SSO), and the hardcoded hint becomes factually incorrect and is flagged by security scanners (Tenable WAS plugin 114625) as information disclosure. There is currently no way to suppress it without forking the dashboard. Behaviour: - Default is unchanged (hint shown), so existing deployments are unaffected. - New field hide_default_credentials_hint on the well-known UI config endpoint, populated from the env var or general_settings. - LoginPage.tsx conditionally renders the Alert based on the flag. Refs: BerriAI/litellm#30232 * fix(router): clean pattern_router state on upsert/delete (#29601) * fix(router): clean pattern_router state on upsert/delete PatternMatchRouter.add_pattern was append-only, and neither Router.upsert_deployment nor Router.delete_deployment removed the existing entry. Rotated-out api_keys stayed in the routing rotation for wildcard deployments (model_name with `*`) until proxy restart, silently defeating key rotation as an admin operation. The same leak applied to provider_default_deployment_ids and per-team pattern routers, and the patterns list grew unboundedly on every edit * test(router): direct unit tests for _remove_deployment_from_wildcard_state router_code_coverage.py greps test files for AST Call nodes and flagged the helper as untested because the existing coverage only exercised it transitively through upsert/delete. Adds two direct tests that pin the helper's contract (cleans across global pattern router, per-team routers with empty-router pop, and provider_default_deployment_ids; noop on falsy model_id) * fix(router): address Greptile review on pattern_router cleanup Widen PatternMatchRouter.remove_deployment annotation to Optional[str]; the implementation already handles None via the falsy guard and the unit test exercises it directly. Move _remove_deployment_from_wildcard_state up one level in upsert_deployment so it runs whenever the prior deployment is on the router, not only when the model_id is present in the fast-mapping index. The scenario is currently unreachable (get_deployment shares the same index), but the cleanup is idempotent so this is defensive against any future divergence between those code paths. * fix(router): widen _remove_deployment_from_wildcard_state to Optional[str] Moving the call out of the inner `deployment_id in deployment_fast_mapping` block in the previous commit lost mypy's narrowing of `deployment_id` from Optional[str] to str, tripping the lint CI. The helper already handles None via its falsy guard, so widening the annotation matches the actual contract. * fix(router): make delete_deployment wildcard cleanup symmetric with upsert After the previous commit moved _remove_deployment_from_wildcard_state out of the inner index-map guard in upsert_deployment, delete_deployment was still calling it only inside `if deployment_idx is not None`. Greptile flagged the asymmetry: under a desynced index_map, delete would silently leave the stale wildcard credential in pattern_router. Moves the cleanup call to the top of the try block, mirroring the upsert path. Cleanup is idempotent so the change is a no-op on the happy path. Adds a regression test that simulates the desync by removing the entry from model_id_to_deployment_index_map and asserts delete still clears pattern_router. * fix(pricing): add 1h cache-write cost for Anthropic Sonnet 4.5/4.6 (#30474) The native anthropic claude-sonnet-4-5/4-6 price-map entries were missing cache_creation_input_token_cost_above_1hr (and the >200K long-context sub-tier for 4.5), so 1-hour-TTL cache writes were costed at the 5-minute rate. Adds 6e-06 regular (and 1.2e-05 long-context) = 2x base input, matching the vertex_ai/azure_ai/bedrock siblings and the older claude-sonnet-4-20250514 entry. Adds a regression test. * fix(proxy): cancel upstream gemini request and release httpx connection on client disconnect (#30075) * fix(proxy): cancel upstream gemini request and release httpx connection on client disconnect - add _check_request_disconnection to common_request_processing; wrap llm_call as asyncio.Task so it can be cancelled; catch CancelledError and raise HTTPException(499) when client disconnects before LLM responds (non-streaming path) - pass raw httpx.Response into ModelResponseIterator in make_call/make_sync_call so the iterator holds a reference to the underlying connection - implement ModelResponseIterator.aclose() and .close(): close the line iterator then explicitly call response.aclose()/response.close() to release the httpx connection when the client drops mid-stream; errors are debug-logged, not raised - add tests for _check_request_disconnection (cancels task, graceful on exception, does not cancel when client stays connected) and base_process_llm_request 499 behavior; add TestModelResponseIteratorCleanup verifying aclose/close propagation through CustomStreamWrapper * fix(proxy): record 499 on streaming disconnect and cancel orphaned gather tasks Wire streaming generator cleanup to log client_disconnected with error_code 499 in spend logs, cancel pending during_call_hook tasks when the LLM call is cancelled on disconnect, and align the 600s poll limit comment with proxy_server. * fix: extract client disconnect logging helper to satisfy PLR0915 * fix: resolve mypy and code-quality CI failures for client disconnect logging Cast client disconnect error_information for mypy, only await pending gather tasks to avoid masking LLM errors, and add tests for the new logging helper and gather cleanup. * fix(proxy): harden gather cleanup so finally cannot mask LLM errors * fix(proxy): shield streaming disconnect logging and strip spoofable metadata Move streaming disconnect recording into a shielded cancel scope, add gather cleanup regression coverage for guardrail-converted cancels, and strip client_disconnected/error_information from user metadata at the proxy boundary. * fix(proxy): only map CancelledError to 499 for client disconnect Track when the disconnect poller cancels the LLM task and re-raise other CancelledError paths so graceful shutdown is not reported as HTTP 499. * fix(proxy): remove dead _check_request_disconnection helper Non-streaming client disconnect is handled by staging's cancel_on_disconnect path via _await_llm_call_cancelling_on_disconnect. Drop the unused is_disconnected poller and its unit tests; rename the remaining integration tests to TestDisconnectGatherCleanup. * feat(mistral): add mistral-medium-3-5 to model_prices_and_context_wind.. (#29303) * feat(mistral): add mistral-medium-3-5 to model_prices_and_context_window.json Mistral's docs page lists mistral-medium-3-5 as a new model offering. Pricing/specs sourced from Mistral's published model metadata: - input: $1.50 / 1M tokens - output: $7.50 / 1M tokens - context: 262,144 tokens - capabilities: vision, function calling, structured outputs, assistant prefill Adds entry: `mistral/mistral-medium-3-5`, mirroring the pattern used for the rest of the Mistral family. test(mistral): add model_info test for mistral-medium-3-5 + sync backup cost map - Mirror mistral/mistral-medium-3-5 entries into litellm/model_prices_and_context_window_backup.json so the bundled model cost map matches the canonical model_prices_and_context_window.json. - Add tests/test_litellm/test_mistral_medium_3_5_model_metadata.py covering pricing tiers, capability flags, context window, provider routing, and parity between the main and backup cost maps. - Point 'source' at the live Mistral models documentation page. * fix(ui): three small UI fixes — Gemini api_base + credential form reset + Mode badge (#30419) * fix(ui): three small UI fixes — Gemini api_base field + credential form reset + Mode badge Three independent fixes; bundled because they all touch the credential-form / logging-callbacks area. 1. expose api_base field on Google AI Studio credential form The runtime gemini provider supports custom api_base via `vertex_llm_base._check_custom_proxy`; the UI just needs to expose the field. Adds api_base to the Google_AI_Studio credential form ordered before api_key (matching OpenAI/Anthropic conventions). Default value matches the canonical Google AI Studio endpoint that LiteLLM's gemini provider talks to when api_base is unset, so leaving the default in the form behaves identically to leaving it blank. 2. reset credential form state when switching providers Switching the Provider select in AddCredentialModal / EditCredentialModal left the previous provider's field values populated. The form then submitted a mixed payload (e.g. Azure deployment fields under an OpenAI credential), producing confusing failures. Extract `getProviderFieldDefaults` helper and reset the form to it on provider change. Unit-tested via the extracted helper because Antd Select's portal/dropdown behaviour is unreliable in jsdom. 3. logging callbacks table reads backend `type` for Mode badge (#35) The `/get_callbacks` proxy endpoint returns each callback as `{name, type, variables}` where `type` is `"success"` or `"failure"`. The same callback name can appear twice (one per event class) and the two entries fire on disjoint events. `LoggingCallbacksTable` ignored `type` and read `record.mode` (always undefined), so every row fell back to the "Success" badge. A `generic_api` callback registered for both classes showed up as two identical "Success" rows + React duplicate-key warning. Read `record.type` first (fall back to `record.mode` for newly- added not-yet-server-acknowledged rows). Composite rowKey `${name}-${type ?? mode ?? 'success'}`. Removed leftover debug `console.log`. * fix(ui): drop api_base default_value to preserve Gemini v1alpha auto-routing Greptile P2 (PR #30419, threads on lines 1255-1256 of provider_create_fields.json): the api_base field's `default_value` was hard-coded to "https://generativelanguage.googleapis.com/v1beta". This: 1. Bakes v1beta into every credential record saved through the form, even when the user never touched the field. If LiteLLM's internal gemini default URL ever changes, those persisted credentials keep hitting the stale path. 2. Bypasses `_get_gemini_url`'s automatic version routing for Gemini 3+ models. That helper picks v1alpha for Gemini 3+ and v1beta for older models when api_base is unset. With the default pre-filled (and `_check_custom_proxy` then taking over because api_base is non-empty), Gemini 3+ requests get pinned to v1beta and may fail or behave unexpectedly — purely because the user accepted the visible default. Fix: set `default_value` to `null` and move the canonical URL guidance into the `placeholder` (visible to the user, never persisted) and an expanded tooltip. UX is unchanged — the URL is still shown in the greyed-out input — but the auto-version-routing path stays default. Updated test_google_ai_studio_provider_fields_expose_api_base to assert the new contract (`default_value is None`, `placeholder` carries the canonical URL), with a comment pointing at the Greptile threads as the rationale so future contributors don't accidentally re-introduce the default. 26/26 tests in the file pass. JSON validates (`json.load` clean). * feat(azure_ai): add gpt-5.5 to model cost map (#30428) * feat(azure_ai): add gpt-5.5 to model cost map Adds azure_ai/gpt-5.5 and its dated snapshot azure_ai/gpt-5.5-2026-04-23 to both the canonical and bundled cost maps. gpt-5.5 is generally available on Azure AI Foundry; pricing mirrors the openai gpt-5.5 entry, matching the established azure_ai convention (verified identical for gpt-5.4), in the azure tier structure (base / above-272k / priority). supports_minimal_ reasoning_effort is false, the capability that changed from gpt-5.4. Fixes #30306 * Update tests/test_litellm/test_gpt_5_5_model_metadata.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: guard check_and_fix_namespace against None key (#30435) * fix: guard check_and_fix_namespace against None key When user_id is None, the cache key can be None, causing AttributeError: 'NoneType' object has no attribute 'startswith' in check_and_fix_namespace. Add an early return for None key to prevent the error and the ERROR-level log noise it produces on every unauthenticated request. Fixes #30424 * fix: update type annotations for check_and_fix_namespace - key: str -> Optional[str] (now handles None input) - return: str -> Optional[str] (returns None when input is None) Addresses Greptile review concern about type signature mismatch. * fix: revert check_and_fix_namespace type signature to str to fix MyPy downstream errors * fix: update type annotations for check_and_fix_namespace - Change signature from str -> str to Optional[str] -> Optional[str] - Remove type: ignore comment on None return - Add None guard in async_set_cache_sadd before passing to helper Addresses review feedback from Sameerlite on type mismatch. * Revert "fix: update type annotations for check_and_fix_namespace" This reverts commit 5272920fa0daab676f5ad46dcadd8cd537cfc96f. --------- Co-authored-by: michaelxer <[email protected]> * fix(cost): apply service_tier suffix to above-threshold cache rates and expose priority+threshold keys in ModelInfo (#30450) * fix(cost): apply service_tier suffix to above-threshold cache rates and expose priority+threshold keys in ModelInfo Models that publish both a service_tier (e.g. priority) rate and an above-threshold tier (e.g. _above_200k_tokens) currently bill cached tokens at the standard above-threshold rate rather than the priority above-threshold rate. Affected entries in the live pricing JSON include gemini-3-pro-preview, gemini-3.1-pro-preview and their vertex_ai/ and gemini/ variants, plus azure/gpt-5.4 and azure_ai/gpt-5.4. For a 250K-token priority request with 200K cached tokens against gemini-3-pro-preview, the leak is about 44 percent of the prompt cost. Two stacked defects caused this. First, ModelInfoBase (and the ModelInfo pydantic class) and the get_model_info construction in litellm/utils.py omit the priority+above-threshold cost keys, so even if the calculator asked for them they would never reach it. Second, in _get_token_base_cost the cache_creation/cache_read tiered keys never get wrapped with _get_service_tier_cost_key, while the input/output tiered keys above and below do. The change here surfaces six new keys (input, output and cache_read at both 200k and 272k priority variants) and wraps the three cache tiered keys in _get_token_base_cost the same way input/output already are. _get_cost_per_unit's existing service_tier-to-base fallback covers models that ship the standard above-threshold rate without a priority variant. Adds one regression test in tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py that drives the actual generic_cost_per_token path for gemini-3-pro-preview at 200K cached + 50K text under priority and asserts the priority above_200k rates are picked. Verified the test fails on litellm_internal_staging without these changes and passes with them. * fix(cost): drop guard on cache tiered keys so service_tier fallback can reach standard above-threshold rate Addresses Greptile P1 on PR 30450. The previous commit wrapped cache_creation_tiered_key, cache_creation_1hr_tiered_key, and cache_read_tiered_key with _get_service_tier_cost_key (matching how the sibling input and output tiered keys are wrapped) but kept the surrounding 'if key in model_info' guards. For models that publish a standard above-threshold cache rate but no priority variant (gpt-5.4-pro, gpt-5.5-pro and their dated siblings, plus vertex_ai/claude-sonnet-4-5 for cache_creation), the guard short-circuits before _get_cost_per_unit's existing service_tier-to-base fallback can strip _priority and find the standard above-threshold key. The result on priority requests over the threshold was that those models silently dropped from the above-threshold rate back to the priority-base rate. Dropping the guard and calling _get_cost_per_unit unconditionally (mirroring how tiered_input_key and tiered_output_key are already handled) restores correct billing for that class of models while keeping the new priority+above-threshold behaviour for gemini-3-pro-preview and friends. Adds a second regression test that pins generic_cost_per_token for vertex_ai/claude-sonnet-4-5 priority + above_200k with cached and cache_creation tokens to the expected standard above-threshold rates, so the guard cannot be silently reintroduced for either the cache_read or cache_creation path. * fix(presidio): skip pre-call masking when guardrail is logging_only (#30461) The Presidio pre-call hook masked the live request unconditionally, ignoring the configured event hook. With mode: logging_only the masked request reached the model, so its response echoed anonymization tokens (e.g. <PERSON>) instead of the real output. Gate async_pre_call_hook on should_run_guardrail, matching every other guardrail; logging_only masking still happens via async_logging_hook. * fix(router): resolve list unhashable crash on model alias (#30464) * fix(router): resolve list unhashable crash on model alias Fixes the fallback parsing logic that mistakenly categorized standard array fallback definitions as override dictionaries when a deployment alias matches the literal string 'model'. Closes https://github.com/BerriAI/litellm/issues/30459 * fix(router): address greptile review for fallback parsing edge cases - Resolves ambiguity in standard vs override fallback dictionaries by iterating over all items and validating that no mapped litellm param resolves to a non-list type. - Adds regression tests in test_router_order_fallback.py to prevent unhashable type crash from silently re-entering the codebase. * chore(router): format code with black to pass CI * fix(hosted_vllm): remove thinking_blocks and convert list content to strings (#30475) * fix: hosted_vllm remove thinking_blocks and convert list content to strings vLLM endpoints reject assistant messages with thinking_blocks converted to content list blocks. This change removes thinking_blocks entirely and converts any list content back to strings. This fixes BadRequestError when using Claude Code with hosted_vllm models that pass thinking_blocks in messages. * fix(hosted_vllm): address Greptile review feedback - Join multiple text blocks with newline instead of empty string - Always set content to string (never None) to avoid vLLM validation errors * fix(hosted_vllm): update chat transformation to clean assistant messages * fix: re-raise exception instead of silently dropping MCP team permissions (#30477) * fix: re-raise exception instead of silently dropping MCP team permissions When MCPRequestHandler.get_allowed_mcp_servers raises, the broad except was swallowing the error and returning only allow_all_server_ids, silently discarding all team-level object_permission grants. Fixes #30476 * fix: log full traceback when MCP permission lookup fails Uses verbose_logger.exception() instead of warning() so operators can see the full traceback when team-level object_permission grants are dropped due to an internal error in get_allowed_mcp_servers. Fixes #30476 * fix: remove timezone date expansion in daily-activity aggregation (#29569) * fix: remove timezone date expansion in daily-activity aggregation Single-day spend queries from non-UTC timezones over-counted by ~2x because the previous implementation widened the SQL date range by a full UTC day on whichever side the offset pointed. Spend is bucketed in whole-UTC-day rows in LiteLLM_DailyUserSpend, so the expansion pulled an extra 24h of unrelated bucket data per boundary. Concretely on IST (UTC+5:30, offset -330): a single-day query for 2026-05-29 was rewritten to date >= 2026-05-28 AND date <= 2026-05-29 and returned spend across both UTC days. Sums of single-day queries across a 5-day window then exceeded the equivalent multi-day aggregate by ~50%, which is mathematically impossible. Treat the local date range as the UTC date range. The aggregation table has no hour-level granularity, so any conversion using only date arithmetic must round to whole UTC days; the previous fix turned that boundary slop into systematic over-counting. Pass-through trades a small one-time slop at each end of the range for correct, monotonic, additive results across single-day and multi-day queries. Repro from production: bedrock/global.anthropic.claude-opus-4-8 over 2026-05-29 to 2026-06-02, IST timezone: - 5-day aggregate: $701.39 / 1,831 reqs - Sum of 5 single-day queries: $1,070.94 / 2,755 reqs - Excess (was 1.527x): now matches within boundary slop Adds regression tests in TestAdjustDatesForTimezone and TestBuildAggregatedSqlQuery that pin the pass-through behavior and the additivity invariant for any future implementation. * ci: rerun checks on litellm_oss_branch base --------- Co-authored-by: Sameer Kankute <[email protected]> * fix: buffer native gemini sse frames (#30225) * fix: buffer native gemini sse frames * fix: scope native gemini sse buffering * fix: check raw sse residual buffer size * feat: updated openrouter provider to map max level to xhigh (#28881) * feat(proxy): allow use_redis_transaction_buffer without redis cache (#28764) * feat(proxy): allow use_redis_transaction_buffer without redis cache * fix(proxy): require host or url for standalone buffer redis * fix(mcp): fail closed when scope filter resolves to no servers (#30353) `_get_allowed_mcp_servers_from_mcp_server_names` returned the caller's full allowed-server set when the requested `mcp_servers` list (path- or header-derived) resolved to nothing. URL/header namespacing therefore appeared to work even when the requested name was unknown or the caller had no grant — `/mcp/<typo>/` silently exposed every server the key could reach. Fail closed instead: when `mcp_servers` is explicitly provided but nothing resolves, return an empty list. The `mcp_servers=None` path (no scope requested) keeps its existing behavior. Co-authored-by: Claude Opus 4.7 <[email protected]> * fix(token-counter): handle Anthropic tool_reference blocks to stop dropped spend logs (#30302) * fix(token-counter): handle Anthropic tool_reference blocks to stop dropped spend logs `token_counter` did not know about Anthropic tool-search `tool_reference` content blocks, a lightweight pointer to a deferred tool that shows up as `{"type": "tool_reference", "tool_name": ...}`. When such a block appeared in message content, `_count_content_list` fell through to its catch-all branch and raised `Invalid content item type: tool_reference`. On the streaming `anthropic_messages` proxy path that exception nulls `response_cost`, which makes the proxy drop the entire SpendLogs row. The result is a silent cost undercount on any tool-search traffic; the request succeeds for the caller but the spend is never recorded. This adds a `tool_reference` branch that counts the referenced `tool_name` (the full tool definition is already counted via the `tools` param, so only the name is added here) and handles an empty/missing name gracefully. The catch-all error message is updated to list `tool_reference` among the expected types. A regression test asserts that a message containing a `tool_reference` block no longer raises and returns a positive token count, and that an empty `tool_name` is handled without error. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * fix(token-counter): collapse explicit None tool_name to empty string In _count_content_list, c.get("tool_name", "") returns None when the key is present with an explicit None value, and str(None) == "None" which is truthy, causing a spurious token to be counted. Use c.get("tool_name") or "" so both a missing key and an explicit None collapse to an empty string and are skipped. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * test(token-counter): cover catch-all for unknown content block type Adds a regression test that calls `_count_content_list` with an unrecognized content block type and asserts it raises `ValueError` whose message names the offending type and lists `tool_reference` among the supported types. This exercises the previously uncovered catch-all branch (codecov patch gap) and pins the error contract. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> * test(token-counter): cover tool_reference on the spend/cost and streaming paths Adds end-to-end regression tests that exercise the real public entry points (`completion_cost` and `stream_chunk_builder`), not just the private `_count_content_list` helper, for Anthropic tool-search `tool_reference` content blocks. These pin the actual bug the fix addresses: before the fix the `tool_reference` block raised out of `completion_cost` -> the proxy logging layer nulled `response_cost` and the spend callback dropped the SpendLogs row (silent cost undercount on all tool-search traffic); and `stream_chunk_builder` swallowed the same raise and collapsed prompt_tokens to 0. With the fix, cost is positive and prompt_tokens are counted. Verified: 3 fail without the fix, 3 pass with it. Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]> --------- Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]> * feat(cost): add cost mapping for deepseek-v4-flash and deepseek-v4-pro (#27056) * feat(cost): add cost mapping for deepseek-v4-flash and deepseek-v4-pro Adds pricing entries for the two new DeepSeek V4 models released on 2026-04-24, for both bare model names and the deepseek/ provider prefix. Prices sourced from https://api-docs.deepseek.com/quick_start/pricing: - deepseek-v4-flash: $0.14/M input, $0.28/M output - deepseek-v4-pro: $1.74/M input, $3.48/M output Cache hit price set to 1/10 of input (per DeepSeek docs). Context window: 1M tokens for both models. Closes #26709 * fix(cost): update backup registry for deepseek-v4 * style: remove print statement from deepseek-v4 test * feat(cost): add cost mapping for deepseek-v4-flash and deepseek-v4-pro Adds pricing entries for the two new DeepSeek V4 models released on 2026-04-24, for both bare model names and the deepseek/ provider prefix. Prices sourced from https://api-docs.deepseek.com/quick_start/pricing: - deepseek-v4-flash: $0.14/M input, $0.28/M output - deepseek-v4-pro: $1.74/M input, $3.48/M output Cache hit price set to 1/10 of input (per DeepSeek docs). Context window: 1M tokens for both models. Closes #26709 * fix: update deepseek-v4 prices to active discounted rates * test: update deepseek-v4 prices in tests to match active discounted rates * fix(deepseek): remove duplicate entries and update backup registry to active discounted rates * fix: update max_output_tokens to 384K for deepseek-v4 * fix: correctly restore upstream models accidentally dropped during merge * fix(tests): resolve failing claude-fable-5 and reasoning tests by safely updating cost map - Pulled the latest cost map from upstream staging - Safely appended deepseek-v4 mapping without deleting duplicate keys or formatting via json.dump * fix(tests): correct deepseek model cache prices and update JSON schema - Appended both prefixed and bare deepseek-v4 models to satisfy test assertions - Corrected deepseek-v4-pro expected cache hit and token prices based on latest review updates - Added missing realtime endpoint to test_utils.py INTENDED_SCHEMA * fix: remove accidental azure/gpt-realtime-whisper addition --------- Co-authored-by: Dushyant Acharya <[email protected]> * feat(key/info): expose per-model budget usage in /key/info response (#30394) * feat(key/info): expose per-model budget usage in /key/info response Add model_max_budget_usage to /key/info and /v2/key/info responses. For each model in model_max_budget, reads current-period spend from the same DualCache used by the budget enforcer and returns it alongside the limit and time period so callers can see how much of each model budget has been consumed in the active window. * test(key/info): add coverage for model_max_budget_usage in v1 and v2 endpoints Add tests for the model_max_budget_usage enrichment in both info_key_fn and info_key_fn_v2, covering the budget-present path, the empty-budget path, and the v2 batch endpoint. * fix(key/info): source model_max_budget current_spend from SpendLogs instead of DualCache The DualCache used for enforcement is ephemeral and only populated when budget metadata is present at request time. Fall back to a direct LiteLLM_SpendLogs DB aggregation using the budget period window (budget_reset_at - budget_duration) for accurate reporting. Also fall back to litellm_budget_table.model_max_budget when the key's top-level field is empty, and round current_spend to 4 decimal places. * test(key/info): cover remaining branches in model_max_budget_usage helpers Add unit tests for: prisma_client=None early return, DB query exception swallowing, invalid budget_duration handled by _compute_budget_period_start, budget_reset_at received as a datetime object (Prisma native type), max_seconds=0 early return, and skipping models that lack a budget_duration. Also remove an unreachable except branch where fromisoformat would fail after _compute_budget_period_start already validated the same value. * test(key/info): cover except path for unparseable per-model budget_duration * fix(key/info): compute per-model rolling windows in model_max_budget_usage Each model in model_max_budget now gets its own time window derived from its own budget_duration, rather than sharing a single window computed as the max (or the budget table's reset_at). This matches what the DualCache enforcer actually tracks and prevents current_spend from being inflated for models with shorter windows. _query_model_spend_for_period is refactored to accept a model filter (handling provider-prefix variants in SQL) and return a float directly. _compute_budget_period_start and the budget_table window path are removed as they are no longer needed. * refactor(model_max_budget_limiter): remove dead get_current_period_spend method * refactor(key/info): strip synthetic formatter noise from PR diff Restore key_management_endpoints.py and test_key_management_endpoints.py to origin/litellm_internal_staging, then re-apply only the intentional additions: _query_model_spend_for_period, _build_model_max_budget_usage, the two endpoint patches (info_key_fn / info_key_fn_v2), and the new test suite. The previous commits had reformatted ~300 pre-existing lines across both files, making the functional diff unreadable. * test(key/info): cover empty-rows path in _query_model_spend_for_period * fix(model_max_budget_limiter): guard BudgetConfig construction inside try/except A malformed model entry in the DB (e.g. non-numeric max_budget from a manually edited or migrated row) caused BudgetConfig(**budget_info) to raise a Pydantic ValidationError outside any exception guard, surfacing as a 500 for the entire /key/info or /v2/key/info call. Merging both try/except blocks into one ensures bad entries are silently skipped, consistent with the existing duration_in_seconds guard. * fix: don't stack provider prefix on wildcard models with a custom prefix (#30360) * fix: don't stack provider prefix on wildcard models with a custom prefix get_known_models_from_wildcard expanded provider-prefixed model ids (e.g. "ollama/gemma3:1b" from get_provider_models) by prepending the wildcard's prefix whenever the id did not already start with it. With a custom wildcard prefix such as "ollama_server1/*" (used to distinguish multiple Ollama instances), this produced "ollama_server1/ollama/gemma3:1b", which is uncallable and breaks /v1/models. When the expanded id already carries a provider prefix, replace it with the wildcard's prefix instead of stacking both. Matching-prefix and bare-model cases are unchanged. Fixes #30358 * fix: only strip a known provider prefix when expanding custom wildcard prefixes The wildcard expansion replaced the leading slash segment of every expanded id with the wildcard prefix whenever the id did not already start with it. For ids whose first segment is an org rather than a litellm provider (for example a provider returning "meta-llama/Llama-3-8B" with no outer provider prefix), that dropped the org and produced an uncallable id Only strip the leading segment when it is a recognized provider (membership in LlmProviders); otherwise keep it and just prepend the wildcard prefix. Provider-prefixed ids like "ollama/gemma3:1b" still have their prefix replaced, so the original fix is unchanged for known providers * address greptile review feedback: log dropped non-text vLLM assistant content blocks (greploop iteration 1) * fix(ci): format credential_form_helpers test + regenerate dashboard schema.d.ts * fix(proxy): raise litellm.BadRequestError for missing model param When no model is passed, route_request now raises a litellm.BadRequestError ('Missing model parameter') instead of falling through to ProxyModelNotFoundError. This keeps the missing-param error clear and independent of router wildcard state. Unknown (non-empty) model names still raise ProxyModelNotFoundError. * Revert "fix(proxy): raise litellm.BadRequestError for missing model param" This reverts commit 9240da403c0432a80473d6c4677ddb7e2bad7420. * Revert "fix(router): clean pattern_router state on upsert/delete (#29601)" This reverts commit ad4e6e2395620ea6d2fe38089a54cde160720de2. * fix: correct streaming and key budget usage reporting * fix(hosted_vllm): type assistant tool_calls to satisfy mypy * feat: aws secret manager cross region replication (#30368) * feat(aws-secret-manager): add replica_regions cross-region replication after CreateSecret When store_virtual_keys is enabled, async_write_secret() only wrote secrets to the primary AWS region. Multi-region proxy deployments had no built-in way to synchronize virtual key secrets across regions through LiteLLM, requiring external replication mechanisms. Add replica_regions support to AWSSecretsManagerV2: - New replica_regions field in KeyManagementSettings (types/secret_managers/main.py) - New async_replicate_secret() method that calls ReplicateSecretToRegions API - async_write_secret() calls replication after successful CreateSecret - Replication failure is logged as a warning but does NOT fail key creation - load_aws_secret_manager() forwards replica_regions from key_management_settings Configuration example: key_management_settings: store_virtual_keys: true replica_regions: - us-west-2 - eu-west-1 When replica_regions is omitted or empty, behavior is unchanged. * test(aws-secret-manager): restore litellm.secret_manager_client after test to prevent state pollution * test(aws-secret-manager): add coverage for HTTP error and replication exception paths * fix: restore litellm.secret_manager_client global state in test; add replication log proof - Global state in test_load_aws_secret_manager_passes_replica_regions was already guarded with try/finally (committed in previous pass); no further change needed for Fix 1. - Fix 2: add verbose_logger.info("ReplicateSecretToRegions called …") inside async_replicate_secret so callers get an observable INFO log line whenever replication fires. - Add test_replication_fires_on_create: calls async_replicate_secret directly with caplog.at_level(INFO, logger="LiteLLM") and asserts "ReplicateSecretToRegions" appears in the captured log output, proving the code path executes. * fix: pass request to streaming generators * fix(hosted-vllm): preserve assistant structured content * fix(hosted_vllm): satisfy mypy on preserved structured content assignment * chore: resolve litellm_internal_staging merge conflicts for #30527 (#30554) * chore(codecov): add Batches, Videos, and Realtime components (#30517) * chore(codecov): add Batches, Videos, and Realtime components Define per-feature Codecov components so PR comments track coverage for batch API, video generation, and realtime streaming paths. Co-authored-by: Cursor <[email protected]> * chore(codecov): use wildcard path for Batches proxy component Align batches_endpoints glob with Videos, Realtime, and Proxy_Authentication. Co-authored-by: Cursor <[email protected]> --------- Co-authored-by: Cursor <[email protected]> * test(batches): move orphan tests into tests/test_litellm for CI coverage (#30510) Four batch-related tests lived under tests/litellm/ and were never picked up by GitHub Actions. Relocate them and fix gemini multimodal e2e to use the batchEmbedContents path expected for gemini/ provider. Co-authored-by: Cursor <[email protected]> * fix(guardrails): run pre_call hook once for model-level guardrails (#30543) * fix(guardrails): run pre_call hook once for model-level guardrails A CustomGuardrail attached to a deployment via litellm_params.guardrails gets its async_pre_call_hook invoked twice per request: once by the proxy pre-call loop and again by async_pre_call_deployment_hook after the router spreads the model-level guardrails into the top-level request kwargs. Record in request metadata that the proxy pre-call loop already ran a given guardrail, and have the deployment hook skip it when the marker is present. Direct-SDK usage never runs the proxy loop, so the deployment hook stays the sole invocation there and still fires exactly once. The marker key is stripped from untrusted caller metadata so a request body cannot suppress a model-only guardrail by pre-seeding it. * fix(guardrails): mark pre_call dedup on the post-hook request data Record the exactly-once marker after async_pre_call_hook runs, on the data object that flows downstream, rather than before it. A guardrail whose hook returns a brand-new request dict (instead of mutating or spreading the one it received) would otherwise discard the marker, letting the deployment hook re-run the guardrail a second time. * fix(guardrails): stop re-initializing DB guardrails on every poll (#30542) * fix(guardrails): stop re-initializing DB guardrails on every poll InMemoryGuardrailHandler._has_guardrail_params_changed compared the in-memory LitellmParams against the raw dict loaded from the DB. The in-memory side carries every field default and coerces enums via model_dump(), while the DB side only holds the keys originally stored, so the two shapes never compared equal and the guardrail was rebuilt on every poll cycle. Each rebuild created a fresh instance, but delete_in_memory_guardrail only removed the old callback from litellm.callbacks. Request handling promotes guardrail callbacks into the success/failure/async lists, so the previous instance stayed referenced there and instances accumulated. Normalize both sides through LitellmParams(...).model_dump() before diffing, and purge the callback from every callback list on delete. * refactor(guardrails): narrow params-normalization fallback to ValidationError The comparison normalizer caught a bare Exception and silently fell back to the raw dict, which hid the cause and quietly degraded the affected guardrail back to re-initializing on every poll. Catch only the ValidationError that LitellmParams construction can raise, log a warning so the offending row is diagnosable, and let any other error surface instead of being swallowed. * refactor(callbacks): add remove_callback_from_all_lists helper to manager Move the knowledge of which callback lists a callback can be promoted into out of the guardrail registry and into LoggingCallbackManager, where the rest of the callback-list bookkeeping already lives. delete_in_memory_guardrail now delegates to the new helper instead of iterating the lists itself. * chore(oss): litellm oss staging 150626 (#30463) * fix(pricing): add GitHub Copilot MAI Code Flash pricing (#30415) * fix(pricing): add GitHub Copilot MAI Code Flash pricing Add GitHub Copilot pricing entries for MAI-Code-1-Flash and the internal Copilot CLI model name so cost calculation can price input, cached input, and output tokens. Co-authored-by: Copilot <[email protected]> * test(pricing): cover GitHub Copilot MAI Code Flash pricing Add regression coverage for both GitHub Copilot MAI-Code-1-Flash model names, including cached input pricing, chat endpoint metadata, and cost_per_token arithmetic. Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: Copilot <[email protected]> * fix(router/proxy): propagate completed_response through FallbackResponsesStreamWrapper for streaming /v1/responses container ownership (#30210) (#30213) * fix(router/proxy): propagate completed_response through FallbackResponsesStreamWrapper for streaming /v1/responses container ownership (#30210) #28990 added ownership recording for streaming /v1/responses via _wrap_responses_stream_for_container_ownership, which reads `getattr(stream_response, 'completed_response', None)` to extract the ResponsesAPIResponse. The unit test bypassed the Router, so it never exercised the production wrapping path. Through the Router (every proxy deployment), the stream is wrapped by FallbackResponsesStreamWrapper (router.py:2527). Its __init__ set `self.completed_response = None` and __anext__ only forwarded chunks — the inner source iterator's terminal event never bubbled up to the attribute the ownership hook reads, so the hook silently recorded nothing and every follow-up /v1/containers/<id>/files call returned 403 for non-admin keys. This commit: - router.py: pre-resolves the responses-API terminal event tuple (response.completed / .incomplete / .failed) once per _aresponses_streaming_iterator call, and has the wrapper's __anext__ sniff each forwarded chunk's .type. First terminal event hit gets stored on the wrapper's completed_response. Iterator-agnostic — works for source_iterator AND any future wrapper. - common_request_processing.py: when _extract_completed_responses_response returns None we now warn instead of silently skipping. Reporter on #30210 lost a day to this exact silent skip; the warning surfaces future regressions of the same shape directly in operator logs. Fixes #30210 * fix(router): type-ignore wrapper getattr-defaults; broaden ownership-skip warning CI lint (mypy) flagged the three pre-existing getattr(..., None) assignments in FallbackResponsesStreamWrapper.__init__: router.py:2564 self.response = getattr(source_iterator, 'response', None) router.py:2565 self.model = getattr(source_iterator, 'model', None) router.py:2566 self.logging_obj = getattr(..., None) Those lines also exist on litellm_internal_staging and pass mypy there. Adding the typed terminal-event tuple above the class made the function body more narrowable, which surfaced the pre-existing mismatch — base class declares non-Optional types but the bridge path (LiteLLMCompletionStreamingIterator) legitimately omits these. Keep the None fallback and silence with type: ignore[assignment]. Greptile 4/5 note: the ownership-skip warning hard-named code_interpreter which misleads operators when a non-code_interpreter stream aborts. Generalize to 'any tool container (e.g. code_interpreter)'. * fix(register_model): drop synthesized zero costs to preserve sparse entries (#30198) (#30201) * fix(register_model): drop synthesized zero costs to preserve sparse entries (#30198) get_model_info synthesizes input_cost_per_token / output_cost_per_token = 0 when they are absent from the raw entry (the price-unknown and free cases share the same representation). register_model then merges that result back into litellm.model_cost, which flips a sparse entry from 'no cost keys' (priced via model name) to 'cost keys = 0' (free). That defeats _is_cost_explicitly_configured (#24949) on re-registration: _is_model_cost_zero returns True, common_checks skips every tag / key / team / user / org budget check for the group, and over-budget traffic keeps returning 200. Spend keeps recording because cost calc still resolves by model name, so the symptom is silent and only triggers on the second register_model pass (router rebuild, /model/update, config sync). Mirror the existing litellm_provider-None guard one block above and pop the cost fields from the synthesized result when they are absent from the raw entry and not in the caller's value. Caller-provided zeros (genuinely free models, BYOK overrides) are preserved. Fixes #30198 * fix(register_model): switch _raw_entry to is-None checks + drop dead test assertion Greptile #30201 review notes: - the `or`-chain in the raw-entry lookup treated an empty dict (a key with no fields) as falsy and fell through to the second arm — replace with explicit `is None` checks so a present-but-empty entry is still taken at face value. - the first assertion in `test_router_double_init_keeps_db_model_entry_sparse` used `in (None, 0)` which passes under the bug condition (cost = 0 matches the tuple); the strong follow-up assertion already covers every shape, so drop the dead branch. * fix(bedrock mantle): use unique function-call id for responses->chat tool calls (#30426) * fix(bedrock mantle): use unique function-call id for responses->chat tool calls ... * fix(bedrock mantle): scope unique tool-call id fallback to degenerate call_id The previous revision preferred the Responses item id for every tool call, which broke providers (and existing tests) where call_id is a unique, canonical correlation key. Restrict the fallback to the degenerate index-based call_id that Bedrock Mantle returns (call_0, call_1, ... resetting per response) and keep call_id otherwise. Revert the change to the OUTPUT_ITEM_DONE streaming handler, whose tool_call_chunk is never emitted (dead code, per review). Extend the regression tests to assert a normal call_id is preserved. * fix(router): preserve azure_ad_token through CredentialLiteLLMParams for /v1/files + batches (#30235) (#30241) * fix(router): preserve azure_ad_token through CredentialLiteLLMParams for /v1/files + batches (#30235) Router.get_deployment_credentials_with_provider re-validates a deployment's litellm_params through CredentialLiteLLMParams before handing them to file/batch/passthrough callers: return CredentialLiteLLMParams( **deployment.litellm_params.model_dump(exclude_none=True) ).model_dump(exclude_none=True) Any field NOT declared on CredentialLiteLLMParams gets silently dropped on the way through. azure_ad_token was undeclared, so Azure deployments using OAuth/M2M (azure_ad_token instead of a static api_key) silently lost their token at the files endpoint and the proxy returned: Missing credentials. Please pass one of api_key, azure_ad_token, azure_ad_token_provider, ... Declare azure_ad_token on CredentialLiteLLMParams alongside api_key / api_base / api_version so it rides through the round-trip. Static-key deployments stay unaffected (Optional, default None, dropped by exclude_none=True). Provider-callable (azure_ad_token_provider) is a separate concern and out of scope here. Fixes #30235 * fix(ui-types): regenerate schema.d.ts for new azure_ad_token field CI's 'Verify schema.d.ts matches the proxy OpenAPI spec' check auto-detected the new field and emitted the exact diff to apply. Two schemas had `aws_secret_access_key` from CredentialLiteLLMParams, both get the new azure_ad_token marker next to it. * fix(proxy): org_admin with own user_id now sees all org teams on /v2/team/list (#30247) When the UI sends the callers own user_id (as it does for non-Admin global roles), _enforce_list_team_v2_access now nulls it out for org admins so _build_team_list_where_conditions scopes by organization_id only -- matching the legacy /team/list behavior and the documented intent. Fixes #30215 Co-authored-by: Claude Opus 4.6 <[email protected]> * test(vertex_ai): multi-region regression coverage for cachedContents host (#29571) (#29707) litellm_internal_staging already routes the cachedContents URL through get_vertex_base_url, fixing the multi-region 404 reported in #29571 — but carries no test coverage for the actual regression scenario (eu/us must resolve to the REP host aiplatform.{geo}.rep.googleapis.com). Add TestContextCachingMultiRegionUrls: parametrized eu/us REP-host assertions (including absence of the old broken {geo}-aiplatform host), plus regional (us-central1) and global no-regression checks. * fix(proxy): close upstream LLM stream when client disconnects mid-stream (#30245) * fix(proxy): close upstream LLM stream when client disconnects mid-stream When a streaming client disconnects, Starlette abandons the response body iterator without calling aclose(), so the proxy's connection to the upstream backend stays open until garbage collection, which may never come. The backend (e.g. vLLM) keeps generating into a dead pipe: small responses drain invisibly into TCP buffers while large ones block the backend on a full send buffer indefinitely (observed via lsof as an ESTABLISHED proxy->backend connection minutes after the client left) create_response now returns a StreamingResponse subclass that closes both its body iterator and the wrapped upstream-facing generator in a shielded finally. The upstream generator is closed directly rather than through a cascade because aclose() on a never-started generator skips its body, which would make the cascade a no-op when the client disconnects before the first chunk is sent. async_streaming_data_generator also gains the same shielded finally-aclose that async_data_generator in proxy_server.py already had, covering the Anthropic and Google SSE paths With this, killing a streaming client causes the backend to observe the abort within about a second and free its slot, while completed streams are unaffected. No flag is needed, unlike the non-streaming opt-in cancel in #30223: this only releases resources after the client is already gone and does not change any response a client can observe Fixes #30244 * fix(proxy): close upstream even when body iterator aclose raises BaseException Addresses the Greptile finding on #30245: the cleanup loop caught only Exception while the generator-level cleanup catches BaseException, so a CancelledError or GeneratorExit escaping body_iterator.aclose() would skip closing the upstream generator. Both sites now use the same scope and a regression test pins that the upstream is closed even when the body iterator explodes with a BaseException * fix(llms): expose aclose on BaseModelResponseIterator so stream close reaches the provider connection The response-level close added for #30244 only worked for SDK-based providers (e.g. openai), whose streams expose aclose all the way down. Providers served by base_llm_http_handler (hosted_vllm and most modern transformation-based providers) wrap a bare response.aiter_lines() generator in BaseModelResponseIterator, which had no aclose or close at all, and nothing retained the httpx response object; so CustomStreamWrapper.aclose() silently did nothing and the upstream connection stayed open. Verified with a vLLM-style mock: with hosted_vllm/ the backend streamed all 100 chunks to completion after the client disconnected, while openai/ aborted at chunk 6 BaseModelResponseIterator now carries an optional http_response and an aclose() that closes it; make_async_call_stream_helper attaches the response after building the iterator. With this, hosted_vllm aborts the backend within ~1.6s of the client dropping, and completed streams are unaffected --------- Co-authored-by: kursad <[email protected]> * feat(anthropic): surface compaction usage iterations data (#27065) * feat(anthropic): surface compaction usage iterations data * style: apply black formatting to fix lint checks * fix(usage): correct calculate usage with cached tokens when use ChatCompletionUsageBlock (#30422) * fix(usage): correct calculate usage with cached tokens when use ChatCompletionUsageBlock * fix(usage): optimize test imports * feat: add fastCRW search provider (#30434) * feat(provider): add LibertAI as a JSON-configured OpenAI-compatible provider (#30203) * feat(provider): add LibertAI as a JSON-configured OpenAI-compatible provider * libertai: update served endpoints backup + add mode/matrix tests Addresses review feedback: - Add libertai to litellm/provider_endpoints_support_backup.json, the file actually served by GET /public/supported_endpoints (the root provider_endpoints_support.json already had it). - Add tests asserting bge-m3 normalizes to mode='embedding' and that the served matrix lists libertai. embeddings stays false: the JSON-configured provider path only wires chat routing (OpenAILike embedding handler is reached only for literal openai_like/llamafile/lm_studio), matching the llamagate precedent; bge-m3 remains in the cost map for metadata. --------- Co-authored-by: Moshe Malawach <[email protected]> * feat(provider): add ModelScope as an OpenAI-compatible provider (#28460) * add ModelScope API support * add modelscope api support * update modelscope model list * add image-genetation support * update test and multimodal * fix: address PR review feedback for modelscope provider * update README * fix(customer_endpoints): restrict /customer/daily/activity to admin-only (#28849) * fix(customer_endpoints): restrict /customer/daily/activity to admin-only * fix(customer_endpoints): check role before prisma_client guard * fix(custom_guardrail): key disable_global_guardrails takes precedence over team guardrail list (#28563) * fix(fallbacks): preserve fallback model in SDK fallback responses (#28260) * fix(fallbacks): preserve fallback model in response when using SDK-level fallbacks * fix(fallbacks): gate x-litellm-* passthrough to trusted callers only The previous patch unconditionally let `x-litellm-*` keys bypass the `llm_provider-` prefix in `process_response_headers`. That function is also called on raw upstream-provider response headers (e.g. from `llm_http_handler.py`), so a malicious provider could return `x-litellm-attempted-fallbacks` and spoof a LiteLLM-internal marker, bypassing the proxy model-override guard. Add a `preserve_litellm_internal_headers` flag (default False). Only `response_metadata.py`, which re-processes the already-built `_hidden_params["additional_headers"]` dict (LiteLLM-owned), passes True. Raw provider header callsites keep the default False, so upstream `x-litellm-*` still gets the `llm_provider-` prefix. Adds a regression test for the spoofing case and renames the existing preserve test to make the trusted-path semantics explicit. * fix(fallbacks): ignore preserve_litellm_internal_headers for raw httpx.Headers inputs * style(core_helpers): apply black formatting * fix(lint): remove banned typing.List/Dict/Any imports and suppress PLR0913 on interface overrides Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(lint): apply black formatting to modelscope chat transformation Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(lint): replace noqa with proper fixes — use **kwargs and Awaitable instead of Any/List Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(lint): remove unused AllMessageValues import Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * revert: restore base_model_iterator.py to original PR state Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(lint): restore full method signatures for MyPy compatibility; bump PLR0913 budget for new provider files Co-Authored-By: Claude Sonnet 4.6 <[email protected]> * fix(lint): use @override to suppress PLR0913 on inherited signatures instead of bumping budget The overrides keep their full base-class signatures for MyPy compatibility, but those signatures carry more than five parameters, which tripped PLR0913 on each subclass redeclaration. Since the arity is dictated by the base class and cannot be reduced, decorate the overrides with typing_extensions.override; ruff treats that as the intended signal that the parameter count is not under the author's control and skips PLR0913. This restores the PLR0913 baseline to 1813. * fix(lint): add @override to modelscope image generation overrides Apply the same typing_extensions.override treatment to the image generation config so its inherited-signature overrides do not count against PLR0913. --------- Co-authored-by: Joel Tony <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: hcl <[email protected]> Co-authored-by: ztko <[email protected]> Co-authored-by: Nahrin <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: Humphrey <[email protected]> Co-authored-by: kursadlacin <[email protected]> Co-authored-by: kursad <[email protected]> Co-authored-by: Dushyant Acharya <[email protected]> Co-authored-by: Yuriy <[email protected]> Co-authored-by: Recep S <[email protected]> Co-authored-by: Moshe Malawach <[email protected]> Co-authored-by: Moshe Malawach <[email protected]> Co-authored-by: Rongkun Yan <[email protected]> Co-authored-by: Varshith <[email protected]> Co-authored-by: Mateo Wang <[email protected]> * ci(lint): add blanket-noqa, dataclass-default, and unused-noqa Ruff rules (#30516) * ci(lint): enforce blanket-noqa, dataclass-default, and unused-noqa rules Enable PGH004 (blanket-noqa), RUF008 (mutable-dataclass-default), RUF009 (function-call-in-dataclass-default-argument), and RUF100 (unused-noqa) in ruff.toml, and clean up every resulting violation. RUF008/RUF009 were already clean. PGH004/RUF100 surfaced ~335 stale or blanket noqas: blanket `# noqa` are now scoped to the rule they actually suppress (mostly T201), dead directives are removed, and inapplicable codes are trimmed (e.g. F401 dropped from `import *`). lint.external lists rules enforced outside this config (the strict-rule gate via ruff-strict.toml and upstream litellm's own ruff config) so RUF100 keeps the noqa directives that protect them instead of stripping coverage this config can't see. * ci(lint): trim RUF100 external list to load-bearing codes only Drop the 9 precautionary strict-gate codes (ANN001/002/003/401, B006, PLR0913, PLW0603, RUF012, TID251) that have zero `# noqa` references in the gated source. Keep only the 11 codes with live suppressions so RUF100 doesn't flag them as unused. Future strict-gate suppressions can re-add codes here (or fix the underlying issue) as needed. * ci: ratchet lint and type-check gates (ruff preview, ANN, mypy, basedpyright) (#30379) * ci: enable ruff preview rules under the budgeted strict gate Turn on ruff preview in the strict-budget lane (ruff-strict.toml) only, leaving the clean gate (ruff.toml) untouched so make lint-ruff stays at zero. Enumerate the 118 firing codes explicitly with explicit-preview-rules so the gate is deterministic and stable across ruff upgrades rather than depending on preview auto-selecting the broad catalog. Grandfather the existing 58438 violations into ruff-strict-budget.json as per-rule baselines with headroom, so only net-new violations fail CI. The existing ten rules keep their hand-tuned slack; the new rules get slack 10 when the baseline is 50 or more and 3 otherwise. * ci: add ANN return-type rules to the budgeted strict gate Add ANN201/202/204/205/206 (missing return annotations) to the strict lane and grandfather the existing counts into ruff-strict-budget.json so the codebase ratchets toward explicit return types without breaking CI. * ci: add mypy (disallow_untyped_defs) and basedpyright strict gates with baselines Add two type-check gates, each grandfathering the current tree so only net-new violations fail CI, matching the ruff strict-budget ratchet. mypy gains disallow_untyped_defs in litellm/mypy.ini (the config the CI invocation actually reads; the root [tool.mypy] is not picked up from the litellm/ working dir). The 4885 existing missing-annotation errors are captured in litellm/.mypy-baseline.txt and the run is piped through mypy-baseline filter so new untyped defs are rejected. basedpyright runs in strict mode over litellm/, with enableTypeIgnoreComments disabled so it only honors '# pyright: ignore' and never polices mypy's '# type: ignore'. The existing strict diagnostics are grandfathered into .basedpyright/baseline.json. Both tools are pinned in the dev group and uv.lock; the lint workflow and Makefile run them filtered through their baselines, with lint-mypy-baseline-update and lint-basedpyright-baseline-update to ratchet. * ci: raise lint job timeout to 15m for the basedpyright strict pass * ci: pin pythonVersion 3.12 and regenerate baselines against merged base Merge litellm_internal_staging so the baselines cover code the CI merge includes (e.g. the cisco_ai_defense guardrail), which otherwise tripped the mypy gate with 3 ungrandfathered no-untyped-def errors. Pin pythonVersion 3.12 in pyrightconfig so basedpyright's strict analysis is reproducible across interpreter versions (CI runs 3.12). * ci: regenerate basedpyright baseline against the frozen lint env The previous baseline was generated with optional provider deps (azure, google, anthropic, mcp, numpydoc, google-genai) installed locally, so CI's dev-only env surfaced ~3500 reportUnknown*/reportMissingTypeStubs errors not in the baseline. Regenerate after uv sync --frozen so the baseline reflects the same dependency set the lint job sees. * ci: regenerate basedpyright baseline on python 3.12 frozen env The prior baseline still carried proxy-dev packages (e.g. prisma) that the lint job's dev-only, python 3.12 env lacks, leaving 2 unresolved-import errors ungrandfathered. Regenerate in a python 3.12 venv synced to the frozen lock with default groups only, so the baseline matches exactly what CI sees. * ci: replace type-check baselines with per-file count budgets The mypy and basedpyright baselines were position-sensitive (and the basedpyright one was a 27MB file), so ordinary line shifts churned them. Replace both with a per-file count gate: scripts/type_check_gate.py reduces each tool's output to errors-per-file and checks it against a committed {file: max} budget, ignoring line and column numbers. A file fails only when it gains more errors than its ceiling; debt can't be shuffled between files because each file has its own cap and new files default to zero. Budgets (mypy-file-budget.json 48K, basedpyright-file-budget.json 96K) are generated in the python 3.12 frozen lint env so they match CI. Drops the mypy-baseline dependency; basedpyright runs without its native baseline. ratchet via make lint-mypy-budget-update / lint-basedpyright-budget-update. * ci: add a small per-file slack to the type-check gate Allow each file to drift PER_FILE_SLACK (5) errors past its recorded count before failing, so a basedpyright inference ripple in an unrelated file doesn't break the build over a couple of errors. Budgets still record exact counts; the tolerance is applied at check time. * ci: move type-check slack into the budget json and trim lint timeout Make slack declarative: the budget is now {"slack": N, "files": {path: count}} so the tolerance is tuned in JSON without editing the script, mirroring how ruff-strict-budget.json carries its slack. --update preserves the existing slack. Also drop the lint job timeout from 15m to 10m; the mypy and basedpyright passes add ~2m, leaving the job around 4-5m, so 10m is a comfortable margin. * ci: collapse fully-adopted ruff categories and drop inert preview flag ANN (all nine non-removed rules) and BLE (its only rule) were spelled out code-by-code; replace each with its category selector, which is exactly equivalent in 0.15.3 (the removed ANN101/ANN102 are skipped by a category selector and error when named explicitly). explicit-preview-rules was inert: every selected rule is stable and nothing is selected by category, so the flag had nothing to gate. Verified the strict-rule counts are identical before and after (62379 each, zero per-rule drift), so no budget change. * ci: drop redundant pyright dev dependency Nothing invokes bare pyright in the Makefile, the linting workflow, or scripts; the basedpyright gate added on this branch is the only type checker that runs. based…

…43-30542-30274-0617 chore(release): backport #30543, #30542 to stable/1.88.x and cut 1.88.3

…0542) * fix(guardrails): stop re-initializing DB guardrails on every poll InMemoryGuardrailHandler._has_guardrail_params_changed compared the in-memory LitellmParams against the raw dict loaded from the DB. The in-memory side carries every field default and coerces enums via model_dump(), while the DB side only holds the keys originally stored, so the two shapes never compared equal and the guardrail was rebuilt on every poll cycle. Each rebuild created a fresh instance, but delete_in_memory_guardrail only removed the old callback from litellm.callbacks. Request handling promotes guardrail callbacks into the success/failure/async lists, so the previous instance stayed referenced there and instances accumulated. Normalize both sides through LitellmParams(...).model_dump() before diffing, and purge the callback from every callback list on delete. * refactor(guardrails): narrow params-normalization fallback to ValidationError The comparison normalizer caught a bare Exception and silently fell back to the raw dict, which hid the cause and quietly degraded the affected guardrail back to re-initializing on every poll. Catch only the ValidationError that LitellmParams construction can raise, log a warning so the offending row is diagnosable, and let any other error surface instead of being swallowed. * refactor(callbacks): add remove_callback_from_all_lists helper to manager Move the knowledge of which callback lists a callback can be promoted into out of the guardrail registry and into LoggingCallbackManager, where the rest of the callback-list bookkeeping already lives. delete_in_memory_guardrail now delegates to the new helper instead of iterating the lists itself. (cherry picked from commit 9fa74ad)

chore(release): backport #29311, #29444, #29447, #29598, #30480, #30543, #30542, #30573 to stable/1.87.x and cut 1.87.4

chore(release): backport #30480, #30543, #30542, #30573 to stable/1.89.x and cut 1.89.3

…rriAI#30542) * fix(guardrails): stop re-initializing DB guardrails on every poll InMemoryGuardrailHandler._has_guardrail_params_changed compared the in-memory LitellmParams against the raw dict loaded from the DB. The in-memory side carries every field default and coerces enums via model_dump(), while the DB side only holds the keys originally stored, so the two shapes never compared equal and the guardrail was rebuilt on every poll cycle. Each rebuild created a fresh instance, but delete_in_memory_guardrail only removed the old callback from litellm.callbacks. Request handling promotes guardrail callbacks into the success/failure/async lists, so the previous instance stayed referenced there and instances accumulated. Normalize both sides through LitellmParams(...).model_dump() before diffing, and purge the callback from every callback list on delete. * refactor(guardrails): narrow params-normalization fallback to ValidationError The comparison normalizer caught a bare Exception and silently fell back to the raw dict, which hid the cause and quietly degraded the affected guardrail back to re-initializing on every poll. Catch only the ValidationError that LitellmParams construction can raise, log a warning so the offending row is diagnosable, and let any other error surface instead of being swallowed. * refactor(callbacks): add remove_callback_from_all_lists helper to manager Move the knowledge of which callback lists a callback can be promoted into out of the guardrail registry and into LoggingCallbackManager, where the rest of the callback-list bookkeeping already lives. delete_in_memory_guardrail now delegates to the new helper instead of iterating the lists itself. (cherry picked from commit 9fa74ad)

….3) (#171) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/berriai/litellm](https://images.chainguard.dev/directory/image/wolfi-base/overview) ([source](https://github.com/BerriAI/litellm)) | patch | `v1.89.2` → `v1.89.3` | --- ### Release Notes <details> <summary>BerriAI/litellm (ghcr.io/berriai/litellm)</summary> ### [`v1.89.3`](https://github.com/BerriAI/litellm/releases/tag/v1.89.3) [Compare Source](BerriAI/litellm@v1.89.3...v1.89.3) ##### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](BerriAI/litellm@0112e53). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.89.3 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.89.3/cosign.pub \ ghcr.io/berriai/litellm:v1.89.3 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** ##### What's Changed - chore(release): backport [#30480](BerriAI/litellm#30480), [#30543](BerriAI/litellm#30543), [#30542](BerriAI/litellm#30542), [#30573](BerriAI/litellm#30573) to stable/1.89.x and cut 1.89.3 by [@yuneng-berri](https://github.com/yuneng-berri) in [#30888](BerriAI/litellm#30888) **Full Changelog**: <BerriAI/litellm@v1.89.2...v1.89.3> ### [`v1.89.3`](https://github.com/BerriAI/litellm/releases/tag/v1.89.3) [Compare Source](BerriAI/litellm@v1.89.2...v1.89.3) ##### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](BerriAI/litellm@0112e53). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.89.3 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.89.3/cosign.pub \ ghcr.io/berriai/litellm:v1.89.3 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** ##### What's Changed - chore(release): backport [#30480](BerriAI/litellm#30480), [#30543](BerriAI/litellm#30543), [#30542](BerriAI/litellm#30542), [#30573](BerriAI/litellm#30573) to stable/1.89.x and cut 1.89.3 by [@yuneng-berri](https://github.com/yuneng-berri) in [#30888](BerriAI/litellm#30888) **Full Changelog**: <BerriAI/litellm@v1.89.2...v1.89.3> </details> --- ### Configuration 📅 **Schedule**: (in timezone Europe/London) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate).  Reviewed-on: https://forgejo.hayden.moe/hayden/phoebe/pulls/171

….3) (#352) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [ghcr.io/berriai/litellm](https://images.chainguard.dev/directory/image/wolfi-base/overview) ([source](https://github.com/BerriAI/litellm)) | patch | `v1.89.2` → `v1.89.3` | --- ### Release Notes <details> <summary>BerriAI/litellm (ghcr.io/berriai/litellm)</summary> ### [`v1.89.3`](https://github.com/BerriAI/litellm/releases/tag/v1.89.3) [Compare Source](BerriAI/litellm@v1.89.3...v1.89.3) ##### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](BerriAI/litellm@0112e53). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.89.3 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.89.3/cosign.pub \ ghcr.io/berriai/litellm:v1.89.3 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** ##### What's Changed - chore(release): backport [#30480](BerriAI/litellm#30480), [#30543](BerriAI/litellm#30543), [#30542](BerriAI/litellm#30542), [#30573](BerriAI/litellm#30573) to stable/1.89.x and cut 1.89.3 by [@yuneng-berri](https://github.com/yuneng-berri) in [#30888](BerriAI/litellm#30888) **Full Changelog**: <BerriAI/litellm@v1.89.2...v1.89.3> ### [`v1.89.3`](https://github.com/BerriAI/litellm/releases/tag/v1.89.3) [Compare Source](BerriAI/litellm@v1.89.2...v1.89.3) ##### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](BerriAI/litellm@0112e53). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.89.3 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.89.3/cosign.pub \ ghcr.io/berriai/litellm:v1.89.3 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** ##### What's Changed - chore(release): backport [#30480](BerriAI/litellm#30480), [#30543](BerriAI/litellm#30543), [#30542](BerriAI/litellm#30542), [#30573](BerriAI/litellm#30573) to stable/1.89.x and cut 1.89.3 by [@yuneng-berri](https://github.com/yuneng-berri) in [#30888](BerriAI/litellm#30888) **Full Changelog**: <BerriAI/litellm@v1.89.2...v1.89.3> </details> --- ### Configuration 📅 **Schedule**: (in timezone America/New_York) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about these updates again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate).  Reviewed-on: https://git.greyrock.io/greyrock-labs/home-ops/pulls/352

…to v1.89.3 (#216) This PR contains the following updates: | Package | Update | Change | |---|---|---| | [https://github.com/BerriAI/litellm.git](https://github.com/BerriAI/litellm) | patch | `v1.89.2` → `v1.89.3` | --- ### Release Notes <details> <summary>BerriAI/litellm (https://github.com/BerriAI/litellm.git)</summary> ### [`v1.89.3`](https://github.com/BerriAI/litellm/releases/tag/v1.89.3) [Compare Source](BerriAI/litellm@v1.89.2...v1.89.3) #### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](BerriAI/litellm@0112e53). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.89.3 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.89.3/cosign.pub \ ghcr.io/berriai/litellm:v1.89.3 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** #### What's Changed - chore(release): backport [#30480](BerriAI/litellm#30480), [#30543](BerriAI/litellm#30543), [#30542](BerriAI/litellm#30542), [#30573](BerriAI/litellm#30573) to stable/1.89.x and cut 1.89.3 by [@yuneng-berri](https://github.com/yuneng-berri) in [#30888](BerriAI/litellm#30888) **Full Changelog**: <BerriAI/litellm@v1.89.2...v1.89.3> </details> --- ### Configuration 📅 **Schedule**: (UTC) - Branch creation - At any time (no schedule defined) - Automerge - At any time (no schedule defined) 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 🔕 **Ignore**: Close this PR and you won't be reminded about this update again. --- - [ ] If you want to rebase/retry this PR, check this box --- This PR has been generated by [Mend Renovate](https://github.com/renovatebot/renovate).  Co-authored-by: Renovate Bot <[email protected]> Reviewed-on: https://codeberg.org/blake-hamm/bhamm-lab/pulls/216

…rriAI#30542) * fix(guardrails): stop re-initializing DB guardrails on every poll InMemoryGuardrailHandler._has_guardrail_params_changed compared the in-memory LitellmParams against the raw dict loaded from the DB. The in-memory side carries every field default and coerces enums via model_dump(), while the DB side only holds the keys originally stored, so the two shapes never compared equal and the guardrail was rebuilt on every poll cycle. Each rebuild created a fresh instance, but delete_in_memory_guardrail only removed the old callback from litellm.callbacks. Request handling promotes guardrail callbacks into the success/failure/async lists, so the previous instance stayed referenced there and instances accumulated. Normalize both sides through LitellmParams(...).model_dump() before diffing, and purge the callback from every callback list on delete. * refactor(guardrails): narrow params-normalization fallback to ValidationError The comparison normalizer caught a bare Exception and silently fell back to the raw dict, which hid the cause and quietly degraded the affected guardrail back to re-initializing on every poll. Catch only the ValidationError that LitellmParams construction can raise, log a warning so the offending row is diagnosable, and let any other error surface instead of being swallowed. * refactor(callbacks): add remove_callback_from_all_lists helper to manager Move the knowledge of which callback lists a callback can be promoted into out of the guardrail registry and into LoggingCallbackManager, where the rest of the callback-list bookkeeping already lives. delete_in_memory_guardrail now delegates to the new helper instead of iterating the lists itself.

* feat(ui): gate "Default Credentials" hint on /ui/login behind env flag (#30234) Adds LITELLM_HIDE_DEFAULT_CREDENTIALS_HINT (and an equivalent general_settings.hide_default_credentials_hint) that suppresses the "By default, Username is admin and Password is your set LiteLLM Proxy MASTER_KEY" info card rendered on /ui/login and /fallback/login. Motivation: in production deployments operators set UI_USERNAME / UI_PASSWORD (or SSO), and the hardcoded hint becomes factually incorrect and is flagged by security scanners (Tenable WAS plugin 114625) as information disclosure. There is currently no way to suppress it without forking the dashboard. Behaviour: - Default is unchanged (hint shown), so existing deployments are unaffected. - New field hide_default_credentials_hint on the well-known UI config endpoint, populated from the env var or general_settings. - LoginPage.tsx conditionally renders the Alert based on the flag. Refs: BerriAI/litellm#30232 * fix(router): clean pattern_router state on upsert/delete (#29601) * fix(router): clean pattern_router state on upsert/delete PatternMatchRouter.add_pattern was append-only, and neither Router.upsert_deployment nor Router.delete_deployment removed the existing entry. Rotated-out api_keys stayed in the routing rotation for wildcard deployments (model_name with `*`) until proxy restart, silently defeating key rotation as an admin operation. The same leak applied to provider_default_deployment_ids and per-team pattern routers, and the patterns list grew unboundedly on every edit * test(router): direct unit tests for _remove_deployment_from_wildcard_state router_code_coverage.py greps test files for AST Call nodes and flagged the helper as untested because the existing coverage only exercised it transitively through upsert/delete. Adds two direct tests that pin the helper's contract (cleans across global pattern router, per-team routers with empty-router pop, and provider_default_deployment_ids; noop on falsy model_id) * fix(router): address Greptile review on pattern_router cleanup Widen PatternMatchRouter.remove_deployment annotation to Optional[str]; the implementation already handles None via the falsy guard and the unit test exercises it directly. Move _remove_deployment_from_wildcard_state up one level in upsert_deployment so it runs whenever the prior deployment is on the router, not only when the model_id is present in the fast-mapping index. The scenario is currently unreachable (get_deployment shares the same index), but the cleanup is idempotent so this is defensive against any future divergence between those code paths. * fix(router): widen _remove_deployment_from_wildcard_state to Optional[str] Moving the call out of the inner `deployment_id in deployment_fast_mapping` block in the previous commit lost mypy's narrowing of `deployment_id` from Optional[str] to str, tripping the lint CI. The helper already handles None via its falsy guard, so widening the annotation matches the actual contract. * fix(router): make delete_deployment wildcard cleanup symmetric with upsert After the previous commit moved _remove_deployment_from_wildcard_state out of the inner index-map guard in upsert_deployment, delete_deployment was still calling it only inside `if deployment_idx is not None`. Greptile flagged the asymmetry: under a desynced index_map, delete would silently leave the stale wildcard credential in pattern_router. Moves the cleanup call to the top of the try block, mirroring the upsert path. Cleanup is idempotent so the change is a no-op on the happy path. Adds a regression test that simulates the desync by removing the entry from model_id_to_deployment_index_map and asserts delete still clears pattern_router. * fix(pricing): add 1h cache-write cost for Anthropic Sonnet 4.5/4.6 (#30474) The native anthropic claude-sonnet-4-5/4-6 price-map entries were missing cache_creation_input_token_cost_above_1hr (and the >200K long-context sub-tier for 4.5), so 1-hour-TTL cache writes were costed at the 5-minute rate. Adds 6e-06 regular (and 1.2e-05 long-context) = 2x base input, matching the vertex_ai/azure_ai/bedrock siblings and the older claude-sonnet-4-20250514 entry. Adds a regression test. * fix(proxy): cancel upstream gemini request and release httpx connection on client disconnect (#30075) * fix(proxy): cancel upstream gemini request and release httpx connection on client disconnect - add _check_request_disconnection to common_request_processing; wrap llm_call as asyncio.Task so it can be cancelled; catch CancelledError and raise HTTPException(499) when client disconnects before LLM responds (non-streaming path) - pass raw httpx.Response into ModelResponseIterator in make_call/make_sync_call so the iterator holds a reference to the underlying connection - implement ModelResponseIterator.aclose() and .close(): close the line iterator then explicitly call response.aclose()/response.close() to release the httpx connection when the client drops mid-stream; errors are debug-logged, not raised - add tests for _check_request_disconnection (cancels task, graceful on exception, does not cancel when client stays connected) and base_process_llm_request 499 behavior; add TestModelResponseIteratorCleanup verifying aclose/close propagation through CustomStreamWrapper * fix(proxy): record 499 on streaming disconnect and cancel orphaned gather tasks Wire streaming generator cleanup to log client_disconnected with error_code 499 in spend logs, cancel pending during_call_hook tasks when the LLM call is cancelled on disconnect, and align the 600s poll limit comment with proxy_server. * fix: extract client disconnect logging helper to satisfy PLR0915 * fix: resolve mypy and code-quality CI failures for client disconnect logging Cast client disconnect error_information for mypy, only await pending gather tasks to avoid masking LLM errors, and add tests for the new logging helper and gather cleanup. * fix(proxy): harden gather cleanup so finally cannot mask LLM errors * fix(proxy): shield streaming disconnect logging and strip spoofable metadata Move streaming disconnect recording into a shielded cancel scope, add gather cleanup regression coverage for guardrail-converted cancels, and strip client_disconnected/error_information from user metadata at the proxy boundary. * fix(proxy): only map CancelledError to 499 for client disconnect Track when the disconnect poller cancels the LLM task and re-raise other CancelledError paths so graceful shutdown is not reported as HTTP 499. * fix(proxy): remove dead _check_request_disconnection helper Non-streaming client disconnect is handled by staging's cancel_on_disconnect path via _await_llm_call_cancelling_on_disconnect. Drop the unused is_disconnected poller and its unit tests; rename the remaining integration tests to TestDisconnectGatherCleanup. * feat(mistral): add mistral-medium-3-5 to model_prices_and_context_wind.. (#29303) * feat(mistral): add mistral-medium-3-5 to model_prices_and_context_window.json Mistral's docs page lists mistral-medium-3-5 as a new model offering. Pricing/specs sourced from Mistral's published model metadata: - input: $1.50 / 1M tokens - output: $7.50 / 1M tokens - context: 262,144 tokens - capabilities: vision, function calling, structured outputs, assistant prefill Adds entry: `mistral/mistral-medium-3-5`, mirroring the pattern used for the rest of the Mistral family. test(mistral): add model_info test for mistral-medium-3-5 + sync backup cost map - Mirror mistral/mistral-medium-3-5 entries into litellm/model_prices_and_context_window_backup.json so the bundled model cost map matches the canonical model_prices_and_context_window.json. - Add tests/test_litellm/test_mistral_medium_3_5_model_metadata.py covering pricing tiers, capability flags, context window, provider routing, and parity between the main and backup cost maps. - Point 'source' at the live Mistral models documentation page. * fix(ui): three small UI fixes — Gemini api_base + credential form reset + Mode badge (#30419) * fix(ui): three small UI fixes — Gemini api_base field + credential form reset + Mode badge Three independent fixes; bundled because they all touch the credential-form / logging-callbacks area. 1. expose api_base field on Google AI Studio credential form The runtime gemini provider supports custom api_base via `vertex_llm_base._check_custom_proxy`; the UI just needs to expose the field. Adds api_base to the Google_AI_Studio credential form ordered before api_key (matching OpenAI/Anthropic conventions). Default value matches the canonical Google AI Studio endpoint that LiteLLM's gemini provider talks to when api_base is unset, so leaving the default in the form behaves identically to leaving it blank. 2. reset credential form state when switching providers Switching the Provider select in AddCredentialModal / EditCredentialModal left the previous provider's field values populated. The form then submitted a mixed payload (e.g. Azure deployment fields under an OpenAI credential), producing confusing failures. Extract `getProviderFieldDefaults` helper and reset the form to it on provider change. Unit-tested via the extracted helper because Antd Select's portal/dropdown behaviour is unreliable in jsdom. 3. logging callbacks table reads backend `type` for Mode badge (#35) The `/get_callbacks` proxy endpoint returns each callback as `{name, type, variables}` where `type` is `"success"` or `"failure"`. The same callback name can appear twice (one per event class) and the two entries fire on disjoint events. `LoggingCallbacksTable` ignored `type` and read `record.mode` (always undefined), so every row fell back to the "Success" badge. A `generic_api` callback registered for both classes showed up as two identical "Success" rows + React duplicate-key warning. Read `record.type` first (fall back to `record.mode` for newly- added not-yet-server-acknowledged rows). Composite rowKey `${name}-${type ?? mode ?? 'success'}`. Removed leftover debug `console.log`. * fix(ui): drop api_base default_value to preserve Gemini v1alpha auto-routing Greptile P2 (PR #30419, threads on lines 1255-1256 of provider_create_fields.json): the api_base field's `default_value` was hard-coded to "https://generativelanguage.googleapis.com/v1beta". This: 1. Bakes v1beta into every credential record saved through the form, even when the user never touched the field. If LiteLLM's internal gemini default URL ever changes, those persisted credentials keep hitting the stale path. 2. Bypasses `_get_gemini_url`'s automatic version routing for Gemini 3+ models. That helper picks v1alpha for Gemini 3+ and v1beta for older models when api_base is unset. With the default pre-filled (and `_check_custom_proxy` then taking over because api_base is non-empty), Gemini 3+ requests get pinned to v1beta and may fail or behave unexpectedly — purely because the user accepted the visible default. Fix: set `default_value` to `null` and move the canonical URL guidance into the `placeholder` (visible to the user, never persisted) and an expanded tooltip. UX is unchanged — the URL is still shown in the greyed-out input — but the auto-version-routing path stays default. Updated test_google_ai_studio_provider_fields_expose_api_base to assert the new contract (`default_value is None`, `placeholder` carries the canonical URL), with a comment pointing at the Greptile threads as the rationale so future contributors don't accidentally re-introduce the default. 26/26 tests in the file pass. JSON validates (`json.load` clean). * feat(azure_ai): add gpt-5.5 to model cost map (#30428) * feat(azure_ai): add gpt-5.5 to model cost map Adds azure_ai/gpt-5.5 and its dated snapshot azure_ai/gpt-5.5-2026-04-23 to both the canonical and bundled cost maps. gpt-5.5 is generally available on Azure AI Foundry; pricing mirrors the openai gpt-5.5 entry, matching the established azure_ai convention (verified identical for gpt-5.4), in the azure tier structure (base / above-272k / priority). supports_minimal_ reasoning_effort is false, the capability that changed from gpt-5.4. Fixes #30306 * Update tests/test_litellm/test_gpt_5_5_model_metadata.py Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> --------- Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com> * fix: guard check_and_fix_namespace against None key (#30435) * fix: guard check_and_fix_namespace against None key When user_id is None, the cache key can be None, causing AttributeError: 'NoneType' object has no attribute 'startswith' in check_and_fix_namespace. Add an early return for None key to prevent the error and the ERROR-level log noise it produces on every unauthenticated request. Fixes #30424 * fix: update type annotations for check_and_fix_namespace - key: str -> Optional[str] (now handles None input) - return: str -> Optional[str] (returns None when input is None) Addresses Greptile review concern about type signature mismatch. * fix: revert check_and_fix_namespace type signature to str to fix MyPy downstream errors * fix: update type annotations for check_and_fix_namespace - Change signature from str -> str to Optional[str] -> Optional[str] - Remove type: ignore comment on None return - Add None guard in async_set_cache_sadd before passing to helper Addresses review feedback from Sameerlite on type mismatch. * Revert "fix: update type annotations for check_and_fix_namespace" This reverts commit 5272920fa0daab676f5ad46dcadd8cd537cfc96f. --------- Co-authored-by: michaelxer <[email protected]> * fix(cost): apply service_tier suffix to above-threshold cache rates and expose priority+threshold keys in ModelInfo (#30450) * fix(cost): apply service_tier suffix to above-threshold cache rates and expose priority+threshold keys in ModelInfo Models that publish both a service_tier (e.g. priority) rate and an above-threshold tier (e.g. _above_200k_tokens) currently bill cached tokens at the standard above-threshold rate rather than the priority above-threshold rate. Affected entries in the live pricing JSON include gemini-3-pro-preview, gemini-3.1-pro-preview and their vertex_ai/ and gemini/ variants, plus azure/gpt-5.4 and azure_ai/gpt-5.4. For a 250K-token priority request with 200K cached tokens against gemini-3-pro-preview, the leak is about 44 percent of the prompt cost. Two stacked defects caused this. First, ModelInfoBase (and the ModelInfo pydantic class) and the get_model_info construction in litellm/utils.py omit the priority+above-threshold cost keys, so even if the calculator asked for them they would never reach it. Second, in _get_token_base_cost the cache_creation/cache_read tiered keys never get wrapped with _get_service_tier_cost_key, while the input/output tiered keys above and below do. The change here surfaces six new keys (input, output and cache_read at both 200k and 272k priority variants) and wraps the three cache tiered keys in _get_token_base_cost the same way input/output already are. _get_cost_per_unit's existing service_tier-to-base fallback covers models that ship the standard above-threshold rate without a priority variant. Adds one regression test in tests/test_litellm/litellm_core_utils/llm_cost_calc/test_llm_cost_calc_utils.py that drives the actual generic_cost_per_token path for gemini-3-pro-preview at 200K cached + 50K text under priority and asserts the priority above_200k rates are picked. Verified the test fails on litellm_internal_staging without these changes and passes with them. * fix(cost): drop guard on cache tiered keys so service_tier fallback can reach standard above-threshold rate Addresses Greptile P1 on PR 30450. The previous commit wrapped cache_creation_tiered_key, cache_creation_1hr_tiered_key, and cache_read_tiered_key with _get_service_tier_cost_key (matching how the sibling input and output tiered keys are wrapped) but kept the surrounding 'if key in model_info' guards. For models that publish a standard above-threshold cache rate but no priority variant (gpt-5.4-pro, gpt-5.5-pro and their dated siblings, plus vertex_ai/claude-sonnet-4-5 for cache_creation), the guard short-circuits before _get_cost_per_unit's existing service_tier-to-base fallback can strip _priority and find the standard above-threshold key. The result on priority requests over the threshold was that those models silently dropped from the above-threshold rate back to the priority-base rate. Dropping the guard and calling _get_cost_per_unit unconditionally (mirroring how tiered_input_key and tiered_output_key are already handled) restores correct billing for that class of models while keeping the new priority+above-threshold behaviour for gemini-3-pro-preview and friends. Adds a second regression test that pins generic_cost_per_token for vertex_ai/claude-sonnet-4-5 priority + above_200k with cached and cache_creation tokens to the expected standard above-threshold rates, so the guard cannot be silently reintroduced for either the cache_read or cache_creation path. * fix(presidio): skip pre-call masking when guardrail is logging_only (#30461) The Presidio pre-call hook masked the live request unconditionally, ignoring the configured event hook. With mode: logging_only the masked request reached the model, so its response echoed anonymization tokens (e.g. <PERSON>) instead of the real output. Gate async_pre_call_hook on should_run_guardrail, matching every other guardrail; logging_only masking still happens via async_logging_hook. * fix(router): resolve list unhashable crash on model alias (#30464) * fix(router): resolve list unhashable crash on model alias Fixes the fallback parsing logic that mistakenly categorized standard array fallback definitions as override dictionaries when a deployment alias matches the literal string 'model'. Closes https://github.com/BerriAI/litellm/issues/30459 * fix(router): address greptile review for fallback parsing edge cases - Resolves ambiguity in standard vs override fallback dictionaries by iterating over all items and validating that no mapped litellm param resolves to a non-list type. - Adds regression tests in test_router_order_fallback.py to prevent unhashable type crash from silently re-entering the codebase. * chore(router): format code with black to pass CI * fix(hosted_vllm): remove thinking_blocks and convert list content to strings (#30475) * fix: hosted_vllm remove thinking_blocks and convert list content to strings vLLM endpoints reject assistant messages with thinking_blocks converted to content list blocks. This change removes thinking_blocks entirely and converts any list content back to strings. This fixes BadRequestError when using Claude Code with hosted_vllm models that pass thinking_blocks in messages. * fix(hosted_vllm): address Greptile review feedback - Join multiple text blocks with newline instead of empty string - Always set content to string (never None) to avoid vLLM validation errors * fix(hosted_vllm): update chat transformation to clean assistant messages * fix: re-raise exception instead of silently dropping MCP team permissions (#30477) * fix: re-raise exception instead of silently dropping MCP team permissions When MCPRequestHandler.get_allowed_mcp_servers raises, the broad except was swallowing the error and returning only allow_all_server_ids, silently discarding all team-level object_permission grants. Fixes #30476 * fix: log full traceback when MCP permission lookup fails Uses verbose_logger.exception() instead of warning() so operators can see the full traceback when team-level object_permission grants are dropped due to an internal error in get_allowed_mcp_servers. Fixes #30476 * fix: remove timezone date expansion in daily-activity aggregation (#29569) * fix: remove timezone date expansion in daily-activity aggregation Single-day spend queries from non-UTC timezones over-counted by ~2x because the previous implementation widened the SQL date range by a full UTC day on whichever side the offset pointed. Spend is bucketed in whole-UTC-day rows in LiteLLM_DailyUserSpend, so the expansion pulled an extra 24h of unrelated bucket data per boundary. Concretely on IST (UTC+5:30, offset -330): a single-day query for 2026-05-29 was rewritten to date >= 2026-05-28 AND date <= 2026-05-29 and returned spend across both UTC days. Sums of single-day queries across a 5-day window then exceeded the equivalent multi-day aggregate by ~50%, which is mathematically impossible. Treat the local date range as the UTC date range. The aggregation table has no hour-level granularity, so any conversion using only date arithmetic must round to whole UTC days; the previous fix turned that boundary slop into systematic over-counting. Pass-through trades a small one-time slop at each end of the range for correct, monotonic, additive results across single-day and multi-day queries. Repro from production: bedrock/global.anthropic.claude-opus-4-8 over 2026-05-29 to 2026-06-02, IST timezone: - 5-day aggregate: $701.39 / 1,831 reqs - Sum of 5 single-day queries: $1,070.94 / 2,755 reqs - Excess (was 1.527x): now matches within boundary slop Adds regression tests in TestAdjustDatesForTimezone and TestBuildAggregatedSqlQuery that pin the pass-through behavior and the additivity invariant for any future implementation. * ci: rerun checks on litellm_oss_branch base --------- Co-authored-by: Sameer Kankute <[email protected]> * fix: buffer native gemini sse frames (#30225) * fix: buffer native gemini sse frames * fix: scope native gemini sse buffering * fix: check raw sse residual buffer size * feat: updated openrouter provider to map max level to xhigh (#28881) * feat(proxy): allow use_redis_transaction_buffer without redis cache (#28764) * feat(proxy): allow use_redis_transaction_buffer without redis cache * fix(proxy): require host or url for standalone buffer redis * fix(mcp): fail closed when scope filter resolves to no servers (#30353) `_get_allowed_mcp_servers_from_mcp_server_names` returned the caller's full allowed-server set when the requested `mcp_servers` list (path- or header-derived) resolved to nothing. URL/header namespacing therefore appeared to work even when the requested name was unknown or the caller had no grant — `/mcp/<typo>/` silently exposed every server the key could reach. Fail closed instead: when `mcp_servers` is explicitly provided but nothing resolves, return an empty list. The `mcp_servers=None` path (no scope requested) keeps its existing behavior. * fix(token-counter): handle Anthropic tool_reference blocks to stop dropped spend logs (#30302) * fix(token-counter): handle Anthropic tool_reference blocks to stop dropped spend logs `token_counter` did not know about Anthropic tool-search `tool_reference` content blocks, a lightweight pointer to a deferred tool that shows up as `{"type": "tool_reference", "tool_name": ...}`. When such a block appeared in message content, `_count_content_list` fell through to its catch-all branch and raised `Invalid content item type: tool_reference`. On the streaming `anthropic_messages` proxy path that exception nulls `response_cost`, which makes the proxy drop the entire SpendLogs row. The result is a silent cost undercount on any tool-search traffic; the request succeeds for the caller but the spend is never recorded. This adds a `tool_reference` branch that counts the referenced `tool_name` (the full tool definition is already counted via the `tools` param, so only the name is added here) and handles an empty/missing name gracefully. The catch-all error message is updated to list `tool_reference` among the expected types. A regression test asserts that a message containing a `tool_reference` block no longer raises and returns a positive token count, and that an empty `tool_name` is handled without error. * fix(token-counter): collapse explicit None tool_name to empty string In _count_content_list, c.get("tool_name", "") returns None when the key is present with an explicit None value, and str(None) == "None" which is truthy, causing a spurious token to be counted. Use c.get("tool_name") or "" so both a missing key and an explicit None collapse to an empty string and are skipped. * test(token-counter): cover catch-all for unknown content block type Adds a regression test that calls `_count_content_list` with an unrecognized content block type and asserts it raises `ValueError` whose message names the offending type and lists `tool_reference` among the supported types. This exercises the previously uncovered catch-all branch (codecov patch gap) and pins the error contract. * test(token-counter): cover tool_reference on the spend/cost and streaming paths Adds end-to-end regression tests that exercise the real public entry points (`completion_cost` and `stream_chunk_builder`), not just the private `_count_content_list` helper, for Anthropic tool-search `tool_reference` content blocks. These pin the actual bug the fix addresses: before the fix the `tool_reference` block raised out of `completion_cost` -> the proxy logging layer nulled `response_cost` and the spend callback dropped the SpendLogs row (silent cost undercount on all tool-search traffic); and `stream_chunk_builder` swallowed the same raise and collapsed prompt_tokens to 0. With the fix, cost is positive and prompt_tokens are counted. Verified: 3 fail without the fix, 3 pass with it. --------- * feat(cost): add cost mapping for deepseek-v4-flash and deepseek-v4-pro (#27056) * feat(cost): add cost mapping for deepseek-v4-flash and deepseek-v4-pro Adds pricing entries for the two new DeepSeek V4 models released on 2026-04-24, for both bare model names and the deepseek/ provider prefix. Prices sourced from https://api-docs.deepseek.com/quick_start/pricing: - deepseek-v4-flash: $0.14/M input, $0.28/M output - deepseek-v4-pro: $1.74/M input, $3.48/M output Cache hit price set to 1/10 of input (per DeepSeek docs). Context window: 1M tokens for both models. Closes #26709 * fix(cost): update backup registry for deepseek-v4 * style: remove print statement from deepseek-v4 test * feat(cost): add cost mapping for deepseek-v4-flash and deepseek-v4-pro Adds pricing entries for the two new DeepSeek V4 models released on 2026-04-24, for both bare model names and the deepseek/ provider prefix. Prices sourced from https://api-docs.deepseek.com/quick_start/pricing: - deepseek-v4-flash: $0.14/M input, $0.28/M output - deepseek-v4-pro: $1.74/M input, $3.48/M output Cache hit price set to 1/10 of input (per DeepSeek docs). Context window: 1M tokens for both models. Closes #26709 * fix: update deepseek-v4 prices to active discounted rates * test: update deepseek-v4 prices in tests to match active discounted rates * fix(deepseek): remove duplicate entries and update backup registry to active discounted rates * fix: update max_output_tokens to 384K for deepseek-v4 * fix: correctly restore upstream models accidentally dropped during merge * fix(tests): resolve failing claude-fable-5 and reasoning tests by safely updating cost map - Pulled the latest cost map from upstream staging - Safely appended deepseek-v4 mapping without deleting duplicate keys or formatting via json.dump * fix(tests): correct deepseek model cache prices and update JSON schema - Appended both prefixed and bare deepseek-v4 models to satisfy test assertions - Corrected deepseek-v4-pro expected cache hit and token prices based on latest review updates - Added missing realtime endpoint to test_utils.py INTENDED_SCHEMA * fix: remove accidental azure/gpt-realtime-whisper addition --------- Co-authored-by: Dushyant Acharya <[email protected]> * feat(key/info): expose per-model budget usage in /key/info response (#30394) * feat(key/info): expose per-model budget usage in /key/info response Add model_max_budget_usage to /key/info and /v2/key/info responses. For each model in model_max_budget, reads current-period spend from the same DualCache used by the budget enforcer and returns it alongside the limit and time period so callers can see how much of each model budget has been consumed in the active window. * test(key/info): add coverage for model_max_budget_usage in v1 and v2 endpoints Add tests for the model_max_budget_usage enrichment in both info_key_fn and info_key_fn_v2, covering the budget-present path, the empty-budget path, and the v2 batch endpoint. * fix(key/info): source model_max_budget current_spend from SpendLogs instead of DualCache The DualCache used for enforcement is ephemeral and only populated when budget metadata is present at request time. Fall back to a direct LiteLLM_SpendLogs DB aggregation using the budget period window (budget_reset_at - budget_duration) for accurate reporting. Also fall back to litellm_budget_table.model_max_budget when the key's top-level field is empty, and round current_spend to 4 decimal places. * test(key/info): cover remaining branches in model_max_budget_usage helpers Add unit tests for: prisma_client=None early return, DB query exception swallowing, invalid budget_duration handled by _compute_budget_period_start, budget_reset_at received as a datetime object (Prisma native type), max_seconds=0 early return, and skipping models that lack a budget_duration. Also remove an unreachable except branch where fromisoformat would fail after _compute_budget_period_start already validated the same value. * test(key/info): cover except path for unparseable per-model budget_duration * fix(key/info): compute per-model rolling windows in model_max_budget_usage Each model in model_max_budget now gets its own time window derived from its own budget_duration, rather than sharing a single window computed as the max (or the budget table's reset_at). This matches what the DualCache enforcer actually tracks and prevents current_spend from being inflated for models with shorter windows. _query_model_spend_for_period is refactored to accept a model filter (handling provider-prefix variants in SQL) and return a float directly. _compute_budget_period_start and the budget_table window path are removed as they are no longer needed. * refactor(model_max_budget_limiter): remove dead get_current_period_spend method * refactor(key/info): strip synthetic formatter noise from PR diff Restore key_management_endpoints.py and test_key_management_endpoints.py to origin/litellm_internal_staging, then re-apply only the intentional additions: _query_model_spend_for_period, _build_model_max_budget_usage, the two endpoint patches (info_key_fn / info_key_fn_v2), and the new test suite. The previous commits had reformatted ~300 pre-existing lines across both files, making the functional diff unreadable. * test(key/info): cover empty-rows path in _query_model_spend_for_period * fix(model_max_budget_limiter): guard BudgetConfig construction inside try/except A malformed model entry in the DB (e.g. non-numeric max_budget from a manually edited or migrated row) caused BudgetConfig(**budget_info) to raise a Pydantic ValidationError outside any exception guard, surfacing as a 500 for the entire /key/info or /v2/key/info call. Merging both try/except blocks into one ensures bad entries are silently skipped, consistent with the existing duration_in_seconds guard. * fix: don't stack provider prefix on wildcard models with a custom prefix (#30360) * fix: don't stack provider prefix on wildcard models with a custom prefix get_known_models_from_wildcard expanded provider-prefixed model ids (e.g. "ollama/gemma3:1b" from get_provider_models) by prepending the wildcard's prefix whenever the id did not already start with it. With a custom wildcard prefix such as "ollama_server1/*" (used to distinguish multiple Ollama instances), this produced "ollama_server1/ollama/gemma3:1b", which is uncallable and breaks /v1/models. When the expanded id already carries a provider prefix, replace it with the wildcard's prefix instead of stacking both. Matching-prefix and bare-model cases are unchanged. Fixes #30358 * fix: only strip a known provider prefix when expanding custom wildcard prefixes The wildcard expansion replaced the leading slash segment of every expanded id with the wildcard prefix whenever the id did not already start with it. For ids whose first segment is an org rather than a litellm provider (for example a provider returning "meta-llama/Llama-3-8B" with no outer provider prefix), that dropped the org and produced an uncallable id Only strip the leading segment when it is a recognized provider (membership in LlmProviders); otherwise keep it and just prepend the wildcard prefix. Provider-prefixed ids like "ollama/gemma3:1b" still have their prefix replaced, so the original fix is unchanged for known providers * address greptile review feedback: log dropped non-text vLLM assistant content blocks (greploop iteration 1) * fix(ci): format credential_form_helpers test + regenerate dashboard schema.d.ts * fix(proxy): raise litellm.BadRequestError for missing model param When no model is passed, route_request now raises a litellm.BadRequestError ('Missing model parameter') instead of falling through to ProxyModelNotFoundError. This keeps the missing-param error clear and independent of router wildcard state. Unknown (non-empty) model names still raise ProxyModelNotFoundError. * Revert "fix(proxy): raise litellm.BadRequestError for missing model param" This reverts commit 9240da403c0432a80473d6c4677ddb7e2bad7420. * Revert "fix(router): clean pattern_router state on upsert/delete (#29601)" This reverts commit ad4e6e2395620ea6d2fe38089a54cde160720de2. * fix: correct streaming and key budget usage reporting * fix(hosted_vllm): type assistant tool_calls to satisfy mypy * feat: aws secret manager cross region replication (#30368) * feat(aws-secret-manager): add replica_regions cross-region replication after CreateSecret When store_virtual_keys is enabled, async_write_secret() only wrote secrets to the primary AWS region. Multi-region proxy deployments had no built-in way to synchronize virtual key secrets across regions through LiteLLM, requiring external replication mechanisms. Add replica_regions support to AWSSecretsManagerV2: - New replica_regions field in KeyManagementSettings (types/secret_managers/main.py) - New async_replicate_secret() method that calls ReplicateSecretToRegions API - async_write_secret() calls replication after successful CreateSecret - Replication failure is logged as a warning but does NOT fail key creation - load_aws_secret_manager() forwards replica_regions from key_management_settings Configuration example: key_management_settings: store_virtual_keys: true replica_regions: - us-west-2 - eu-west-1 When replica_regions is omitted or empty, behavior is unchanged. * test(aws-secret-manager): restore litellm.secret_manager_client after test to prevent state pollution * test(aws-secret-manager): add coverage for HTTP error and replication exception paths * fix: restore litellm.secret_manager_client global state in test; add replication log proof - Global state in test_load_aws_secret_manager_passes_replica_regions was already guarded with try/finally (committed in previous pass); no further change needed for Fix 1. - Fix 2: add verbose_logger.info("ReplicateSecretToRegions called …") inside async_replicate_secret so callers get an observable INFO log line whenever replication fires. - Add test_replication_fires_on_create: calls async_replicate_secret directly with caplog.at_level(INFO, logger="LiteLLM") and asserts "ReplicateSecretToRegions" appears in the captured log output, proving the code path executes. * fix: pass request to streaming generators * fix(hosted-vllm): preserve assistant structured content * fix(hosted_vllm): satisfy mypy on preserved structured content assignment * chore: resolve litellm_internal_staging merge conflicts for #30527 (#30554) * chore(codecov): add Batches, Videos, and Realtime components (#30517) * chore(codecov): add Batches, Videos, and Realtime components Define per-feature Codecov components so PR comments track coverage for batch API, video generation, and realtime streaming paths. Co-authored-by: Cursor <[email protected]> * chore(codecov): use wildcard path for Batches proxy component Align batches_endpoints glob with Videos, Realtime, and Proxy_Authentication. Co-authored-by: Cursor <[email protected]> --------- Co-authored-by: Cursor <[email protected]> * test(batches): move orphan tests into tests/test_litellm for CI coverage (#30510) Four batch-related tests lived under tests/litellm/ and were never picked up by GitHub Actions. Relocate them and fix gemini multimodal e2e to use the batchEmbedContents path expected for gemini/ provider. Co-authored-by: Cursor <[email protected]> * fix(guardrails): run pre_call hook once for model-level guardrails (#30543) * fix(guardrails): run pre_call hook once for model-level guardrails A CustomGuardrail attached to a deployment via litellm_params.guardrails gets its async_pre_call_hook invoked twice per request: once by the proxy pre-call loop and again by async_pre_call_deployment_hook after the router spreads the model-level guardrails into the top-level request kwargs. Record in request metadata that the proxy pre-call loop already ran a given guardrail, and have the deployment hook skip it when the marker is present. Direct-SDK usage never runs the proxy loop, so the deployment hook stays the sole invocation there and still fires exactly once. The marker key is stripped from untrusted caller metadata so a request body cannot suppress a model-only guardrail by pre-seeding it. * fix(guardrails): mark pre_call dedup on the post-hook request data Record the exactly-once marker after async_pre_call_hook runs, on the data object that flows downstream, rather than before it. A guardrail whose hook returns a brand-new request dict (instead of mutating or spreading the one it received) would otherwise discard the marker, letting the deployment hook re-run the guardrail a second time. * fix(guardrails): stop re-initializing DB guardrails on every poll (#30542) * fix(guardrails): stop re-initializing DB guardrails on every poll InMemoryGuardrailHandler._has_guardrail_params_changed compared the in-memory LitellmParams against the raw dict loaded from the DB. The in-memory side carries every field default and coerces enums via model_dump(), while the DB side only holds the keys originally stored, so the two shapes never compared equal and the guardrail was rebuilt on every poll cycle. Each rebuild created a fresh instance, but delete_in_memory_guardrail only removed the old callback from litellm.callbacks. Request handling promotes guardrail callbacks into the success/failure/async lists, so the previous instance stayed referenced there and instances accumulated. Normalize both sides through LitellmParams(...).model_dump() before diffing, and purge the callback from every callback list on delete. * refactor(guardrails): narrow params-normalization fallback to ValidationError The comparison normalizer caught a bare Exception and silently fell back to the raw dict, which hid the cause and quietly degraded the affected guardrail back to re-initializing on every poll. Catch only the ValidationError that LitellmParams construction can raise, log a warning so the offending row is diagnosable, and let any other error surface instead of being swallowed. * refactor(callbacks): add remove_callback_from_all_lists helper to manager Move the knowledge of which callback lists a callback can be promoted into out of the guardrail registry and into LoggingCallbackManager, where the rest of the callback-list bookkeeping already lives. delete_in_memory_guardrail now delegates to the new helper instead of iterating the lists itself. * chore(oss): litellm oss staging 150626 (#30463) * fix(pricing): add GitHub Copilot MAI Code Flash pricing (#30415) * fix(pricing): add GitHub Copilot MAI Code Flash pricing Add GitHub Copilot pricing entries for MAI-Code-1-Flash and the internal Copilot CLI model name so cost calculation can price input, cached input, and output tokens. Co-authored-by: Copilot <[email protected]> * test(pricing): cover GitHub Copilot MAI Code Flash pricing Add regression coverage for both GitHub Copilot MAI-Code-1-Flash model names, including cached input pricing, chat endpoint metadata, and cost_per_token arithmetic. Co-authored-by: Copilot <[email protected]> --------- Co-authored-by: Copilot <[email protected]> * fix(router/proxy): propagate completed_response through FallbackResponsesStreamWrapper for streaming /v1/responses container ownership (#30210) (#30213) * fix(router/proxy): propagate completed_response through FallbackResponsesStreamWrapper for streaming /v1/responses container ownership (#30210) #28990 added ownership recording for streaming /v1/responses via _wrap_responses_stream_for_container_ownership, which reads `getattr(stream_response, 'completed_response', None)` to extract the ResponsesAPIResponse. The unit test bypassed the Router, so it never exercised the production wrapping path. Through the Router (every proxy deployment), the stream is wrapped by FallbackResponsesStreamWrapper (router.py:2527). Its __init__ set `self.completed_response = None` and __anext__ only forwarded chunks — the inner source iterator's terminal event never bubbled up to the attribute the ownership hook reads, so the hook silently recorded nothing and every follow-up /v1/containers/<id>/files call returned 403 for non-admin keys. This commit: - router.py: pre-resolves the responses-API terminal event tuple (response.completed / .incomplete / .failed) once per _aresponses_streaming_iterator call, and has the wrapper's __anext__ sniff each forwarded chunk's .type. First terminal event hit gets stored on the wrapper's completed_response. Iterator-agnostic — works for source_iterator AND any future wrapper. - common_request_processing.py: when _extract_completed_responses_response returns None we now warn instead of silently skipping. Reporter on #30210 lost a day to this exact silent skip; the warning surfaces future regressions of the same shape directly in operator logs. Fixes #30210 * fix(router): type-ignore wrapper getattr-defaults; broaden ownership-skip warning CI lint (mypy) flagged the three pre-existing getattr(..., None) assignments in FallbackResponsesStreamWrapper.__init__: router.py:2564 self.response = getattr(source_iterator, 'response', None) router.py:2565 self.model = getattr(source_iterator, 'model', None) router.py:2566 self.logging_obj = getattr(..., None) Those lines also exist on litellm_internal_staging and pass mypy there. Adding the typed terminal-event tuple above the class made the function body more narrowable, which surfaced the pre-existing mismatch — base class declares non-Optional types but the bridge path (LiteLLMCompletionStreamingIterator) legitimately omits these. Keep the None fallback and silence with type: ignore[assignment]. Greptile 4/5 note: the ownership-skip warning hard-named code_interpreter which misleads operators when a non-code_interpreter stream aborts. Generalize to 'any tool container (e.g. code_interpreter)'. * fix(register_model): drop synthesized zero costs to preserve sparse entries (#30198) (#30201) * fix(register_model): drop synthesized zero costs to preserve sparse entries (#30198) get_model_info synthesizes input_cost_per_token / output_cost_per_token = 0 when they are absent from the raw entry (the price-unknown and free cases share the same representation). register_model then merges that result back into litellm.model_cost, which flips a sparse entry from 'no cost keys' (priced via model name) to 'cost keys = 0' (free). That defeats _is_cost_explicitly_configured (#24949) on re-registration: _is_model_cost_zero returns True, common_checks skips every tag / key / team / user / org budget check for the group, and over-budget traffic keeps returning 200. Spend keeps recording because cost calc still resolves by model name, so the symptom is silent and only triggers on the second register_model pass (router rebuild, /model/update, config sync). Mirror the existing litellm_provider-None guard one block above and pop the cost fields from the synthesized result when they are absent from the raw entry and not in the caller's value. Caller-provided zeros (genuinely free models, BYOK overrides) are preserved. Fixes #30198 * fix(register_model): switch _raw_entry to is-None checks + drop dead test assertion Greptile #30201 review notes: - the `or`-chain in the raw-entry lookup treated an empty dict (a key with no fields) as falsy and fell through to the second arm — replace with explicit `is None` checks so a present-but-empty entry is still taken at face value. - the first assertion in `test_router_double_init_keeps_db_model_entry_sparse` used `in (None, 0)` which passes under the bug condition (cost = 0 matches the tuple); the strong follow-up assertion already covers every shape, so drop the dead branch. * fix(bedrock mantle): use unique function-call id for responses->chat tool calls (#30426) * fix(bedrock mantle): use unique function-call id for responses->chat tool calls ... * fix(bedrock mantle): scope unique tool-call id fallback to degenerate call_id The previous revision preferred the Responses item id for every tool call, which broke providers (and existing tests) where call_id is a unique, canonical correlation key. Restrict the fallback to the degenerate index-based call_id that Bedrock Mantle returns (call_0, call_1, ... resetting per response) and keep call_id otherwise. Revert the change to the OUTPUT_ITEM_DONE streaming handler, whose tool_call_chunk is never emitted (dead code, per review). Extend the regression tests to assert a normal call_id is preserved. * fix(router): preserve azure_ad_token through CredentialLiteLLMParams for /v1/files + batches (#30235) (#30241) * fix(router): preserve azure_ad_token through CredentialLiteLLMParams for /v1/files + batches (#30235) Router.get_deployment_credentials_with_provider re-validates a deployment's litellm_params through CredentialLiteLLMParams before handing them to file/batch/passthrough callers: return CredentialLiteLLMParams( **deployment.litellm_params.model_dump(exclude_none=True) ).model_dump(exclude_none=True) Any field NOT declared on CredentialLiteLLMParams gets silently dropped on the way through. azure_ad_token was undeclared, so Azure deployments using OAuth/M2M (azure_ad_token instead of a static api_key) silently lost their token at the files endpoint and the proxy returned: Missing credentials. Please pass one of api_key, azure_ad_token, azure_ad_token_provider, ... Declare azure_ad_token on CredentialLiteLLMParams alongside api_key / api_base / api_version so it rides through the round-trip. Static-key deployments stay unaffected (Optional, default None, dropped by exclude_none=True). Provider-callable (azure_ad_token_provider) is a separate concern and out of scope here. Fixes #30235 * fix(ui-types): regenerate schema.d.ts for new azure_ad_token field CI's 'Verify schema.d.ts matches the proxy OpenAPI spec' check auto-detected the new field and emitted the exact diff to apply. Two schemas had `aws_secret_access_key` from CredentialLiteLLMParams, both get the new azure_ad_token marker next to it. * fix(proxy): org_admin with own user_id now sees all org teams on /v2/team/list (#30247) When the UI sends the callers own user_id (as it does for non-Admin global roles), _enforce_list_team_v2_access now nulls it out for org admins so _build_team_list_where_conditions scopes by organization_id only -- matching the legacy /team/list behavior and the documented intent. Fixes #30215 * test(vertex_ai): multi-region regression coverage for cachedContents host (#29571) (#29707) litellm_internal_staging already routes the cachedContents URL through get_vertex_base_url, fixing the multi-region 404 reported in #29571 — but carries no test coverage for the actual regression scenario (eu/us must resolve to the REP host aiplatform.{geo}.rep.googleapis.com). Add TestContextCachingMultiRegionUrls: parametrized eu/us REP-host assertions (including absence of the old broken {geo}-aiplatform host), plus regional (us-central1) and global no-regression checks. * fix(proxy): close upstream LLM stream when client disconnects mid-stream (#30245) * fix(proxy): close upstream LLM stream when client disconnects mid-stream When a streaming client disconnects, Starlette abandons the response body iterator without calling aclose(), so the proxy's connection to the upstream backend stays open until garbage collection, which may never come. The backend (e.g. vLLM) keeps generating into a dead pipe: small responses drain invisibly into TCP buffers while large ones block the backend on a full send buffer indefinitely (observed via lsof as an ESTABLISHED proxy->backend connection minutes after the client left) create_response now returns a StreamingResponse subclass that closes both its body iterator and the wrapped upstream-facing generator in a shielded finally. The upstream generator is closed directly rather than through a cascade because aclose() on a never-started generator skips its body, which would make the cascade a no-op when the client disconnects before the first chunk is sent. async_streaming_data_generator also gains the same shielded finally-aclose that async_data_generator in proxy_server.py already had, covering the Anthropic and Google SSE paths With this, killing a streaming client causes the backend to observe the abort within about a second and free its slot, while completed streams are unaffected. No flag is needed, unlike the non-streaming opt-in cancel in #30223: this only releases resources after the client is already gone and does not change any response a client can observe Fixes #30244 * fix(proxy): close upstream even when body iterator aclose raises BaseException Addresses the Greptile finding on #30245: the cleanup loop caught only Exception while the generator-level cleanup catches BaseException, so a CancelledError or GeneratorExit escaping body_iterator.aclose() would skip closing the upstream generator. Both sites now use the same scope and a regression test pins that the upstream is closed even when the body iterator explodes with a BaseException * fix(llms): expose aclose on BaseModelResponseIterator so stream close reaches the provider connection The response-level close added for #30244 only worked for SDK-based providers (e.g. openai), whose streams expose aclose all the way down. Providers served by base_llm_http_handler (hosted_vllm and most modern transformation-based providers) wrap a bare response.aiter_lines() generator in BaseModelResponseIterator, which had no aclose or close at all, and nothing retained the httpx response object; so CustomStreamWrapper.aclose() silently did nothing and the upstream connection stayed open. Verified with a vLLM-style mock: with hosted_vllm/ the backend streamed all 100 chunks to completion after the client disconnected, while openai/ aborted at chunk 6 BaseModelResponseIterator now carries an optional http_response and an aclose() that closes it; make_async_call_stream_helper attaches the response after building the iterator. With this, hosted_vllm aborts the backend within ~1.6s of the client dropping, and completed streams are unaffected --------- Co-authored-by: kursad <[email protected]> * feat(anthropic): surface compaction usage iterations data (#27065) * feat(anthropic): surface compaction usage iterations data * style: apply black formatting to fix lint checks * fix(usage): correct calculate usage with cached tokens when use ChatCompletionUsageBlock (#30422) * fix(usage): correct calculate usage with cached tokens when use ChatCompletionUsageBlock * fix(usage): optimize test imports * feat: add fastCRW search provider (#30434) * feat(provider): add LibertAI as a JSON-configured OpenAI-compatible provider (#30203) * feat(provider): add LibertAI as a JSON-configured OpenAI-compatible provider * libertai: update served endpoints backup + add mode/matrix tests Addresses review feedback: - Add libertai to litellm/provider_endpoints_support_backup.json, the file actually served by GET /public/supported_endpoints (the root provider_endpoints_support.json already had it). - Add tests asserting bge-m3 normalizes to mode='embedding' and that the served matrix lists libertai. embeddings stays false: the JSON-configured provider path only wires chat routing (OpenAILike embedding handler is reached only for literal openai_like/llamafile/lm_studio), matching the llamagate precedent; bge-m3 remains in the cost map for metadata. --------- Co-authored-by: Moshe Malawach <[email protected]> * feat(provider): add ModelScope as an OpenAI-compatible provider (#28460) * add ModelScope API support * add modelscope api support * update modelscope model list * add image-genetation support * update test and multimodal * fix: address PR review feedback for modelscope provider * update README * fix(customer_endpoints): restrict /customer/daily/activity to admin-only (#28849) * fix(customer_endpoints): restrict /customer/daily/activity to admin-only * fix(customer_endpoints): check role before prisma_client guard * fix(custom_guardrail): key disable_global_guardrails takes precedence over team guardrail list (#28563) * fix(fallbacks): preserve fallback model in SDK fallback responses (#28260) * fix(fallbacks): preserve fallback model in response when using SDK-level fallbacks * fix(fallbacks): gate x-litellm-* passthrough to trusted callers only The previous patch unconditionally let `x-litellm-*` keys bypass the `llm_provider-` prefix in `process_response_headers`. That function is also called on raw upstream-provider response headers (e.g. from `llm_http_handler.py`), so a malicious provider could return `x-litellm-attempted-fallbacks` and spoof a LiteLLM-internal marker, bypassing the proxy model-override guard. Add a `preserve_litellm_internal_headers` flag (default False). Only `response_metadata.py`, which re-processes the already-built `_hidden_params["additional_headers"]` dict (LiteLLM-owned), passes True. Raw provider header callsites keep the default False, so upstream `x-litellm-*` still gets the `llm_provider-` prefix. Adds a regression test for the spoofing case and renames the existing preserve test to make the trusted-path semantics explicit. * fix(fallbacks): ignore preserve_litellm_internal_headers for raw httpx.Headers inputs * style(core_helpers): apply black formatting * fix(lint): remove banned typing.List/Dict/Any imports and suppress PLR0913 on interface overrides * fix(lint): apply black formatting to modelscope chat transformation * fix(lint): replace noqa with proper fixes — use **kwargs and Awaitable instead of Any/List * fix(lint): remove unused AllMessageValues import * revert: restore base_model_iterator.py to original PR state * fix(lint): restore full method signatures for MyPy compatibility; bump PLR0913 budget for new provider files * fix(lint): use @override to suppress PLR0913 on inherited signatures instead of bumping budget The overrides keep their full base-class signatures for MyPy compatibility, but those signatures carry more than five parameters, which tripped PLR0913 on each subclass redeclaration. Since the arity is dictated by the base class and cannot be reduced, decorate the overrides with typing_extensions.override; ruff treats that as the intended signal that the parameter count is not under the author's control and skips PLR0913. This restores the PLR0913 baseline to 1813. * fix(lint): add @override to modelscope image generation overrides Apply the same typing_extensions.override treatment to the image generation config so its inherited-signature overrides do not count against PLR0913. --------- Co-authored-by: Joel Tony <[email protected]> Co-authored-by: Copilot <[email protected]> Co-authored-by: hcl <[email protected]> Co-authored-by: ztko <[email protected]> Co-authored-by: Nahrin <[email protected]> Co-authored-by: Humphrey <[email protected]> Co-authored-by: kursadlacin <[email protected]> Co-authored-by: kursad <[email protected]> Co-authored-by: Dushyant Acharya <[email protected]> Co-authored-by: Yuriy <[email protected]> Co-authored-by: Recep S <[email protected]> Co-authored-by: Moshe Malawach <[email protected]> Co-authored-by: Moshe Malawach <[email protected]> Co-authored-by: Rongkun Yan <[email protected]> Co-authored-by: Varshith <[email protected]> Co-authored-by: Mateo Wang <[email protected]> * ci(lint): add blanket-noqa, dataclass-default, and unused-noqa Ruff rules (#30516) * ci(lint): enforce blanket-noqa, dataclass-default, and unused-noqa rules Enable PGH004 (blanket-noqa), RUF008 (mutable-dataclass-default), RUF009 (function-call-in-dataclass-default-argument), and RUF100 (unused-noqa) in ruff.toml, and clean up every resulting violation. RUF008/RUF009 were already clean. PGH004/RUF100 surfaced ~335 stale or blanket noqas: blanket `# noqa` are now scoped to the rule they actually suppress (mostly T201), dead directives are removed, and inapplicable codes are trimmed (e.g. F401 dropped from `import *`). lint.external lists rules enforced outside this config (the strict-rule gate via ruff-strict.toml and upstream litellm's own ruff config) so RUF100 keeps the noqa directives that protect them instead of stripping coverage this config can't see. * ci(lint): trim RUF100 external list to load-bearing codes only Drop the 9 precautionary strict-gate codes (ANN001/002/003/401, B006, PLR0913, PLW0603, RUF012, TID251) that have zero `# noqa` references in the gated source. Keep only the 11 codes with live suppressions so RUF100 doesn't flag them as unused. Future strict-gate suppressions can re-add codes here (or fix the underlying issue) as needed. * ci: ratchet lint and type-check gates (ruff preview, ANN, mypy, basedpyright) (#30379) * ci: enable ruff preview rules under the budgeted strict gate Turn on ruff preview in the strict-budget lane (ruff-strict.toml) only, leaving the clean gate (ruff.toml) untouched so make lint-ruff stays at zero. Enumerate the 118 firing codes explicitly with explicit-preview-rules so the gate is deterministic and stable across ruff upgrades rather than depending on preview auto-selecting the broad catalog. Grandfather the existing 58438 violations into ruff-strict-budget.json as per-rule baselines with headroom, so only net-new violations fail CI. The existing ten rules keep their hand-tuned slack; the new rules get slack 10 when the baseline is 50 or more and 3 otherwise. * ci: add ANN return-type rules to the budgeted strict gate Add ANN201/202/204/205/206 (missing return annotations) to the strict lane and grandfather the existing counts into ruff-strict-budget.json so the codebase ratchets toward explicit return types without breaking CI. * ci: add mypy (disallow_untyped_defs) and basedpyright strict gates with baselines Add two type-check gates, each grandfathering the current tree so only net-new violations fail CI, matching the ruff strict-budget ratchet. mypy gains disallow_untyped_defs in litellm/mypy.ini (the config the CI invocation actually reads; the root [tool.mypy] is not picked up from the litellm/ working dir). The 4885 existing missing-annotation errors are captured in litellm/.mypy-baseline.txt and the run is piped through mypy-baseline filter so new untyped defs are rejected. basedpyright runs in strict mode over litellm/, with enableTypeIgnoreComments disabled so it only honors '# pyright: ignore' and never polices mypy's '# type: ignore'. The existing strict diagnostics are grandfathered into .basedpyright/baseline.json. Both tools are pinned in the dev group and uv.lock; the lint workflow and Makefile run them filtered through their baselines, with lint-mypy-baseline-update and lint-basedpyright-baseline-update to ratchet. * ci: raise lint job timeout to 15m for the basedpyright strict pass * ci: pin pythonVersion 3.12 and regenerate baselines against merged base Merge litellm_internal_staging so the baselines cover code the CI merge includes (e.g. the cisco_ai_defense guardrail), which otherwise tripped the mypy gate with 3 ungrandfathered no-untyped-def errors. Pin pythonVersion 3.12 in pyrightconfig so basedpyright's strict analysis is reproducible across interpreter versions (CI runs 3.12). * ci: regenerate basedpyright baseline against the frozen lint env The previous baseline was generated with optional provider deps (azure, google, anthropic, mcp, numpydoc, google-genai) installed locally, so CI's dev-only env surfaced ~3500 reportUnknown*/reportMissingTypeStubs errors not in the baseline. Regenerate after uv sync --frozen so the baseline reflects the same dependency set the lint job sees. * ci: regenerate basedpyright baseline on python 3.12 frozen env The prior baseline still carried proxy-dev packages (e.g. prisma) that the lint job's dev-only, python 3.12 env lacks, leaving 2 unresolved-import errors ungrandfathered. Regenerate in a python 3.12 venv synced to the frozen lock with default groups only, so the baseline matches exactly what CI sees. * ci: replace type-check baselines with per-file count budgets The mypy and basedpyright baselines were position-sensitive (and the basedpyright one was a 27MB file), so ordinary line shifts churned them. Replace both with a per-file count gate: scripts/type_check_gate.py reduces each tool's output to errors-per-file and checks it against a committed {file: max} budget, ignoring line and column numbers. A file fails only when it gains more errors than its ceiling; debt can't be shuffled between files because each file has its own cap and new files default to zero. Budgets (mypy-file-budget.json 48K, basedpyright-file-budget.json 96K) are generated in the python 3.12 frozen lint env so they match CI. Drops the mypy-baseline dependency; basedpyright runs without its native baseline. ratchet via make lint-mypy-budget-update / lint-basedpyright-budget-update. * ci: add a small per-file slack to the type-check gate Allow each file to drift PER_FILE_SLACK (5) errors past its recorded count before failing, so a basedpyright inference ripple in an unrelated file doesn't break the build over a couple of errors. Budgets still record exact counts; the tolerance is applied at check time. * ci: move type-check slack into the budget json and trim lint timeout Make slack declarative: the budget is now {"slack": N, "files": {path: count}} so the tolerance is tuned in JSON without editing the script, mirroring how ruff-strict-budget.json carries its slack. --update preserves the existing slack. Also drop the lint job timeout from 15m to 10m; the mypy and basedpyright passes add ~2m, leaving the job around 4-5m, so 10m is a comfortable margin. * ci: collapse fully-adopted ruff categories and drop inert preview flag ANN (all nine non-removed rules) and BLE (its only rule) were spelled out code-by-code; replace each with its category selector, which is exactly equivalent in 0.15.3 (the removed ANN101/ANN102 are skipped by a category selector and error when named explicitly). explicit-preview-rules was inert: every selected rule is stable and nothing is selected by category, so the flag had nothing to gate. Verified the strict-rule counts are identical before and after (62379 each, zero per-rule drift), so no budget change. * ci: drop redundant pyright dev dependency Nothing invokes bare pyright in the Makefile, the linting workflow, or scripts; the basedpyright gate added on this branch is the only type checker that runs. basedpyright is a superset fork that reads the same pyrightconfig.json and honors the same "# pyright: ignore" comments, so pyright==1.1.408 in the ci group was dead weight. Regenerated uv.lock under the same exclude-newer cutoff so the only change is removing pyright and its package stanza * ci: un-weaken mypy and error on Any in basedpyright mypy: enable warn_return_any, drop the valid-type silencer, and stop globally ignoring missing first-party imports via [mypy-litellm.*] ignore_missing_imports = False, which surfaced eight real broken litellm.* imports the blanket ignore was hiding; third-party imports stay ignored. The per-file budget moves 4888 -> 5799 (902 no-any-return, 1 valid-type, 8 import-not-found), all grandfathered so only net-new errors fail and the ceilings ratchet down basedpyright: error on reportExplicitAny and reportAny. The per…

This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | [ghcr.io/berriai/litellm](https://images.chainguard.dev/directory/image/wolfi-base/overview) ([source](https://github.com/BerriAI/litellm)) | final | minor | `v1.85.1` → `v1.90.0` | --- ### Release Notes <details> <summary>BerriAI/litellm (ghcr.io/berriai/litellm)</summary> ### [`v1.90.0`](https://github.com/BerriAI/litellm/releases/tag/v1.90.0) [Compare Source](https://github.com/BerriAI/litellm/compare/v1.90.0...v1.90.0) ##### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](https://github.com/BerriAI/litellm/commit/0112e53046018d726492c814b3644b7d376029d0). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.90.0 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.90.0/cosign.pub \ ghcr.io/berriai/litellm:v1.90.0 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** ##### What's Changed - fix(responses-bridge): map system-only chat request to system input item by [@milan-berri](https://github.com/milan-berri) in [#29817](https://github.com/BerriAI/litellm/pull/29817) - feat(bedrock): forward strict and additionalProperties to Converse toolSpec by [@mateo-berri](https://github.com/mateo-berri) in [#29814](https://github.com/BerriAI/litellm/pull/29814) - fix(mcp): highlight MCP cards red when the logged-in user is missing per-user env vars by [@mateo-berri](https://github.com/mateo-berri) in [#29856](https://github.com/BerriAI/litellm/pull/29856) - feat(ui): add budget duration to edit team member form by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#29717](https://github.com/BerriAI/litellm/pull/29717) - fix(ui): make workflow runs page fill full width by [@yuneng-berri](https://github.com/yuneng-berri) in [#29868](https://github.com/BerriAI/litellm/pull/29868) - feat: standardize rate limit errors with category, rate\_limit\_type, model, and llm\_provider fields by [@mateo-berri](https://github.com/mateo-berri) in [#27687](https://github.com/BerriAI/litellm/pull/27687) - fix(ui): default guardrails page to the Guardrails tab by [@yuneng-berri](https://github.com/yuneng-berri) in [#29872](https://github.com/BerriAI/litellm/pull/29872) - docs(readme): add Deploy on AWS/GCP Terraform section and fix deploy button rendering by [@mateo-berri](https://github.com/mateo-berri) in [#29879](https://github.com/BerriAI/litellm/pull/29879) - refactor(bedrock): build Converse toolSpec via a BedrockToolSpec dict subclass by [@mateo-berri](https://github.com/mateo-berri) in [#29869](https://github.com/BerriAI/litellm/pull/29869) - feat(litellm): add models and repository layers by [@yassin-berriai](https://github.com/yassin-berriai) in [#29686](https://github.com/BerriAI/litellm/pull/29686) - feat(ui): include internal routes in the dashboard's generated OpenAPI types by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#29885](https://github.com/BerriAI/litellm/pull/29885) - feat(proxy): publish /v2/model/info in Swagger OpenAPI spec by [@Sameerlite](https://github.com/Sameerlite) in [#29900](https://github.com/BerriAI/litellm/pull/29900) - refactor(ui): single source of truth for migrated-page routing by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#29949](https://github.com/BerriAI/litellm/pull/29949) - fix(ui/model-hub): render provider icons on the public model hub by [@yuneng-berri](https://github.com/yuneng-berri) in [#29958](https://github.com/BerriAI/litellm/pull/29958) - fix(ui): keep create guardrail modal open on outside click by [@yuneng-berri](https://github.com/yuneng-berri) in [#29871](https://github.com/BerriAI/litellm/pull/29871) - fix(ui): label default key type as "Full Access" on key edit page by [@yuneng-berri](https://github.com/yuneng-berri) in [#29870](https://github.com/BerriAI/litellm/pull/29870) - fix(ui): unify migrated-route URLs and migrate the API Reference page by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#29953](https://github.com/BerriAI/litellm/pull/29953) - fix(mcp): let non-creator users OAuth into OBO-mode MCP servers from the Tools page by [@tin-berri](https://github.com/tin-berri) in [#29867](https://github.com/BerriAI/litellm/pull/29867) - Litellm oss staging 080626 by [@Sameerlite](https://github.com/Sameerlite) in [#29932](https://github.com/BerriAI/litellm/pull/29932) - feat(galileo): add health check support for UI callback test by [@Sameerlite](https://github.com/Sameerlite) in [#29908](https://github.com/BerriAI/litellm/pull/29908) - fix(model-management): allow deleting a BYOK model after its team is deleted by [@yuneng-berri](https://github.com/yuneng-berri) in [#29875](https://github.com/BerriAI/litellm/pull/29875) - feat(jwt-auth): opt-in fallback to DB team on unresolved JWT claim by [@milan-berri](https://github.com/milan-berri) in [#28913](https://github.com/BerriAI/litellm/pull/28913) - fix(team\_endpoints): don't block /team/update on unchanged team budget by [@milan-berri](https://github.com/milan-berri) in [#29525](https://github.com/BerriAI/litellm/pull/29525) - fix(fireworks): enable tool calling for glm-5p1 in model cost map by [@milan-berri](https://github.com/milan-berri) in [#29697](https://github.com/BerriAI/litellm/pull/29697) - fix(vertex): propagate Vertex AI metadata in streaming success callbacks by [@Sameerlite](https://github.com/Sameerlite) in [#29899](https://github.com/BerriAI/litellm/pull/29899) - fix(ui): show team projects to internal users on key creation by [@milan-berri](https://github.com/milan-berri) in [#28855](https://github.com/BerriAI/litellm/pull/28855) - build(deps): bump pyjwt to 2.13.0 and ws override to 8.20.1 by [@yuneng-berri](https://github.com/yuneng-berri) in [#29982](https://github.com/BerriAI/litellm/pull/29982) - fix(team-management): delete a team's BYOK models when the team is deleted by [@yuneng-berri](https://github.com/yuneng-berri) in [#29977](https://github.com/BerriAI/litellm/pull/29977) - feat(vantage): include organization metadata in FOCUS Tags export by [@milan-berri](https://github.com/milan-berri) in [#28184](https://github.com/BerriAI/litellm/pull/28184) - fix(guardrails): read CrowdStrike AIDR identity from both metadata bags by [@yuneng-berri](https://github.com/yuneng-berri) in [#29991](https://github.com/BerriAI/litellm/pull/29991) - fix(mcp): mirror upstream token lifetime instead of forcing a 1h OBO expiry by [@tin-berri](https://github.com/tin-berri) in [#29951](https://github.com/BerriAI/litellm/pull/29951) - feat(azure\_ai): add MAI-Image-2.5 image generation support by [@Sameerlite](https://github.com/Sameerlite) in [#29688](https://github.com/BerriAI/litellm/pull/29688) - fix(mcp): load MCP tool configuration tools via the OBO/passthrough-aware GET path by [@tin-berri](https://github.com/tin-berri) in [#29960](https://github.com/BerriAI/litellm/pull/29960) - fix(team): reserve team budget raises for proxy admins on /team/update by [@milan-berri](https://github.com/milan-berri) in [#30030](https://github.com/BerriAI/litellm/pull/30030) - test(ui): data-driven App Router migration E2E smoke (default + server-root-path) by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#29974](https://github.com/BerriAI/litellm/pull/29974) - fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through by [@michelligabriele](https://github.com/michelligabriele) in [#24232](https://github.com/BerriAI/litellm/pull/24232) - chore(ui): remove dead App Router route stubs under (dashboard) by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30045](https://github.com/BerriAI/litellm/pull/30045) - fix(ui/mcp): reset OAuth state on create-server modal close so a prior server's token no longer leaks into the next add-server session by [@tin-berri](https://github.com/tin-berri) in [#30000](https://github.com/BerriAI/litellm/pull/30000) - fix(mcp): allow team access-group grants in OAuth authorize/token access check by [@tin-berri](https://github.com/tin-berri) in [#30041](https://github.com/BerriAI/litellm/pull/30041) - docs(security): require a reproduction video for vulnerability reports by [@yuneng-berri](https://github.com/yuneng-berri) in [#30063](https://github.com/BerriAI/litellm/pull/30063) - feat(ui): add admin flag to disable in-product UI nudges for everyone by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#29796](https://github.com/BerriAI/litellm/pull/29796) - chore(ui): remove dead dashboard files and unused dependencies by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30047](https://github.com/BerriAI/litellm/pull/30047) - fix(proxy): authorize batch files using upload target\_model\_names (LIT-3593) by [@Sameerlite](https://github.com/Sameerlite) in [#30009](https://github.com/BerriAI/litellm/pull/30009) - Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI by [@mateo-berri](https://github.com/mateo-berri) in [#30064](https://github.com/BerriAI/litellm/pull/30064) - Add Claude Fable 5 cost map entries (data-only hotfix for the hosted map) by [@mateo-berri](https://github.com/mateo-berri) in [#30076](https://github.com/BerriAI/litellm/pull/30076) - fix(caching): restore stored prompt\_tokens on embedding cache hits instead of recomputing by [@michelligabriele](https://github.com/michelligabriele) in [#30046](https://github.com/BerriAI/litellm/pull/30046) - Litellm oss 090626 by [@Sameerlite](https://github.com/Sameerlite) in [#30021](https://github.com/BerriAI/litellm/pull/30021) - fix(proxy): self-heal startup/reload prisma reads on engine disconnect by [@michelligabriele](https://github.com/michelligabriele) in [#28803](https://github.com/BerriAI/litellm/pull/28803) - chore(ui): make knip recognize .mjs scripts and openapi-typescript by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30052](https://github.com/BerriAI/litellm/pull/30052) - fix(register\_model): preserve built-in cache pricing when registering custom overrides under unmapped keys by [@yassin-berriai](https://github.com/yassin-berriai) in [#30044](https://github.com/BerriAI/litellm/pull/30044) - \[internal copy of [#28007](https://github.com/BerriAI/litellm/issues/28007)] Fix/gcp model garden streaming by [@mateo-berri](https://github.com/mateo-berri) in [#28363](https://github.com/BerriAI/litellm/pull/28363) - feat(cli): per-agent `lite claude` / `codex` / `opencode` commands that wrap coding agents through the proxy by [@mateo-berri](https://github.com/mateo-berri) in [#29850](https://github.com/BerriAI/litellm/pull/29850) - fix(callbacks): forward callback\_settings to callback initializers and guard consumers against non-dict values by [@yuneng-berri](https://github.com/yuneng-berri) in [#30161](https://github.com/BerriAI/litellm/pull/30161) - fix(mcp): drop orphaned per-user credential rows when an MCP server is deleted by [@tin-berri](https://github.com/tin-berri) in [#30141](https://github.com/BerriAI/litellm/pull/30141) - fix(proxy): recover from cached-plan errors by reconnecting the Prisma client by [@yassin-berriai](https://github.com/yassin-berriai) in [#29983](https://github.com/BerriAI/litellm/pull/29983) - feat(proxy): add option to disable server-side prepared statements for DB lookups by [@yassin-berriai](https://github.com/yassin-berriai) in [#29984](https://github.com/BerriAI/litellm/pull/29984) - fix(release): stop backport releases from overwriting the latest badge by [@yuneng-berri](https://github.com/yuneng-berri) in [#30005](https://github.com/BerriAI/litellm/pull/30005) - feat: add conventional commits and coding guidelines by [@mateo-berri](https://github.com/mateo-berri) in [#30159](https://github.com/BerriAI/litellm/pull/30159) - fix(proxy): return 5xx on DB infra errors during auth; reserve 401 for genuine auth failures by [@yassin-berriai](https://github.com/yassin-berriai) in [#29986](https://github.com/BerriAI/litellm/pull/29986) - fix(ui): dev server 404s on migrated-page links because uiBase hardcodes /ui by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30169](https://github.com/BerriAI/litellm/pull/30169) - refactor(ui): consolidate dashboard to one shell in the (dashboard) layout by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30166](https://github.com/BerriAI/litellm/pull/30166) - fix(proxy): align /v1/model/info with router deployments by [@Sameerlite](https://github.com/Sameerlite) in [#30025](https://github.com/BerriAI/litellm/pull/30025) - fix: completion\_cost AttributeError on streaming Anthropic web\_search responses ([#26153](https://github.com/BerriAI/litellm/issues/26153)) by [@ishaan-berri](https://github.com/ishaan-berri) in [#27346](https://github.com/BerriAI/litellm/pull/27346) - \[internal copy of [#30137](https://github.com/BerriAI/litellm/issues/30137)] perf(realtime): eliminate redundant per-frame JSON work on OpenAI realtime relay by [@mateo-berri](https://github.com/mateo-berri) in [#30142](https://github.com/BerriAI/litellm/pull/30142) - feat(bedrock): aws\_bedrock\_project\_id for bedrock-mantle project / workspace association by [@mateo-berri](https://github.com/mateo-berri) in [#30163](https://github.com/BerriAI/litellm/pull/30163) - chore(hooks): enforce Conventional Commits and Conventional Branches by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30174](https://github.com/BerriAI/litellm/pull/30174) - feat(rate-limiter): allow opting out of v3 TPM reservation and Redis circuit breaker by [@yassin-berriai](https://github.com/yassin-berriai) in [#30211](https://github.com/BerriAI/litellm/pull/30211) - feat(spend\_logs): opt-in native Postgres partitioning for SpendLogs retention by [@yassin-berriai](https://github.com/yassin-berriai) in [#29466](https://github.com/BerriAI/litellm/pull/29466) - feat(ui): migrate playground to path routing and colocate its files by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30185](https://github.com/BerriAI/litellm/pull/30185) - feat(ui): migrate projects and access-groups to path routes by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30226](https://github.com/BerriAI/litellm/pull/30226) - fix(proxy): coalesce NULL rollup metrics in aggregated daily-activity by [@michelligabriele](https://github.com/michelligabriele) in [#30151](https://github.com/BerriAI/litellm/pull/30151) - fix(anthropic\_passthrough): resolve costing model from message\_start chunk, litellm\_params and model\_group instead of 'unknown' by [@yassin-berriai](https://github.com/yassin-berriai) in [#30160](https://github.com/BerriAI/litellm/pull/30160) - feat(ui): migrate budgets, workflows, and guardrails-monitor to path routes by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30236](https://github.com/BerriAI/litellm/pull/30236) - feat(ui): migrate mcp-servers, search-tools, tag-management, vector-stores, and memory to path routes by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30261](https://github.com/BerriAI/litellm/pull/30261) - fix(a2a): forward agent\_extra\_headers through completion bridge by [@mateo-berri](https://github.com/mateo-berri) in [#28277](https://github.com/BerriAI/litellm/pull/28277) - fix(gemini-live): forward audio buffer commit and correct Vertex PCM rate by [@Sameerlite](https://github.com/Sameerlite) in [#29946](https://github.com/BerriAI/litellm/pull/29946) - fix(proxy): skip double-wrapping unified batch output file ids on retrieve by [@Sameerlite](https://github.com/Sameerlite) in [#30011](https://github.com/BerriAI/litellm/pull/30011) - feat: litellm oss 110626 by [@Sameerlite](https://github.com/Sameerlite) in [#30202](https://github.com/BerriAI/litellm/pull/30202) - fix(docker): copy only runtime artifacts into the final image by [@yuneng-berri](https://github.com/yuneng-berri) in [#30243](https://github.com/BerriAI/litellm/pull/30243) - feat(proxy): enforce key/team guardrails on bedrock passthrough routes by [@Sameerlite](https://github.com/Sameerlite) in [#30194](https://github.com/BerriAI/litellm/pull/30194) - feat(gemini): forward web search tools in image generation by [@Sameerlite](https://github.com/Sameerlite) in [#30119](https://github.com/BerriAI/litellm/pull/30119) - fix: bedrock mantle fixes by [@Sameerlite](https://github.com/Sameerlite) in [#30083](https://github.com/BerriAI/litellm/pull/30083) - feat(proxy): add require\_managed\_files setting for file uploads by [@Sameerlite](https://github.com/Sameerlite) in [#30186](https://github.com/BerriAI/litellm/pull/30186) - fix(mcp): honor server\_id for REST tool calls with shared upstream URLs by [@Sameerlite](https://github.com/Sameerlite) in [#30184](https://github.com/BerriAI/litellm/pull/30184) - fix(responses): presidio PII masking for Azure WebSocket and streaming by [@Sameerlite](https://github.com/Sameerlite) in [#30003](https://github.com/BerriAI/litellm/pull/30003) - feat(passthrough): add configurable pass-through request timeouts by [@Sameerlite](https://github.com/Sameerlite) in [#30266](https://github.com/BerriAI/litellm/pull/30266) - fix(google\_genai): preserve complete SSE events in Vertex/Gemini image streaming by [@Sameerlite](https://github.com/Sameerlite) in [#30270](https://github.com/BerriAI/litellm/pull/30270) - fix(proxy): populate access\_via\_team\_ids on /v1/model/info by [@Sameerlite](https://github.com/Sameerlite) in [#30274](https://github.com/BerriAI/litellm/pull/30274) - chore(oss): litellm oss staging 120626 by [@Sameerlite](https://github.com/Sameerlite) in [#30292](https://github.com/BerriAI/litellm/pull/30292) - feat(ui): migrate policies, guardrails, prompts, tool-policies, and skills to path routes by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30263](https://github.com/BerriAI/litellm/pull/30263) - feat(ui): migrate caching, cost-tracking, transform-request, ui-theme, and logs to path routes by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30267](https://github.com/BerriAI/litellm/pull/30267) - fix(ui): gate dashboard layout on ui config load so deep links work under SERVER\_ROOT\_PATH by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30312](https://github.com/BerriAI/litellm/pull/30312) - feat(ui): migrate admin-panel, logging-and-alerts, model-hub-table, and usage to path routes by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30268](https://github.com/BerriAI/litellm/pull/30268) - fix(otel): cap metric attribute cardinality with include/exclude lists by [@yassin-berriai](https://github.com/yassin-berriai) in [#30257](https://github.com/BerriAI/litellm/pull/30257) - fix(proxy): grace-period key rotation 401s; return deprecated-key lookup result directly by [@yuneng-berri](https://github.com/yuneng-berri) in [#30327](https://github.com/BerriAI/litellm/pull/30327) - chore(deps): bump vitest, brace-expansion, pypdf and tornado by [@yuneng-berri](https://github.com/yuneng-berri) in [#30220](https://github.com/BerriAI/litellm/pull/30220) - refactor(ui): remove unreachable /chat page by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30178](https://github.com/BerriAI/litellm/pull/30178) - feat(ui): migrate agents and router-settings to path routes by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30323](https://github.com/BerriAI/litellm/pull/30323) - feat: strengthen coding conventions in CLAUDE.md by [@mateo-berri](https://github.com/mateo-berri) in [#30333](https://github.com/BerriAI/litellm/pull/30333) - feat(ui): cut the users page over to the /ui/users path route by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30334](https://github.com/BerriAI/litellm/pull/30334) - feat: ruff strict-rule suppressions baseline gate by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30303](https://github.com/BerriAI/litellm/pull/30303) - feat(guardrails): add Cisco AI Defense integration ([#28249](https://github.com/BerriAI/litellm/issues/28249)) by [@yassin-berriai](https://github.com/yassin-berriai) in [#30338](https://github.com/BerriAI/litellm/pull/30338) - chore(ui): remove dead UI components unreferenced by any page by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30340](https://github.com/BerriAI/litellm/pull/30340) - ci: add osv-scanner lockfile scan workflow by [@yuneng-berri](https://github.com/yuneng-berri) in [#30222](https://github.com/BerriAI/litellm/pull/30222) - fix(otel): record full error message on standard exception event in otel v2 by [@yassin-berriai](https://github.com/yassin-berriai) in [#30380](https://github.com/BerriAI/litellm/pull/30380) - test(fireworks): mock whisper transcription tests instead of live calls by [@yuneng-berri](https://github.com/yuneng-berri) in [#30391](https://github.com/BerriAI/litellm/pull/30391) - build(ui): pin esbuild to 0.28.1 via overrides by [@yuneng-berri](https://github.com/yuneng-berri) in [#30390](https://github.com/BerriAI/litellm/pull/30390) - feat(ui): cut the organizations page over to the /ui/organizations path route by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30336](https://github.com/BerriAI/litellm/pull/30336) - fix(proxy): support SMTP implicit SSL (port 465) by [@yuneng-berri](https://github.com/yuneng-berri) in [#30395](https://github.com/BerriAI/litellm/pull/30395) - fix(mcp): default Linear MCP registry entry to streamable HTTP by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30396](https://github.com/BerriAI/litellm/pull/30396) - fix(ui): stop Virtual Keys page from infinite render loop by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30397](https://github.com/BerriAI/litellm/pull/30397) - fix(streaming): guard raise\_on\_model\_repetition against empty choices by [@shivamrawat1](https://github.com/shivamrawat1) in [#30485](https://github.com/BerriAI/litellm/pull/30485) - feat(otel-v2): emit the 6 gen\_ai.client.\* metrics at parity with v1 by [@yassin-berriai](https://github.com/yassin-berriai) in [#30326](https://github.com/BerriAI/litellm/pull/30326) - fix(mcp): drop phantom 401 span on delegated OAuth2 tool calls by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30494](https://github.com/BerriAI/litellm/pull/30494) - feat(ui): cut the teams page over to the /ui/teams path route by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30343](https://github.com/BerriAI/litellm/pull/30343) - fix(integrations): cap Anthropic cache\_control injection at 4 blocks by [@shivamrawat1](https://github.com/shivamrawat1) in [#30480](https://github.com/BerriAI/litellm/pull/30480) - chore(codecov): add Batches, Videos, and Realtime components by [@Sameerlite](https://github.com/Sameerlite) in [#30517](https://github.com/BerriAI/litellm/pull/30517) - test(batches): move orphan tests into tests/test\_litellm for CI coverage by [@Sameerlite](https://github.com/Sameerlite) in [#30510](https://github.com/BerriAI/litellm/pull/30510) - fix(guardrails): run pre\_call hook once for model-level guardrails by [@yassin-berriai](https://github.com/yassin-berriai) in [#30543](https://github.com/BerriAI/litellm/pull/30543) - fix(guardrails): stop re-initializing DB guardrails on every poll by [@yassin-berriai](https://github.com/yassin-berriai) in [#30542](https://github.com/BerriAI/litellm/pull/30542) - chore(oss): litellm oss staging 150626 by [@Sameerlite](https://github.com/Sameerlite) in [#30463](https://github.com/BerriAI/litellm/pull/30463) - ci(lint): add blanket-noqa, dataclass-default, and unused-noqa Ruff rules by [@mateo-berri](https://github.com/mateo-berri) in [#30516](https://github.com/BerriAI/litellm/pull/30516) - ci: ratchet lint and type-check gates (ruff preview, ANN, mypy, basedpyright) by [@mateo-berri](https://github.com/mateo-berri) in [#30379](https://github.com/BerriAI/litellm/pull/30379) - fix(proxy): allow internal roles to access vector store CRUD routes by [@shivamrawat1](https://github.com/shivamrawat1) in [#30503](https://github.com/BerriAI/litellm/pull/30503) - fix(otel): stamp gen\_ai.input/output.messages on v2 spans by [@yassin-berriai](https://github.com/yassin-berriai) in [#30548](https://github.com/BerriAI/litellm/pull/30548) - fix(otel): export v2 gen\_ai client metrics to the configured meter provider by [@yassin-berriai](https://github.com/yassin-berriai) in [#30549](https://github.com/BerriAI/litellm/pull/30549) - fix(bedrock): preserve cache\_control for ARN models in /v1/messages adapter by [@mateo-berri](https://github.com/mateo-berri) in [#29823](https://github.com/BerriAI/litellm/pull/29823) - fix: greatly increase basedpyright slack by [@mateo-berri](https://github.com/mateo-berri) in [#30563](https://github.com/BerriAI/litellm/pull/30563) - fix(budget): recompute budget\_reset\_at when budget\_duration changes on /budget/update by [@yassin-berriai](https://github.com/yassin-berriai) in [#30555](https://github.com/BerriAI/litellm/pull/30555) - fix(otel): accept UPPER\_SNAKE\_CASE OTEL\_INSTRUMENTATION\_GENAI\_CAPTURE\_MESSAGE\_CONTENT in v2 by [@yassin-berriai](https://github.com/yassin-berriai) in [#30562](https://github.com/BerriAI/litellm/pull/30562) - chore(lint): remove PLR0915 too-many-statements ruff rule by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30574](https://github.com/BerriAI/litellm/pull/30574) - ci(lint): ratcheted type-discipline gate (mutable collections, casts, guards, kwargs, suppressions) by [@mateo-berri](https://github.com/mateo-berri) in [#30500](https://github.com/BerriAI/litellm/pull/30500) - feat(proxy): add verification\_uri\_complete to CLI SSO device flow by [@yassin-berriai](https://github.com/yassin-berriai) in [#30571](https://github.com/BerriAI/litellm/pull/30571) - chore: litellm oss staging160626 by [@Sameerlite](https://github.com/Sameerlite) in [#30527](https://github.com/BerriAI/litellm/pull/30527) - fix(guardrails): return 400 not 500 when AIM blocks a request by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30573](https://github.com/BerriAI/litellm/pull/30573) - ci(lint): grandfather any-discipline with a per-file ratchet budget (50% headroom) by [@mateo-berri](https://github.com/mateo-berri) in [#30582](https://github.com/BerriAI/litellm/pull/30582) - fix(audio): don't override explicit response\_format with verbose\_json by [@mateo-berri](https://github.com/mateo-berri) in [#30599](https://github.com/BerriAI/litellm/pull/30599) - fix(anthropic): price and surface response service\_tier in cost tracking by [@mateo-berri](https://github.com/mateo-berri) in [#30558](https://github.com/BerriAI/litellm/pull/30558) - feat: add dev and wildcard proxy configs for local testing by [@mateo-berri](https://github.com/mateo-berri) in [#30556](https://github.com/BerriAI/litellm/pull/30556) - fix(proxy): list public team model name in /v1/models by [@ishaan-berri](https://github.com/ishaan-berri) in [#30588](https://github.com/BerriAI/litellm/pull/30588) - ci: drop mypy entirely, standardize type checking on basedpyright by [@mateo-berri](https://github.com/mateo-berri) in [#30648](https://github.com/BerriAI/litellm/pull/30648) - feat(guardrails): surface OpenAI moderation violation\_categories on guardrail traces by [@yassin-berriai](https://github.com/yassin-berriai) in [#30659](https://github.com/BerriAI/litellm/pull/30659) - fix(proxy): resolve list files credentials from team BYOK deployments by [@shivamrawat1](https://github.com/shivamrawat1) in [#30495](https://github.com/BerriAI/litellm/pull/30495) - feat(proxy): add --max\_requests\_before\_restart\_jitter to stagger worker restarts by [@yassin-berriai](https://github.com/yassin-berriai) in [#30601](https://github.com/BerriAI/litellm/pull/30601) - fix(health): correct bedrock embedding health checks by [@mateo-berri](https://github.com/mateo-berri) in [#30583](https://github.com/BerriAI/litellm/pull/30583) - test: harden remaining pass-through CI flakes (image-gen spend poll, ruby assistants timeout) by [@mateo-berri](https://github.com/mateo-berri) in [#30685](https://github.com/BerriAI/litellm/pull/30685) - test(pass\_through): harden vertex spendlog poll against transient empty reads by [@mateo-berri](https://github.com/mateo-berri) in [#30683](https://github.com/BerriAI/litellm/pull/30683) - fix(cost): stop non-string service\_tier from silently dropping cost tracking by [@mateo-berri](https://github.com/mateo-berri) in [#30690](https://github.com/BerriAI/litellm/pull/30690) - feat(proxy): warn at startup when custom\_auth skips common\_checks enforcement by [@tin-berri](https://github.com/tin-berri) in [#30665](https://github.com/BerriAI/litellm/pull/30665) - fix(pod\_lock): release cron lock by matching async\_set\_cache JSON encoding by [@yassin-berriai](https://github.com/yassin-berriai) in [#30600](https://github.com/BerriAI/litellm/pull/30600) - ci: run a local fake OpenAI endpoint instead of the shared Railway mock by [@mateo-berri](https://github.com/mateo-berri) in [#30695](https://github.com/BerriAI/litellm/pull/30695) - ci(windows): pin uv to Python 3.11 so it ignores the preinstalled 3.14 by [@mateo-berri](https://github.com/mateo-berri) in [#30704](https://github.com/BerriAI/litellm/pull/30704) - feat(ui): migrate models page to App Router path route by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30677](https://github.com/BerriAI/litellm/pull/30677) - refactor(ui): remove orphaned pass-through-settings route by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30692](https://github.com/BerriAI/litellm/pull/30692) - fix(cost): stop non-string response service\_tier from dropping cost tracking by [@yuneng-berri](https://github.com/yuneng-berri) in [#30706](https://github.com/BerriAI/litellm/pull/30706) - feat(agent-shin): automated PR/issue triage, low-quality auto-close, and review-gate label lifecycle by [@mateo-berri](https://github.com/mateo-berri) in [#30433](https://github.com/BerriAI/litellm/pull/30433) - chore: litellm oss 170626 by [@Sameerlite](https://github.com/Sameerlite) in [#30637](https://github.com/BerriAI/litellm/pull/30637) - fix(bedrock\_mantle): add SigV4 fallback to chat completions auth by [@mateo-berri](https://github.com/mateo-berri) in [#30714](https://github.com/BerriAI/litellm/pull/30714) - feat(search): add TinyFish as search provider by [@simantak-dabhade](https://github.com/simantak-dabhade) in [#30634](https://github.com/BerriAI/litellm/pull/30634) - feat(ui): migrate old usage report to App Router path route by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30694](https://github.com/BerriAI/litellm/pull/30694) - fix(proxy): enforce budgets against authoritative DB spend when the cross-pod counter is stale by [@yassin-berriai](https://github.com/yassin-berriai) in [#30684](https://github.com/BerriAI/litellm/pull/30684) - chore(ci): remove Agent Shin pull\_request\_target workflows by [@yuneng-berri](https://github.com/yuneng-berri) in [#30784](https://github.com/BerriAI/litellm/pull/30784) - chore: litellm oss staging by [@Sameerlite](https://github.com/Sameerlite) in [#30745](https://github.com/BerriAI/litellm/pull/30745) - ci(zizmor): also run on litellm\_internal\_staging by [@yuneng-berri](https://github.com/yuneng-berri) in [#30789](https://github.com/BerriAI/litellm/pull/30789) - fix(test): drop references to removed Agent Shin workflows by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30791](https://github.com/BerriAI/litellm/pull/30791) - chore: remove in-product survey and Claude Code feedback nudges by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30773](https://github.com/BerriAI/litellm/pull/30773) - feat(ui): migrate api-keys landing to App Router path route by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30699](https://github.com/BerriAI/litellm/pull/30699) - feat(proxy): configurable response headers and login-page hint by [@yucheng-berri](https://github.com/yucheng-berri) in [#30792](https://github.com/BerriAI/litellm/pull/30792) - ci(zizmor): gate PRs on medium+ findings and clear existing ones by [@yuneng-berri](https://github.com/yuneng-berri) in [#30797](https://github.com/BerriAI/litellm/pull/30797) - fix(proxy): use e.request\_data for logging\_obj in ModifyResponseException streaming passthrough by [@mateo-berri](https://github.com/mateo-berri) in [#30800](https://github.com/BerriAI/litellm/pull/30800) - chore: make pr template linear portion clearer by [@mateo-berri](https://github.com/mateo-berri) in [#30766](https://github.com/BerriAI/litellm/pull/30766) - chore(typing): add boto3/botocore stubs so basedpyright resolves the AWS SDK by [@mateo-berri](https://github.com/mateo-berri) in [#30815](https://github.com/BerriAI/litellm/pull/30815) - fix(otel): one v2 logger owns the global provider; scope tenant OTLP creds per exporter by [@yucheng-berri](https://github.com/yucheng-berri) in [#30590](https://github.com/BerriAI/litellm/pull/30590) - fix(passthrough): recover output tokens for interrupted anthropic streams by [@yassin-berriai](https://github.com/yassin-berriai) in [#30787](https://github.com/BerriAI/litellm/pull/30787) - fix(proxy): record partial spend on the failure row for interrupted streams by [@yassin-berriai](https://github.com/yassin-berriai) in [#30788](https://github.com/BerriAI/litellm/pull/30788) - fix(ui): repoint dead usage guide link to cost tracking docs by [@yuneng-berri](https://github.com/yuneng-berri) in [#30859](https://github.com/BerriAI/litellm/pull/30859) - fix(ui): warn that team models are deleted in the delete-team modal by [@yuneng-berri](https://github.com/yuneng-berri) in [#29990](https://github.com/BerriAI/litellm/pull/29990) - feat(caching): add valkey-semantic cache backend and fix semantic cache scope keys by [@yassin-berriai](https://github.com/yassin-berriai) in [#30675](https://github.com/BerriAI/litellm/pull/30675) - test(ui): isolate OldTeams delete-warning tests from leaked mock by [@mateo-berri](https://github.com/mateo-berri) in [#30871](https://github.com/BerriAI/litellm/pull/30871) - feat: add lint-gate target and truncation-proof summary to the strict ruff gate by [@mateo-berri](https://github.com/mateo-berri) in [#30877](https://github.com/BerriAI/litellm/pull/30877) - chore(ui): rebuild ui for release by [@yuneng-berri](https://github.com/yuneng-berri) in [#30894](https://github.com/BerriAI/litellm/pull/30894) - chore(ci): bump deps by [@yuneng-berri](https://github.com/yuneng-berri) in [#30899](https://github.com/BerriAI/litellm/pull/30899) - fix(watsonx): wrap string embedding input in array for WatsonX API by [@shivamrawat1](https://github.com/shivamrawat1) in [#30897](https://github.com/BerriAI/litellm/pull/30897) - test: point router/completion/triton tests at the local fake OpenAI endpoint by [@mateo-berri](https://github.com/mateo-berri) in [#30900](https://github.com/BerriAI/litellm/pull/30900) - feat(sandbox): e2b code execution primitive by [@krrish-berri-2](https://github.com/krrish-berri-2) in [#30898](https://github.com/BerriAI/litellm/pull/30898) - fix(ui): source api-keys identity from useAuthorized to stop "User ID is not set" by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30903](https://github.com/BerriAI/litellm/pull/30903) - chore(ui): rebuild ui by [@yuneng-berri](https://github.com/yuneng-berri) in [#30906](https://github.com/BerriAI/litellm/pull/30906) - chore(ci): promote internal staging to main by [@yuneng-berri](https://github.com/yuneng-berri) in [#30907](https://github.com/BerriAI/litellm/pull/30907) - fix(redis): prevent forcing SSLConnection when ssl=False in connection pool by [@Jacopos311](https://github.com/Jacopos311) in [#30770](https://github.com/BerriAI/litellm/pull/30770) - fix(proxy): log UI setup failures instead of silently swallowing by [@sarvesh1327](https://github.com/sarvesh1327) in [#30819](https://github.com/BerriAI/litellm/pull/30819) ##### New Contributors - [@simantak-dabhade](https://github.com/simantak-dabhade) made their first contribution in [#30634](https://github.com/BerriAI/litellm/pull/30634) - [@Jacopos311](https://github.com/Jacopos311) made their first contribution in [#30770](https://github.com/BerriAI/litellm/pull/30770) - [@sarvesh1327](https://github.com/sarvesh1327) made their first contribution in [#30819](https://github.com/BerriAI/litellm/pull/30819) **Full Changelog**: <https://github.com/BerriAI/litellm/compare/v1.89.0...v1.90.0> ### [`v1.90.0`](https://github.com/BerriAI/litellm/releases/tag/v1.90.0) [Compare Source](https://github.com/BerriAI/litellm/compare/v1.89.4...v1.90.0) ##### Verify Docker Image Signature All LiteLLM Docker images are signed with [cosign](https://docs.sigstore.dev/cosign/overview/). Every release is signed with the same key introduced in [commit `0112e53`](https://github.com/BerriAI/litellm/commit/0112e53046018d726492c814b3644b7d376029d0). **Verify using the pinned commit hash (recommended):** A commit hash is cryptographically immutable, so this is the strongest way to ensure you are using the original signing key: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/0112e53046018d726492c814b3644b7d376029d0/cosign.pub \ ghcr.io/berriai/litellm:v1.90.0 ``` **Verify using the release tag (convenience):** Tags are protected in this repository and resolve to the same key. This option is easier to read but relies on tag protection rules: ```bash cosign verify \ --key https://raw.githubusercontent.com/BerriAI/litellm/v1.90.0/cosign.pub \ ghcr.io/berriai/litellm:v1.90.0 ``` Expected output: ``` The following checks were performed on each of these signatures: - The cosign claims were validated - The signatures were verified against the specified public key ``` *** ##### What's Changed - fix(responses-bridge): map system-only chat request to system input item by [@milan-berri](https://github.com/milan-berri) in [#29817](https://github.com/BerriAI/litellm/pull/29817) - feat(bedrock): forward strict and additionalProperties to Converse toolSpec by [@mateo-berri](https://github.com/mateo-berri) in [#29814](https://github.com/BerriAI/litellm/pull/29814) - fix(mcp): highlight MCP cards red when the logged-in user is missing per-user env vars by [@mateo-berri](https://github.com/mateo-berri) in [#29856](https://github.com/BerriAI/litellm/pull/29856) - feat(ui): add budget duration to edit team member form by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#29717](https://github.com/BerriAI/litellm/pull/29717) - fix(ui): make workflow runs page fill full width by [@yuneng-berri](https://github.com/yuneng-berri) in [#29868](https://github.com/BerriAI/litellm/pull/29868) - feat: standardize rate limit errors with category, rate\_limit\_type, model, and llm\_provider fields by [@mateo-berri](https://github.com/mateo-berri) in [#27687](https://github.com/BerriAI/litellm/pull/27687) - fix(ui): default guardrails page to the Guardrails tab by [@yuneng-berri](https://github.com/yuneng-berri) in [#29872](https://github.com/BerriAI/litellm/pull/29872) - docs(readme): add Deploy on AWS/GCP Terraform section and fix deploy button rendering by [@mateo-berri](https://github.com/mateo-berri) in [#29879](https://github.com/BerriAI/litellm/pull/29879) - refactor(bedrock): build Converse toolSpec via a BedrockToolSpec dict subclass by [@mateo-berri](https://github.com/mateo-berri) in [#29869](https://github.com/BerriAI/litellm/pull/29869) - feat(litellm): add models and repository layers by [@yassin-berriai](https://github.com/yassin-berriai) in [#29686](https://github.com/BerriAI/litellm/pull/29686) - feat(ui): include internal routes in the dashboard's generated OpenAPI types by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#29885](https://github.com/BerriAI/litellm/pull/29885) - feat(proxy): publish /v2/model/info in Swagger OpenAPI spec by [@Sameerlite](https://github.com/Sameerlite) in [#29900](https://github.com/BerriAI/litellm/pull/29900) - refactor(ui): single source of truth for migrated-page routing by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#29949](https://github.com/BerriAI/litellm/pull/29949) - fix(ui/model-hub): render provider icons on the public model hub by [@yuneng-berri](https://github.com/yuneng-berri) in [#29958](https://github.com/BerriAI/litellm/pull/29958) - fix(ui): keep create guardrail modal open on outside click by [@yuneng-berri](https://github.com/yuneng-berri) in [#29871](https://github.com/BerriAI/litellm/pull/29871) - fix(ui): label default key type as "Full Access" on key edit page by [@yuneng-berri](https://github.com/yuneng-berri) in [#29870](https://github.com/BerriAI/litellm/pull/29870) - fix(ui): unify migrated-route URLs and migrate the API Reference page by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#29953](https://github.com/BerriAI/litellm/pull/29953) - fix(mcp): let non-creator users OAuth into OBO-mode MCP servers from the Tools page by [@tin-berri](https://github.com/tin-berri) in [#29867](https://github.com/BerriAI/litellm/pull/29867) - Litellm oss staging 080626 by [@Sameerlite](https://github.com/Sameerlite) in [#29932](https://github.com/BerriAI/litellm/pull/29932) - feat(galileo): add health check support for UI callback test by [@Sameerlite](https://github.com/Sameerlite) in [#29908](https://github.com/BerriAI/litellm/pull/29908) - fix(model-management): allow deleting a BYOK model after its team is deleted by [@yuneng-berri](https://github.com/yuneng-berri) in [#29875](https://github.com/BerriAI/litellm/pull/29875) - feat(jwt-auth): opt-in fallback to DB team on unresolved JWT claim by [@milan-berri](https://github.com/milan-berri) in [#28913](https://github.com/BerriAI/litellm/pull/28913) - fix(team\_endpoints): don't block /team/update on unchanged team budget by [@milan-berri](https://github.com/milan-berri) in [#29525](https://github.com/BerriAI/litellm/pull/29525) - fix(fireworks): enable tool calling for glm-5p1 in model cost map by [@milan-berri](https://github.com/milan-berri) in [#29697](https://github.com/BerriAI/litellm/pull/29697) - fix(vertex): propagate Vertex AI metadata in streaming success callbacks by [@Sameerlite](https://github.com/Sameerlite) in [#29899](https://github.com/BerriAI/litellm/pull/29899) - fix(ui): show team projects to internal users on key creation by [@milan-berri](https://github.com/milan-berri) in [#28855](https://github.com/BerriAI/litellm/pull/28855) - build(deps): bump pyjwt to 2.13.0 and ws override to 8.20.1 by [@yuneng-berri](https://github.com/yuneng-berri) in [#29982](https://github.com/BerriAI/litellm/pull/29982) - fix(team-management): delete a team's BYOK models when the team is deleted by [@yuneng-berri](https://github.com/yuneng-berri) in [#29977](https://github.com/BerriAI/litellm/pull/29977) - feat(vantage): include organization metadata in FOCUS Tags export by [@milan-berri](https://github.com/milan-berri) in [#28184](https://github.com/BerriAI/litellm/pull/28184) - fix(guardrails): read CrowdStrike AIDR identity from both metadata bags by [@yuneng-berri](https://github.com/yuneng-berri) in [#29991](https://github.com/BerriAI/litellm/pull/29991) - fix(mcp): mirror upstream token lifetime instead of forcing a 1h OBO expiry by [@tin-berri](https://github.com/tin-berri) in [#29951](https://github.com/BerriAI/litellm/pull/29951) - feat(azure\_ai): add MAI-Image-2.5 image generation support by [@Sameerlite](https://github.com/Sameerlite) in [#29688](https://github.com/BerriAI/litellm/pull/29688) - fix(mcp): load MCP tool configuration tools via the OBO/passthrough-aware GET path by [@tin-berri](https://github.com/tin-berri) in [#29960](https://github.com/BerriAI/litellm/pull/29960) - fix(team): reserve team budget raises for proxy admins on /team/update by [@milan-berri](https://github.com/milan-berri) in [#30030](https://github.com/BerriAI/litellm/pull/30030) - test(ui): data-driven App Router migration E2E smoke (default + server-root-path) by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#29974](https://github.com/BerriAI/litellm/pull/29974) - fix(proxy): extend response headers hook to streaming, TTS, image gen, and pass-through by [@michelligabriele](https://github.com/michelligabriele) in [#24232](https://github.com/BerriAI/litellm/pull/24232) - chore(ui): remove dead App Router route stubs under (dashboard) by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30045](https://github.com/BerriAI/litellm/pull/30045) - fix(ui/mcp): reset OAuth state on create-server modal close so a prior server's token no longer leaks into the next add-server session by [@tin-berri](https://github.com/tin-berri) in [#30000](https://github.com/BerriAI/litellm/pull/30000) - fix(mcp): allow team access-group grants in OAuth authorize/token access check by [@tin-berri](https://github.com/tin-berri) in [#30041](https://github.com/BerriAI/litellm/pull/30041) - docs(security): require a reproduction video for vulnerability reports by [@yuneng-berri](https://github.com/yuneng-berri) in [#30063](https://github.com/BerriAI/litellm/pull/30063) - feat(ui): add admin flag to disable in-product UI nudges for everyone by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#29796](https://github.com/BerriAI/litellm/pull/29796) - chore(ui): remove dead dashboard files and unused dependencies by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30047](https://github.com/BerriAI/litellm/pull/30047) - fix(proxy): authorize batch files using upload target\_model\_names (LIT-3593) by [@Sameerlite](https://github.com/Sameerlite) in [#30009](https://github.com/BerriAI/litellm/pull/30009) - Add Claude Fable 5 across Anthropic, Bedrock, Vertex AI, and Azure AI by [@mateo-berri](https://github.com/mateo-berri) in [#30064](https://github.com/BerriAI/litellm/pull/30064) - Add Claude Fable 5 cost map entries (data-only hotfix for the hosted map) by [@mateo-berri](https://github.com/mateo-berri) in [#30076](https://github.com/BerriAI/litellm/pull/30076) - fix(caching): restore stored prompt\_tokens on embedding cache hits instead of recomputing by [@michelligabriele](https://github.com/michelligabriele) in [#30046](https://github.com/BerriAI/litellm/pull/30046) - Litellm oss 090626 by [@Sameerlite](https://github.com/Sameerlite) in [#30021](https://github.com/BerriAI/litellm/pull/30021) - fix(proxy): self-heal startup/reload prisma reads on engine disconnect by [@michelligabriele](https://github.com/michelligabriele) in [#28803](https://github.com/BerriAI/litellm/pull/28803) - chore(ui): make knip recognize .mjs scripts and openapi-typescript by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30052](https://github.com/BerriAI/litellm/pull/30052) - fix(register\_model): preserve built-in cache pricing when registering custom overrides under unmapped keys by [@yassin-berriai](https://github.com/yassin-berriai) in [#30044](https://github.com/BerriAI/litellm/pull/30044) - \[internal copy of [#28007](https://github.com/BerriAI/litellm/issues/28007)] Fix/gcp model garden streaming by [@mateo-berri](https://github.com/mateo-berri) in [#28363](https://github.com/BerriAI/litellm/pull/28363) - feat(cli): per-agent `lite claude` / `codex` / `opencode` commands that wrap coding agents through the proxy by [@mateo-berri](https://github.com/mateo-berri) in [#29850](https://github.com/BerriAI/litellm/pull/29850) - fix(callbacks): forward callback\_settings to callback initializers and guard consumers against non-dict values by [@yuneng-berri](https://github.com/yuneng-berri) in [#30161](https://github.com/BerriAI/litellm/pull/30161) - fix(mcp): drop orphaned per-user credential rows when an MCP server is deleted by [@tin-berri](https://github.com/tin-berri) in [#30141](https://github.com/BerriAI/litellm/pull/30141) - fix(proxy): recover from cached-plan errors by reconnecting the Prisma client by [@yassin-berriai](https://github.com/yassin-berriai) in [#29983](https://github.com/BerriAI/litellm/pull/29983) - feat(proxy): add option to disable server-side prepared statements for DB lookups by [@yassin-berriai](https://github.com/yassin-berriai) in [#29984](https://github.com/BerriAI/litellm/pull/29984) - fix(release): stop backport releases from overwriting the latest badge by [@yuneng-berri](https://github.com/yuneng-berri) in [#30005](https://github.com/BerriAI/litellm/pull/30005) - feat: add conventional commits and coding guidelines by [@mateo-berri](https://github.com/mateo-berri) in [#30159](https://github.com/BerriAI/litellm/pull/30159) - fix(proxy): return 5xx on DB infra errors during auth; reserve 401 for genuine auth failures by [@yassin-berriai](https://github.com/yassin-berriai) in [#29986](https://github.com/BerriAI/litellm/pull/29986) - fix(ui): dev server 404s on migrated-page links because uiBase hardcodes /ui by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30169](https://github.com/BerriAI/litellm/pull/30169) - refactor(ui): consolidate dashboard to one shell in the (dashboard) layout by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30166](https://github.com/BerriAI/litellm/pull/30166) - fix(proxy): align /v1/model/info with router deployments by [@Sameerlite](https://github.com/Sameerlite) in [#30025](https://github.com/BerriAI/litellm/pull/30025) - fix: completion\_cost AttributeError on streaming Anthropic web\_search responses ([#26153](https://github.com/BerriAI/litellm/issues/26153)) by [@ishaan-berri](https://github.com/ishaan-berri) in [#27346](https://github.com/BerriAI/litellm/pull/27346) - \[internal copy of [#30137](https://github.com/BerriAI/litellm/issues/30137)] perf(realtime): eliminate redundant per-frame JSON work on OpenAI realtime relay by [@mateo-berri](https://github.com/mateo-berri) in [#30142](https://github.com/BerriAI/litellm/pull/30142) - feat(bedrock): aws\_bedrock\_project\_id for bedrock-mantle project / workspace association by [@mateo-berri](https://github.com/mateo-berri) in [#30163](https://github.com/BerriAI/litellm/pull/30163) - chore(hooks): enforce Conventional Commits and Conventional Branches by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30174](https://github.com/BerriAI/litellm/pull/30174) - feat(rate-limiter): allow opting out of v3 TPM reservation and Redis circuit breaker by [@yassin-berriai](https://github.com/yassin-berriai) in [#30211](https://github.com/BerriAI/litellm/pull/30211) - feat(spend\_logs): opt-in native Postgres partitioning for SpendLogs retention by [@yassin-berriai](https://github.com/yassin-berriai) in [#29466](https://github.com/BerriAI/litellm/pull/29466) - feat(ui): migrate playground to path routing and colocate its files by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30185](https://github.com/BerriAI/litellm/pull/30185) - feat(ui): migrate projects and access-groups to path routes by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30226](https://github.com/BerriAI/litellm/pull/30226) - fix(proxy): coalesce NULL rollup metrics in aggregated daily-activity by [@michelligabriele](https://github.com/michelligabriele) in [#30151](https://github.com/BerriAI/litellm/pull/30151) - fix(anthropic\_passthrough): resolve costing model from message\_start chunk, litellm\_params and model\_group instead of 'unknown' by [@yassin-berriai](https://github.com/yassin-berriai) in [#30160](https://github.com/BerriAI/litellm/pull/30160) - feat(ui): migrate budgets, workflows, and guardrails-monitor to path routes by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30236](https://github.com/BerriAI/litellm/pull/30236) - feat(ui): migrate mcp-servers, search-tools, tag-management, vector-stores, and memory to path routes by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30261](https://github.com/BerriAI/litellm/pull/30261) - fix(a2a): forward agent\_extra\_headers through completion bridge by [@mateo-berri](https://github.com/mateo-berri) in [#28277](https://github.com/BerriAI/litellm/pull/28277) - fix(gemini-live): forward audio buffer commit and correct Vertex PCM rate by [@Sameerlite](https://github.com/Sameerlite) in [#29946](https://github.com/BerriAI/litellm/pull/29946) - fix(proxy): skip double-wrapping unified batch output file ids on retrieve by [@Sameerlite](https://github.com/Sameerlite) in [#30011](https://github.com/BerriAI/litellm/pull/30011) - feat: litellm oss 110626 by [@Sameerlite](https://github.com/Sameerlite) in [#30202](https://github.com/BerriAI/litellm/pull/30202) - fix(docker): copy only runtime artifacts into the final image by [@yuneng-berri](https://github.com/yuneng-berri) in [#30243](https://github.com/BerriAI/litellm/pull/30243) - feat(proxy): enforce key/team guardrails on bedrock passthrough routes by [@Sameerlite](https://github.com/Sameerlite) in [#30194](https://github.com/BerriAI/litellm/pull/30194) - feat(gemini): forward web search tools in image generation by [@Sameerlite](https://github.com/Sameerlite) in [#30119](https://github.com/BerriAI/litellm/pull/30119) - fix: bedrock mantle fixes by [@Sameerlite](https://github.com/Sameerlite) in [#30083](https://github.com/BerriAI/litellm/pull/30083) - feat(proxy): add require\_managed\_files setting for file uploads by [@Sameerlite](https://github.com/Sameerlite) in [#30186](https://github.com/BerriAI/litellm/pull/30186) - fix(mcp): honor server\_id for REST tool calls with shared upstream URLs by [@Sameerlite](https://github.com/Sameerlite) in [#30184](https://github.com/BerriAI/litellm/pull/30184) - fix(responses): presidio PII masking for Azure WebSocket and streaming by [@Sameerlite](https://github.com/Sameerlite) in [#30003](https://github.com/BerriAI/litellm/pull/30003) - feat(passthrough): add configurable pass-through request timeouts by [@Sameerlite](https://github.com/Sameerlite) in [#30266](https://github.com/BerriAI/litellm/pull/30266) - fix(google\_genai): preserve complete SSE events in Vertex/Gemini image streaming by [@Sameerlite](https://github.com/Sameerlite) in [#30270](https://github.com/BerriAI/litellm/pull/30270) - fix(proxy): populate access\_via\_team\_ids on /v1/model/info by [@Sameerlite](https://github.com/Sameerlite) in [#30274](https://github.com/BerriAI/litellm/pull/30274) - chore(oss): litellm oss staging 120626 by [@Sameerlite](https://github.com/Sameerlite) in [#30292](https://github.com/BerriAI/litellm/pull/30292) - feat(ui): migrate policies, guardrails, prompts, tool-policies, and skills to path routes by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30263](https://github.com/BerriAI/litellm/pull/30263) - feat(ui): migrate caching, cost-tracking, transform-request, ui-theme, and logs to path routes by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30267](https://github.com/BerriAI/litellm/pull/30267) - fix(ui): gate dashboard layout on ui config load so deep links work under SERVER\_ROOT\_PATH by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30312](https://github.com/BerriAI/litellm/pull/30312) - feat(ui): migrate admin-panel, logging-and-alerts, model-hub-table, and usage to path routes by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30268](https://github.com/BerriAI/litellm/pull/30268) - fix(otel): cap metric attribute cardinality with include/exclude lists by [@yassin-berriai](https://github.com/yassin-berriai) in [#30257](https://github.com/BerriAI/litellm/pull/30257) - fix(proxy): grace-period key rotation 401s; return deprecated-key lookup result directly by [@yuneng-berri](https://github.com/yuneng-berri) in [#30327](https://github.com/BerriAI/litellm/pull/30327) - chore(deps): bump vitest, brace-expansion, pypdf and tornado by [@yuneng-berri](https://github.com/yuneng-berri) in [#30220](https://github.com/BerriAI/litellm/pull/30220) - refactor(ui): remove unreachable /chat page by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30178](https://github.com/BerriAI/litellm/pull/30178) - feat(ui): migrate agents and router-settings to path routes by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30323](https://github.com/BerriAI/litellm/pull/30323) - feat: strengthen coding conventions in CLAUDE.md by [@mateo-berri](https://github.com/mateo-berri) in [#30333](https://github.com/BerriAI/litellm/pull/30333) - feat(ui): cut the users page over to the /ui/users path route by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30334](https://github.com/BerriAI/litellm/pull/30334) - feat: ruff strict-rule suppressions baseline gate by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30303](https://github.com/BerriAI/litellm/pull/30303) - feat(guardrails): add Cisco AI Defense integration ([#28249](https://github.com/BerriAI/litellm/issues/28249)) by [@yassin-berriai](https://github.com/yassin-berriai) in [#30338](https://github.com/BerriAI/litellm/pull/30338) - chore(ui): remove dead UI components unreferenced by any page by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30340](https://github.com/BerriAI/litellm/pull/30340) - ci: add osv-scanner lockfile scan workflow by [@yuneng-berri](https://github.com/yuneng-berri) in [#30222](https://github.com/BerriAI/litellm/pull/30222) - fix(otel): record full error message on standard exception event in otel v2 by [@yassin-berriai](https://github.com/yassin-berriai) in [#30380](https://github.com/BerriAI/litellm/pull/30380) - test(fireworks): mock whisper transcription tests instead of live calls by [@yuneng-berri](https://github.com/yuneng-berri) in [#30391](https://github.com/BerriAI/litellm/pull/30391) - build(ui): pin esbuild to 0.28.1 via overrides by [@yuneng-berri](https://github.com/yuneng-berri) in [#30390](https://github.com/BerriAI/litellm/pull/30390) - feat(ui): cut the organizations page over to the /ui/organizations path route by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30336](https://github.com/BerriAI/litellm/pull/30336) - fix(proxy): support SMTP implicit SSL (port 465) by [@yuneng-berri](https://github.com/yuneng-berri) in [#30395](https://github.com/BerriAI/litellm/pull/30395) - fix(mcp): default Linear MCP registry entry to streamable HTTP by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30396](https://github.com/BerriAI/litellm/pull/30396) - fix(ui): stop Virtual Keys page from infinite render loop by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30397](https://github.com/BerriAI/litellm/pull/30397) - fix(streaming): guard raise\_on\_model\_repetition against empty choices by [@shivamrawat1](https://github.com/shivamrawat1) in [#30485](https://github.com/BerriAI/litellm/pull/30485) - feat(otel-v2): emit the 6 gen\_ai.client.\* metrics at parity with v1 by [@yassin-berriai](https://github.com/yassin-berriai) in [#30326](https://github.com/BerriAI/litellm/pull/30326) - fix(mcp): drop phantom 401 span on delegated OAuth2 tool calls by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30494](https://github.com/BerriAI/litellm/pull/30494) - feat(ui): cut the teams page over to the /ui/teams path route by [@ryan-crabbe-berri](https://github.com/ryan-crabbe-berri) in [#30343](https://github.com/BerriAI/litellm/pull/30343) - fix(integrations): cap Anthropic cache\_control injection at 4 blocks by [@shivamrawat1](https://github.com/shivamrawat1) in [#30480](https://github.com/BerriAI/litellm/pull/30480) - chore(codecov): add Batches, Videos, and Realtime components by [@Sameerlite](https://github.com/Sameerlite) in [#30517](https://github.com/BerriAI/litellm/pull/30517) - test(batches): move orphan tests into tests/test\_litellm for CI coverage by [@Sameerlite](https://github.com/Sameerlite) in [#30510](https://github.com/BerriAI/litellm/pull/30510) - fix(guardrails): run pre\_call hook once for model-level guardrails by [@yassin-berriai](https://github.com/yassin-berriai) in [#30543](http…

greptile-apps Bot reviewed Jun 16, 2026

View reviewed changes

Comment thread litellm/proxy/guardrails/guardrail_registry.py

ishaan-berri requested changes Jun 16, 2026

View reviewed changes

Comment thread litellm/proxy/guardrails/guardrail_registry.py Outdated

yassin-berriai commented Jun 16, 2026

View reviewed changes

yassin-berriai enabled auto-merge (squash) June 16, 2026 18:16

ishaan-berri approved these changes Jun 16, 2026

View reviewed changes

yassin-berriai merged commit 9fa74ad into litellm_internal_staging Jun 16, 2026
120 checks passed

yassin-berriai deleted the litellm_fix_guardrail_dup_init branch June 16, 2026 18:17

yuneng-berri mentioned this pull request Jun 17, 2026

chore(release): backport #30543, #30542 to stable/1.88.x and cut 1.88.3 #30680

Merged

3 tasks

yuneng-berri added a commit that referenced this pull request Jun 17, 2026

Merge pull request #30680 from BerriAI/litellm_backport_1_88_x_bp-305…

9c135ab

…43-30542-30274-0617 chore(release): backport #30543, #30542 to stable/1.88.x and cut 1.88.3

This was referenced Jun 20, 2026

chore(release): backport #30480, #30543, #30542, #30573 to stable/1.89.x and cut 1.89.3 #30888

Merged

chore(release): backport #29311, #29444, #29447, #29598, #30480, #30543, #30542, #30573 to stable/1.87.x and cut 1.87.4 #30890

Merged

yuneng-berri added a commit that referenced this pull request Jun 20, 2026

Merge pull request #30890 from BerriAI/litellm_backport_1_87_x_0620

faa2f13

chore(release): backport #29311, #29444, #29447, #29598, #30480, #30543, #30542, #30573 to stable/1.87.x and cut 1.87.4

yuneng-berri added a commit that referenced this pull request Jun 20, 2026

Merge pull request #30888 from BerriAI/litellm_backport_1_89_x_0620

33df589

chore(release): backport #30480, #30543, #30542, #30573 to stable/1.89.x and cut 1.89.3

mubashir1osmani mentioned this pull request Jun 23, 2026

fix(guardrails): run pre_call hook once for model-level guardrails (#30543) #31144

Closed

4 tasks

claude Bot mentioned this pull request Jun 24, 2026

fix(container): update image ghcr.io/berriai/litellm-non_root (v1.89.2 ➔ v1.89.3) drag0n141/home-ops#6493

Merged

1 task

ishaan-berri mentioned this pull request Jun 25, 2026

LiteLLM Stability Sprint Roadmap #30484

Open

13 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(guardrails): stop re-initializing DB guardrails on every poll#30542

fix(guardrails): stop re-initializing DB guardrails on every poll#30542
yassin-berriai merged 3 commits into
litellm_internal_stagingfrom
litellm_fix_guardrail_dup_init

yassin-berriai commented Jun 16, 2026 •

edited

Loading

Uh oh!

CLAassistant commented Jun 16, 2026

Uh oh!

codecov Bot commented Jun 16, 2026 •

edited

Loading

Uh oh!

greptile-apps Bot commented Jun 16, 2026 •

edited

Loading

Important Files Changed

Uh oh!

Uh oh!

yassin-berriai commented Jun 16, 2026

Uh oh!

yassin-berriai commented Jun 16, 2026

Uh oh!

Uh oh!

yassin-berriai commented Jun 16, 2026

Uh oh!

yassin-berriai Jun 16, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Uh oh!

Conversation

yassin-berriai commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Relevant issues

Linear ticket

Pre-Submission checklist

Delays in PR merge?

CI (LiteLLM team)

Screenshots / Proof of Fix

Type

Changes

Uh oh!

CLAassistant commented Jun 16, 2026

Uh oh!

codecov Bot commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

greptile-apps Bot commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Greptile Summary

Confidence Score: 5/5

Important Files Changed

Uh oh!

Uh oh!

yassin-berriai commented Jun 16, 2026

Uh oh!

yassin-berriai commented Jun 16, 2026

Uh oh!

Uh oh!

yassin-berriai commented Jun 16, 2026

Uh oh!

yassin-berriai Jun 16, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

yassin-berriai commented Jun 16, 2026 •

edited

Loading

codecov Bot commented Jun 16, 2026 •

edited

Loading

greptile-apps Bot commented Jun 16, 2026 •

edited

Loading