* fix(proxy): optionally surface public team model name in /v1/models

Behind general_settings.use_team_public_model_name (default False). When
enabled, /v1/models and /models surface the public team_public_model_name
for team-scoped (BYOK) models instead of the internal routing key
model_name_{team_id}_{uuid} -- consistent with /v1/model/info and
OpenAI-compatible. Off by default so the listing's model ids stay
backward-compatible for callers that scripted against the internal name;
routing by the internal name is unchanged regardless of the flag.

Presentation-layer only: access-group, auth, and routing semantics are
unchanged; non-team models are pass-through.

* fix(proxy): default team model listings to public names

* test(proxy): cover team model listing metadata

* test(proxy): cover empty team listing deployments

* refactor(proxy): simplify team model listing translation

* fix(proxy): resolve public team model name on GET /v1/models/{id}

The listing endpoints advertise team_public_model_name, but the retrieve
endpoint validated and looked up by the raw id, so a public name 404'd.
Resolve the public name back to the internal routing key (scoped to the
caller's accessible models so colliding names never cross teams), look up
by it, and echo the public name back as the response id.

* test(proxy): cover public-name resolution on model retrieve

* refactor(proxy): extract team model-name translation into TeamModelNameTranslator

Move the team-scoped (BYOK) listing/retrieve name translation out of
proxy_server.py into a dedicated common_utils module. Static methods with
general_settings injected so the logic is unit-testable without globals and
proxy_server.py stays thin.

* refactor(proxy): use TeamModelNameTranslator in model_list and model_info

* test(proxy): target TeamModelNameTranslator for model-name translation

* fix(proxy): type create_model_info_response return as dict[str, object]

* fix(proxy): keep internal routing key for team model listing metadata lookup

Add listing_entries returning (public response id, internal lookup id) so
include_metadata=true resolves fallbacks against the routing key the router
indexes by, instead of the translated public name (which never matches).

* fix(proxy): build /v1/models metadata from internal key, show public id

* test(proxy): cover team listing fallback metadata via internal key

* fix(proxy): use builtin dict generics in create_model_info_response (UP006)

---------

Co-authored-by: Tushar More <[email protected]>
Co-authored-by: Ishaan Jaffer <[email protected]>

…0648)

* ci: drop redundant mypy type-check gate, standardize on basedpyright

Type checking ran both mypy (via the pydantic.mypy plugin) and basedpyright.
pydantic v2 emits dataclass_transform, so basedpyright understands models
natively with no plugin, and its gated rules already cover what the mypy pass
caught (no-untyped-def, no-any-return, valid-type, import-not-found all map to
basedpyright equivalents). Running both meant two checkers, two budgets, and a
plugin only mypy could load.

This removes the mypy type-check gate: the lint-mypy/lint-mypy-budget-update
Makefile targets, the CI MyPy step, mypy-code-budget.json, the budget-ratchet
entry, and the vestigial [tool.mypy] pydantic plugin block (the gating pass used
litellm/mypy.ini, which never loaded the plugin). type_check_gate.py is
specialized to basedpyright since the mypy parsing path is now unused.

mypy stays a dev dependency because the Any-discipline gate
(scripts/check_any_discipline.py) imports it as a library to detect Any-typed
values; it is no longer run as a type checker.

* ci: remove the Any-discipline gate, rely on basedpyright's reportAny

The Any-discipline gate (scripts/check_any_discipline.py) was the last consumer
of mypy: it imported mypy as a library to detect values whose inferred type
contains Any, gated per-file against any-discipline-budget.json. basedpyright
already reports the same class of finding through reportAny/reportExplicitAny,
which are gated tree-wide in basedpyright-code-budget.json, so the separate gate
(and the mypy dependency behind it) is redundant.

Removes the gate end to end: check_any_discipline.py and its test, the
any-discipline CI job, the lint-any/lint-any-budget-update Makefile targets,
any-discipline-budget.json, litellm/mypy.ini, the .mypy_cache_any references,
and mypy from the dev dependencies. budget_ratchet_check.py drops the
any-discipline entry and the now-unused zero-floor mechanism (rewritten as a
comprehension). check_type_discipline.py drops the any-ok suppression token,
since # any-ok suppressed only the deleted gate; the 134 now-orphaned
# any-ok comments across 14 files are stripped (they never affected
basedpyright, which uses # pyright: ignore).

uv.lock is intentionally left untouched: uv still considers it consistent with
the mypy-removed pyproject (uv lock --check and uv sync --frozen both pass), and
a relock bumps 30+ unrelated packages because of the moving exclude-newer window.
A future intentional relock will prune the now-unreferenced mypy entry.

* build: relock to drop mypy from uv.lock

CI's uv 0.10.9 honors the repo's exclude-newer window and correctly flags the
lockfile as out of sync once mypy leaves pyproject; my earlier local uv 0.8.17
could not parse exclude-newer and silently passed --check. Relocking with the
pinned CI version removes only mypy and its transitive librt, with no other
version changes.

…uardrail traces (#30659)

The OpenAI moderation guardrail (and the ai-platform-moderation guardrail
built on it) stamped the whole moderation model response into the guardrail
trace as guardrail_response. That blob carries the full category_scores map
plus categories and category_applied_input_types, which on OTEL backends that
index span attributes (for example ELK, which caps indexed attribute values at
1024 chars) overflows the limit and gets truncated, so the violated categories
cannot be reliably searched.

Extract the flagged category names from the moderation response and pass them
through tracing_detail to add_standard_logging_guardrail_information_to_request_data,
mirroring the Bedrock hook. Both the legacy and v2 OTEL integrations already
read violation_categories off the standard logging guardrail information and
emit it as a short, queryable guardrail_violation_categories attribute, so
dashboards can group and filter by violation category without parsing the large
guardrail_response blob.

Resolves LIT-3801

…#30495)

* fix(proxy): resolve list files credentials from team BYOK deployments

GET /v1/files without target_model_names now prefers the team's own
deployment (model_info.team_id) over shared global provider keys, so JWT
team auth lists files against the correct upstream account.

Co-authored-by: Cursor <[email protected]>

* fix(proxy): scope list files credential lookup to team allowlist

Remove the unrestricted deployment scan that could leak global provider
keys to teams without access, normalize all-proxy-models to the team-scoped
model list, and fix TID251 violations by using dict instead of Dict/Any.

Co-authored-by: Cursor <[email protected]>

---------

Co-authored-by: Cursor <[email protected]>

…er restarts (#30601)

Setting --max_requests_before_restart alone recycles every worker at almost the
same time once they have served a similar number of requests, which under
sustained load can drop a whole pod's capacity at once roughly every 7-10 days.

This exposes a jitter knob that adds a random amount in [0, jitter] to the
restart threshold per worker so restarts are staggered. It maps to uvicorn's
limit_max_requests_jitter and gunicorn's max_requests_jitter. uvicorn only
gained limit_max_requests_jitter in 0.41.0 while litellm still allows
uvicorn>=0.33.0, so the uvicorn path feature-detects the parameter via the
Config signature and warns instead of crashing on older versions. The flag has
no effect without --max_requests_before_restart, so the kwarg is not forwarded
in that case and a warning is printed on both the uvicorn and gunicorn paths.

Resolves LIT-3774

* fix(health): correct bedrock embedding health checks

Health checks for Bedrock embedding deployments failed in two ways. A
deployment configured without an explicit model_info.mode was probed as
chat, so max_tokens was injected and Bedrock embeddings rejected it with
400 "extraneous key [max_tokens]". Separately, stripping the bedrock/
routing prefix dropped the provider, so a cross-region inference-profile
id like us.cohere.embed-v4:0 failed downstream with "LLM Provider NOT
provided".

Resolve the deployment mode from the model cost map (which understands
the bedrock/ and us./eu./apac. prefixes) before deciding whether to
inject max_tokens, and pin custom_llm_provider to bedrock when stripping
the prefix so the bare model id still resolves. ahealth_check now accepts
any string mode so the resolved embedding mode routes the probe to the
embedding handler.

* fix(health): preserve explicit custom_llm_provider on bedrock probe

The bedrock prefix-strip pinned custom_llm_provider to bedrock
unconditionally, so a deployment that set custom_llm_provider:
bedrock_converse had it overwritten at health-check time and the probe
hit the Invoke endpoint instead of Converse, a different request format
that can report a spurious failure. Only fill in bedrock when the
deployment left the provider blank, which still resolves bare
cross-region ids like us.cohere.embed-v4:0 while leaving an explicit
provider untouched.

* test(health): assert resolved mode reaches the ahealth_check probe

The existing tests check _resolve_health_check_mode and the params builder
in isolation, but nothing verified that _run_model_health_check actually
threads the resolved mode into litellm.ahealth_check. Without that, a
refactor that probed with model_info.get("mode") again would reintroduce
the chat fallback for embedding deployments while every test stayed green.
This drives _run_model_health_check with a bedrock embedding deployment and
asserts the probe is called with mode=embedding and the embedding params.

* fix(health): resolve probe mode once for reasoning_effort and audio_speech

The reasoning_effort and audio_speech branches read model_info.mode
directly, so an embedding deployment declared without an explicit mode (the
case this PR targets) was still treated as chat-like: a configured
health_check_reasoning_effort got injected into the embedding probe, which
embeddings reject as an unknown field, and an auto-detected audio_speech
deployment never had its voice set. Resolve the effective mode once from the
cost map and reuse it for the max_tokens, reasoning_effort, and audio_speech
decisions so they all agree with the mode threaded into ahealth_check.

…ruby assistants timeout) (#30685)

* test(proxy): poll for image-gen spend instead of a fixed 5s sleep

test_key_info_spend_values_image_generation failed once on litellm_internal_staging
(pipeline 82282) with "spend did not increase on an identical repeat image call"
(assert 0.24966 > 0.24966). The test made the second image call, slept 5s, then
read the key's spend once. Response caching is commented out in
proxy_server_config.yaml and no sibling test enables it, so the likely cause is
async/batched spend logging not having flushed the repeat call's cost within 5s,
which the build_and_test job aggravates by running every tests/test_*.py against
one shared proxy under pytest -n 4.

Poll the key's spend for up to 60s and break as soon as it grows. This removes
the timing flake while preserving the canary: if the repeat were genuinely
unbilled (for example the proxy response cache being on), spend never grows, the
poll times out, and the assertion still fails.

* test(pass_through): raise ruby assistants client request_timeout to 600s

The streaming assistants example in openai_assistants_passthrough_spec.rb hit
Net::ReadTimeout on litellm_internal_staging (pipeline 82280), failing at roughly
125s which is ruby-openai's default request_timeout of 120s. An assistants run
with the code_interpreter tool can occasionally take longer than that to stream
its first content back through the pass-through.

Raise the client's request_timeout to 600s, matching the 600s timeout the Python
pass-through e2e tests already use, so a slow-but-healthy streaming run no longer
trips the default read timeout.

…ty reads (#30683)

test_basic_vertex_ai_pass_through_with_spendlog failed intermittently on
litellm_internal_staging (pipelines 82155, 82196, 82209, 82230) with "Spend
should be greater than before after 120s". Spend logging is async and batched,
so the pass-through call's cost sometimes had not landed within the 120s poll
window; one run ended on spend_after 0.0 because the final /global/spend/logs
read returned nothing and "or 0.0" recorded that as zero spend.

Widen the poll window to 240s and skip a transient empty read instead of
treating it as 0.0, so a momentary endpoint hiccup on the last poll no longer
fails an otherwise-billed call. The spend_after > spend_before assertion is
unchanged, so a genuinely unbilled call still fails the test

…racking (#30690)

completion_cost read service_tier straight from the request optional_params
and called service_tier.lower() on it, so a non-string value (dict/int/list,
reachable via allowed_openai_params/drop_params) raised AttributeError.
_response_cost_calculator swallowed that and returned response_cost=None, so
the request's cost was silently lost.

The isinstance guard alone is not enough: a surviving dict would crash again
downstream in _get_service_tier_cost_key, which also calls .lower(). A
request-level service_tier is only meaningful for pricing when it is a concrete
billable tier string, so coerce any non-string value to None and defer to the
tier the provider reports on the response usage, the same way "auto" already
does.

Adds a regression test driving a dict service_tier through completion_cost; it
raises AttributeError before the fix and prices at the served tier after.

…orcement (#30665)

When general_settings.custom_auth is configured but custom_auth_run_common_checks
is not set, project/team/org enforcement (budgets, model-level rate limits, and
model-access lists) silently does nothing for custom-auth requests, since the
centralized common_checks gate returns early for custom auth. Emit a startup
warning pointing operators at the flag so the misconfiguration is visible instead
of failing silently.

…oding (#30600)

acquire_lock stores the pod_id through async_set_cache, which JSON-encodes
the value, so Redis holds the quoted string "<pod_id>". release_lock's Lua
compare-and-delete compared the raw pod_id, so the equality check never
matched and the lock was never deleted; it only cleared on TTL expiry. That
stalled the spend-update drain whenever the leader pod restarted, letting the
litellm_daily_*_spend_update_buffer lists grow unbounded in Redis.

Compare against json.dumps(self.pod_id) so the release matches the stored
value. The GET+DEL fallback already round-trips through async_get_cache and is
unaffected.

Co-authored-by: Claude <[email protected]>

…ck (#30695)

Several CI jobs run the proxy against a model whose api_base is a shared
"fake OpenAI endpoint" hosted on Railway
(exampleopenaiendpoint-production.up.railway.app) so the E2E runs return
canned responses without paying for or depending on a live provider. When
that single deployment is down, every one of those jobs fails with
"404 Application not found" even though nothing in the PR is broken; the
whole repo is coupled to the uptime of one free external service.

This adds tests/_fake_openai_endpoint_server.py, a small canned-response
OpenAI-shaped server (chat, text, embeddings, streaming with usage, and the
"429" rate-limit special case), and a reusable start_fake_openai_endpoint
CircleCI command that runs it on host port 8190 and waits until healthy. The
affected jobs now inject FAKE_OPENAI_API_BASE pointing at the local server,
and the example configs they mount resolve api_base from that env var. The
intentionally bad fallback URL in proxy_server_config.yaml is left untouched
so the fallback test still exercises a failing upstream.

Wired into build_and_test, litellm_router_testing,
db_migration_disable_update_check, proxy_logging_guardrails_model_info_tests,
proxy_spend_accuracy_tests, proxy_multi_instance_tests,
proxy_store_model_in_db_tests, and proxy_build_from_pip_tests.

#30704)

* feat(ui): migrate models page to App Router path route

Cut the Models + Endpoints page over from the legacy ?page=models switch
in (dashboard)/page.tsx to a path route at (dashboard)/models-and-endpoints.
Adding the MIGRATED_PAGES entry repoints the sidebar link and redirects old
?page=models bookmarks to /ui/models-and-endpoints.

ModelsAndEndpointsView already sourced identity from useAuthorized() and its
own data via useModelsInfo(), so the token/keys/modelData/setModelData props
were dead; drop them from ModelDashboardProps (and the parent's now-unused
setModelData state) to sever the last of the shared-state coupling.

* test(ui): scope migration smoke's shell probe to the exact sidebar link

The migration smoke used a loose `locator("a", { hasText: "Virtual Keys" })`
to assert the dashboard shell rendered. The Models + Endpoints page content
itself links to the "Virtual Keys page", so on that route the substring filter
matched two anchors and tripped Playwright strict mode. Match the sidebar link
by its exact accessible name instead, which resolves to just the nav item.

The `page == "pass-through-settings"` arm in (dashboard)/page.tsx is
unreachable: it isn't a sidebar item and nothing in the app sets
?page=pass-through-settings. The Pass-Through Endpoints UI lives as a tab
inside the Models + Endpoints view (ModelsAndEndpointsView renders
PassThroughSettings), so the standalone switch arm is dead code. Remove it,
its now-unused import, and the matching enum member in the e2e pages fixture.

…racking (#30706)

completion_cost extracted service_tier from the response object and the usage
object without an isinstance guard, so a non-string value (e.g. a dict) flowed
straight into _get_service_tier_cost_key and raised AttributeError on
service_tier.lower(). completion_cost re-raises, so the request's cost was lost.

PR #30690 fixed only the request-level optional_params path. This extends the
same guard to the response and usage paths by normalizing each extracted value:
a non-string tier (and the routing-only "auto" sentinel) is not billable, so it
coerces to None and pricing defers to the next concrete tier the provider served,
falling back to standard pricing when none is present.

Adds two regression tests driving a dict service_tier through completion_cost,
one on the response object (defers to the served usage tier) and one on the usage
object (prices at standard); both raise AttributeError before the fix.

…and review-gate label lifecycle (#30433)

* feat(triage): auto-close stale PRs with Greptile score <4/5

Adds .github/scripts/close_low_quality_prs.py and a daily workflow that
closes PRs which:
  - are open for at least 7 days, and
  - carry a most-recent greptile-apps review with Confidence Score <4/5,
  - and are not drafts or opt-out-labeled ('do not close', 'wip', etc.).

Each closure posts an explanatory comment telling the contributor how to
bring the PR back (rebase, re-request greptile, reopen at 4+/5). The
4/5 bar is already documented in the PR template
(.github/pull_request_template.md), so this just enforces it.

Tested with a dry run against the live BerriAI/litellm backlog of 1000
open PRs: 100 candidates identified, 598 PRs pass the bar (4+/5), 186
are too young, 97 are drafts, 19 lack any Greptile review and are left
alone.

Workflow defaults to closing 25 PRs/run as a safety net and supports
workflow_dispatch with overrides (close=false for a dry run, custom
min_age_days/min_score/limit).

18 unit tests cover score extraction (HTML/markdown/plain text, login
variants, multi-review picks latest) and per-PR evaluation (drafts,
opt-out labels, age, missing/passing/failing scores).

Co-authored-by: Mateo Wang <[email protected]>

* docs(templates): require expected/actual + QA proof for external contributions

PR template:
- Make the rubric explicit at the top: link an issue, OR provide a clear
  problem description + expected vs. actual + visual QA proof.
- Add dedicated sections for each piece so the bot has a deterministic
  shape to read.
- Keep the existing 'Linear ticket' section for internal contributors
  (they're exempt from the auto-triage rubric).

Bug report template:
- Split 'What happened?' into 'Actual behavior' + 'Expected behavior'.
- Make logs/screenshot a required textarea.
- Warning banner at the top tells external contributors that incomplete
  reports will be auto-closed (with re-evaluation on reopen).

Feature request template:
- Require a concrete use case + example in the motivation field, not just
  a one-liner pitch.
- Same auto-triage warning banner.

Co-authored-by: Mateo Wang <[email protected]>

* feat(triage): Agent Shin LLM-as-judge for external PRs and issues

Adds a new triage flow that evaluates external pull requests and issues
against the project's contribution rubric and, when configured to do so,
auto-closes non-conforming ones with an explanatory comment. Contributors
can update + reopen to be re-evaluated.

Scope:
- Internal BerriAI contributors (author_association OWNER/MEMBER/COLLABORATOR)
  and bot accounts are skipped entirely.
- 'Fixes #1234' / 'Resolves https://github.com/.../issues/N' in the PR body
  short-circuits to PASS without burning LLM tokens.
- LLM judge returns structured JSON (verdict, missing[], explanation);
  parser tolerates markdown fences and embedded JSON.
- LLM errors NEVER close PRs/issues — failure surfaces as 'skip-llm-error'.

Safety:
- pull_request_target / issues triggers are FORCED dry-run in the workflow;
  only manual workflow_dispatch with close=true (and AGENT_SHIN_ENABLED=true)
  takes destructive action.
- Default mode writes verdicts to GITHUB_STEP_SUMMARY only — no public
  comments until the team flips the AGENT_SHIN_ENABLED repo variable.
- LLM uses an OpenAI-compatible endpoint (model and base URL configurable
  via repo variables; key via OPENAI_API_KEY secret).

Files:
- .github/scripts/triage_with_llm.py   - judge orchestrator + CLI
- .github/workflows/triage_pr_with_llm.yml
- .github/workflows/triage_issue_with_llm.yml
- tests/test_litellm/test_github_triage_with_llm.py - 33 unit tests

End-to-end validated against four real PRs (#28117 internal collaborator,
#28108 bot, #28129 'Fixes #28128', #28116 no linked issue) and issue
#28132 with a stubbed LLM judge: each path produces the expected action.

Co-authored-by: Mateo Wang <[email protected]>

* feat(triage): scope Greptile auto-closer to external contributors + dry-run by default

- close_low_quality_prs.py now filters by GitHub author_association via
  the REST API: PRs from OWNER / MEMBER / COLLABORATOR (and bot accounts)
  are skipped with a new 'skip-internal' summary bucket.
- close_low_quality_prs.yml now defaults workflow_dispatch close=false,
  and ignores 'close=true' unless the new repo variable
  AGENT_SHIN_ENABLED is set to 'true'. Scheduled runs are dry-run only
  until the team flips that switch.
- Updated unit tests: one new test asserting internal authors are
  skipped, and an autouse fixture treats unspecified test PRs as
  external so the rest of the suite still exercises the close path.

Co-authored-by: Mateo Wang <[email protected]>

* fix(workflows): scheduled cron closes PRs; safe --close strip in triage

Co-authored-by: Yassin Kortam <[email protected]>

* fix(triage): scheduled cron stays dry-run; dedent prompts before interpolation

- close_low_quality_prs.yml: only workflow_dispatch with close=true (and
  AGENT_SHIN_ENABLED=true) actually closes PRs. Scheduled runs are always
  dry-run, matching the safety invariant documented for triage_pr/issue.
- triage_with_llm.py: textwrap.dedent on an f-string with multi-line
  interpolated bodies fails because the body's 2nd+ lines start at column 0,
  making the common-indent zero. Dedent the static template first, then
  .format() the title/body in.

Co-authored-by: Yassin Kortam <[email protected]>

* Fix bugs in auto-close PR triage scripts

- close_low_quality_prs.py: Treat author_association API lookup failures
  as internal (fail-safe) so transient errors don't cause internal
  contributors' PRs to be auto-closed.
- triage_with_llm.py: Update summary heading from 'Would post comment:'
  to 'Posted comment:' since this branch only runs after the comment
  has already been posted.

Co-authored-by: Yassin Kortam <[email protected]>

* feat(triage): default Agent Shin to gpt-5.4-mini with reasoning_effort=none

- Bump DEFAULT_MODEL from gpt-4o-mini to gpt-5.4-mini (more modern;
  4M total context window per OpenAI catalog, JSON-schema response
  format, function calling all supported).
- For gpt-5.x family models, pass reasoning_effort="none" via
  extra_body. gpt-5.x rejects temperature != 1 unless reasoning_effort
  is explicitly "none"; setting it lets us keep temperature=0 for
  deterministic JSON rubric judgments. extra_body works across openai
  SDK versions regardless of whether they natively type the kwarg.
- For non-gpt5 overrides (TRIAGE_MODEL=gpt-4o-mini etc.), reasoning_effort
  is not sent.
- 4 new unit tests cover: gpt-5.4-mini -> reasoning_effort=none,
  capitalized/dated gpt-5 variants -> reasoning_effort=none,
  gpt-4o-mini -> no extra_body, base_url passthrough.

Co-authored-by: Mateo Wang <[email protected]>

* fix(triage): bugbot — drop dead gh_json and fix --optout-label append-with-default

- Removed the unused gh_json helper (bugbot low-severity dead code).
- Replaced argparse `action="append", default=[...]` with default=None
  + DEFAULT_OPTOUT_LABELS fallback. The mutable-default + append combo
  silently APPENDS to the canonical defaults instead of replacing them,
  so --optout-label could not actually scope the opt-out list.
- Added tests covering both the canonical default and the
  flag-replaces-defaults behavior.

Co-authored-by: Mateo Wang <[email protected]>

* fix(triage): bugbot — tighten linked-issue regex, fail-safe author_association, fix empty TRIAGE_MODEL

Three independent bugbot findings against triage_with_llm.py:

1. LINKED_ISSUE_PATTERN included weak keywords (`see`, `ref`,
   `addresses`) so casual mentions like "See #1234 for context" were
   short-circuited to pass-linked-issue without ever calling the LLM —
   contradicting the prompt's own "a bare issue number without a closing
   keyword counts only if it's clearly the related issue (not a passing
   mention)" rubric. Limit the regex to GitHub's documented PR-closing
   keywords (fixes/fix/fixed/closes/close/closed/resolves/resolve/resolved).

2. is_internal_contributor() treated an empty/missing author_association
   as external (eligible for the destructive close path), while the sibling
   is_external_pr_author() in close_low_quality_prs.py fail-safes the same
   case as internal. Align the two so a partial/unknown GitHub response can
   never make a PR eligible for auto-close.

3. argparse `default=os.environ.get("TRIAGE_MODEL", DEFAULT_MODEL)` returns
   the empty string when GitHub Actions exposes an unset repo variable as
   an empty-string env var (the optional vars.TRIAGE_MODEL case in the
   workflow). Use `os.environ.get(...) or DEFAULT_MODEL` so empty -> default,
   matching the existing OPENAI_BASE_URL pattern.

Tests:
- Casual mentions now must fall through to the LLM (parametrized);
  added an orchestration test ensuring "See #1234" reaches the judge.
- Empty/missing author_association now fails safe (parametrized).
- Empty TRIAGE_MODEL env var falls back to DEFAULT_MODEL; explicit
  TRIAGE_MODEL is still honored.

Co-authored-by: Mateo Wang <[email protected]>

* fix(workflows): bugbot — gate Agent Shin --close on '= true' not '!= false'

The PR and issue Agent Shin workflows gated the destructive --close
flag with [ "${DISPATCH_CLOSE:-false}" != "false" ]. That pattern
treats anything other than the literal string "false" as enabling
closure — "True", "yes", "1", typos, accidental whitespace, etc.
The workflow_dispatch input UI is a 'true'/'false' choice dropdown so
the form is constrained, but the API (`gh workflow run -f close=...`)
accepts any string, and a CI cron / external invoker passing a
non-canonical truthy value would have silently enabled real
contributor PR closures.

Mirror the sibling Greptile closer's [ "${CLOSE_FLAG}" = "true" ]
pattern: only the EXACT string "true" enables --close; every other
value (including the unset/empty default) resolves to dry-run. This is
the fail-safe philosophy applied everywhere else in this PR.

Added tests/test_litellm/test_github_triage_workflows.py with two
parametrized invariants:
  1. The destructive gate uses '= "true"' for its env-var
     comparison (either bare '${ENV}' or '${ENV:-false}' form
     accepted), and never the fail-open '!= "false"' pattern.
  2. Every destructive gate is also gated on AGENT_SHIN_ENABLED being
     "true" — either by entering the close branch on '=' or by
     bailing out early on '!=' — so flipping the repo variable off is
     a true kill switch regardless of per-run inputs.

Manually verified the test fails on the buggy '!= "false"' pattern and
passes on the fix, so it would have caught the regression at PR time.

Co-authored-by: Mateo Wang <[email protected]>

* feat(triage): close any PR (incl. drafts, any age); add @agent-shin reconsider flow

Follow-up to PR #28117. Three behavior changes + one new workflow,
addressing the team's concerns on the original review:

1) Apply auto-close to ALL open PRs, not just those over a week old.

   - close_low_quality_prs.py: --min-age-days default flipped from 7 to
     0. The flag is preserved as an opt-in safety net for one-off
     backfill runs that want to spare very-young PRs, but the daily
     scheduled sweep now closes external-author PRs as soon as Greptile
     scores them <4/5.
   - close_low_quality_prs.yml: workflow_dispatch input default also
     flipped to 0; doc comments updated.

2) Apply auto-close to draft PRs too.

   - close_low_quality_prs.py: removed the skip-draft branch in
     evaluate_pr. Drafts are NOT a free pass — the team's intent is
     'open PR count == PRs internal collaborators need to action on',
     so a draft Greptile scored 2/5 still belongs in the closed bucket.
     Authors who genuinely need a long-lived draft can attach the 'wip'
     opt-out label, which is unchanged.
   - The 'skip-draft' action is gone; the 'wip' label still skips.

3) Address the 'OSS contributors cannot reopen a bot-closed PR' wrinkle.

   GitHub does NOT let an external (non-write-access) contributor
   reopen a PR that was closed by a bot or maintainer (long-standing
   limitation). The original PR's close-comments told contributors to
   'Reopen the PR — I'll re-evaluate automatically', which is broken
   for the very audience this triage targets. Two changes:

   a) Reword every close-comment (Greptile sweep + Agent Shin PR
      close + Agent Shin issue close + PR template) to recommend:
        - Open a new PR with the updated branch (primary path).
        - Or comment '@agent-shin reconsider' on the closed PR for a
          re-evaluation that, on pass, reopens the PR via the bot's
          GH_TOKEN write access.

   b) Add the @agent-shin reconsider workflow:
        - .github/workflows/triage_reconsider.yml: new
          'issue_comment'-triggered workflow. Authorizes only the
          PR/issue author or an internal collaborator
          (OWNER/MEMBER/COLLABORATOR), gated via a step output so
          unauthorized commenters never reach the destructive steps.
          Globally gated on AGENT_SHIN_ENABLED='true' (positive form,
          matching the test_github_triage_workflows guardrail
          patterns).
        - triage_with_llm.py: --reconsider mode. On a closed PR/issue,
          re-runs the LLM judge (or linked-issue regex short-circuit)
          and:
            - on pass: reopens via reopen_pr/reopen_issue + posts a
              'Re-evaluated and reopened' comment.
            - on fail: leaves closed and posts a 'still missing X'
              comment so the contributor can iterate again.
          Reconsider-on-open is a no-op ('skip-not-closed').
          Internal-author + bot-account skips still take priority over
          reconsider.

4) Greptile-on-closed-PRs question: the team asked whether Greptile can
   re-review a closed PR. Greptile's docs don't address this and we
   shouldn't promise behavior we can't verify, so the new close-comment
   wording does NOT instruct contributors to 're-request greptile on
   the closed PR'. Instead it points them at the new-PR path (which
   Greptile definitely reviews) or the @agent-shin reconsider trigger
   (which re-runs the LiteLLM-side rubric judge, not Greptile).

Tests: 93 passing (was 59).

  - test_github_close_low_quality_prs.py: replaced 'skip drafts' test
    with 'closes drafts when score is low' + 'closes brand-new PR when
    min_age=0' + 'no skip when min_age=0'. The 'skip too young'
    assertion is preserved as opt-in.
  - test_github_triage_with_llm.py: 6 new TestTriageOrchestration cases
    for reconsider mode (skip-not-closed on open, reopen on pass,
    still-failing comment on fail, linked-issue short-circuit reopen,
    skip internal author in reconsider, reopen-issue on pass) + a new
    TestCloseCommentText class that pins the user-facing 'open a new
    PR' + '@agent-shin reconsider' wording.
  - test_github_triage_workflows.py: added triage_reconsider.yml to
    the destructive-gate guardrail table; AGENT_SHIN_ENABLED is its
    own destructive gate (no separate per-run flag needed).

Co-authored-by: Mateo Wang <[email protected]>

* test(triage): pin safe behavior for curly braces in PR/issue title+body

Adds regression tests covering the bugbot high-severity finding that
str.format() would crash on user-supplied content containing { or }.
Empirically str.format() does NOT re-parse interpolated values — only
the template literal is scanned for replacement fields — so the bug
does not exist in the current code, but pinning the safe behavior
prevents a future templating change from silently reintroducing it.

Also pins the dedented prompt shape (no leading 8-space indentation on
template lines) so a future change to the build_*_prompt functions can't
silently regress the LLM judge prompt format on multi-line bodies.

Co-authored-by: Mateo Wang <[email protected]>

* fix(triage): bugbot — reconsider dry-run + bot-closed guard + rate limit

Address three Greptile/veria-ai concerns on the @agent-shin reconsider
flow:

1. **Reconsider had no dry-run path.** The previous reconsider mode
   ignored `--close` and always posted comments + reopened on a pass.
   A local operator running
   `python triage_with_llm.py --reconsider --pr N` would silently
   take destructive GitHub actions with no way to preview. Reconsider
   now honors `close=False` the same way regular triage does and
   returns `would-reopen` / `would-reconsider-still-failing` for
   step-summary rendering.

2. **Reconsider could reopen maintainer-closed PRs/issues** (Medium
   security finding from veria-ai). The workflow only checked that the
   commenter was authorized — it did NOT check that the most recent
   close was performed by Agent Shin. A contributor could comment
   `@agent-shin reconsider` on a PR a maintainer closed for non-rubric
   reasons (duplicate, security report, design rejection) and have the
   bot reopen it. Add `was_closed_by_agent_shin()` which inspects the
   issue events API for the most recent `closed` actor and only
   permits reopen when that actor matches the configured bot login
   (default `github-actions[bot]`, overridable via env). Fail-closed
   on missing events.

3. **No rate-limiting on the reconsider trigger.** Every
   `@agent-shin reconsider` comment burns CI minutes + an OpenAI API
   call. Add a 10-minute cooldown via
   `seconds_since_last_reconsider_verdict()` which greps the issue's
   comment list for the bot's own verdict marker
   (`<!-- agent-shin:reconsider-verdict -->`). Inside the window the
   triage returns `skip-rate-limited` and the LLM never runs.

Workflow update:
- `triage_reconsider.yml` now passes `--close` only when
  `AGENT_SHIN_ENABLED=true`, matching the pattern of
  `triage_pr_with_llm.yml`. The script runs in both states so the
  verdict still appears in the step summary for QA.

Tests:
- Add 5 reconsider safety tests: dry-run for pass / fail / linked-issue
  short-circuit, bot-closed-guard refusal on maintainer close,
  rate-limit refusal inside the cooldown window, and cooldown-elapsed
  acceptance.
- Add unit tests for `was_closed_by_agent_shin` (bot / maintainer /
  missing actor / env-override) and
  `seconds_since_last_reconsider_verdict` (no marker / multiple
  markers / non-bot comment with marker / bot comment without marker).
- Pin the `<!-- agent-shin:reconsider-verdict -->` marker in both
  reopen and still-failing comments — dropping it would silently
  break the cooldown.

Existing reconsider tests updated to pass `close=True` (the
production path now) + stub the new guards via
`_stub_reconsider_guards`. 112 tests pass (was 93).

Co-authored-by: Mateo Wang <[email protected]>

* feat(triage): 1-day grace period before close + SwiftWinds immediate-close bypass

- Add a 24-hour grace window between the first low-quality detection
  and the actual auto-close. The first detection posts a warning
  comment that explicitly says "You have 1 day to address this before
  this PR is auto-closed" and points the contributor at:
    * `@agent-shin reconsider` to request another look (and re-open)
    * `@greptileai` to request a fresh Greptile review — works
      even after the PR is closed
- Both `triage_with_llm.py` (LLM judge) and `close_low_quality_prs.py`
  (Greptile-score closer) share the same `<!-- agent-shin:grace-warning -->`
  HTML marker so a warning posted by either path is recognized by both.
- Add IMMEDIATE_CLOSE_LOGINS = {swiftwinds} to bypass BOTH the grace
  period AND the dry-run / AGENT_SHIN_ENABLED gating. SwiftWinds is the
  user's personal account (no push permissions to litellm) used to
  dogfood the bot; user explicitly asked: "For SwiftWinds, just close
  immediately. Faster iteration that way."
- Update the standard close comments to mention that `@greptileai`
  works even after the PR is closed.
- Add 23 new tests covering: warn-grace on first detection, skip during
  grace window, close after grace expires, SwiftWinds bypass (case
  insensitive, with close=False, no random-login false positives), the
  grace-warning text invariants, and the SwiftWinds entry in the
  IMMEDIATE_CLOSE_LOGINS constant.

Co-authored-by: Mateo Wang <[email protected]>

* fix: skip grace-period text in close comment for IMMEDIATE_CLOSE_LOGINS

For PRs from IMMEDIATE_CLOSE_LOGINS (e.g. swiftwinds), evaluate_pr
returns 'close' immediately without ever posting a grace warning, so
the close comment should not reference a 1-day grace period.

Make close_pr take a grace_period_elapsed flag, default True, and
pass False from the main loop when the close path was the
immediate-close branch.

Co-authored-by: Yassin Kortam <[email protected]>

* fix(close-low-quality-prs): report actual closes in dry-run summary

IMMEDIATE_CLOSE_LOGINS PRs are closed even when the global --close flag is
not set, but the summary used the global dry-run flag to choose between
'would close' and 'closed'. Split the count so operators can see both
actual closures and dry-run would-be closures.

Co-authored-by: Yassin Kortam <[email protected]>

* chore(triage): vendor Agent Shin (#28117) onto demo branch

Brings the Agent Shin OSS-triage scripts, workflows, issue/PR templates, and
tests from PR #28117 onto this branch so the new review-gate feature and its
end-to-end demo are self-contained and runnable in CI.

https://claude.ai/code/session_01XyyWa8t2VYmoGd6mKMEqkZ

* feat(triage): add "ready for review" label lifecycle to Agent Shin

Adds review_gate(), a state machine that keeps a `ready for review` label in
sync with whether an external PR clears BOTH gates — the LLM rubric and
Greptile's most recent confidence score:

- pass (untagged)            -> add label + "ready for review" / "all clear" comment
- pass (already tagged)      -> no-op (idempotent across re-runs)
- regress (Greptile < 4/5 or QA proof removed) -> remove label + "what's missing"
  comment, PR stays open
- recover after a regression -> "all clear again" comment + re-add the label
- fail & untagged, < 24h old -> one-time "what's missing" notice (grace window)
- fail & untagged, > 24h old -> close + comment (reopen via @agent-shin reconsider)

The label itself is the persisted state, so comments fire only on transitions
(never on every scheduled run). All side effects are gated behind --close, so
the dry-run contract matches the existing triage flow. Lifecycle comments use
hidden HTML markers and deliberately avoid the auto-close marker so they never
trip the reconsider provenance check.

Relocates the shared Greptile helpers (extract_greptile_score, SCORE_PATTERN,
GREPTILE_BOT_LOGINS, parse_iso8601) into triage_with_llm.py so the daily sweep
and the review gate read the score through one implementation, and adds the
review_gate.yml workflow (dry-run unless AGENT_SHIN_ENABLED=true) plus 18 unit
tests covering every branch and a full pass->regress->recover cycle.

https://claude.ai/code/session_01XyyWa8t2VYmoGd6mKMEqkZ

* Port review-gate feature from #28758 onto #28147 triage scripts

Adds the "ready for review" label lifecycle (originally PR #28758) on top
of #28147's refactored triage_with_llm.py. The original commit was
authored against an older snapshot of #28117 and could not be applied
cleanly, so the additions were re-applied surgically:

- New constants: READY_FOR_REVIEW_LABEL, DEFAULT_GRACE_DAYS,
  DEFAULT_MIN_GREPTILE_SCORE, READY/REGRESSED/WITHIN_GRACE markers,
  GREPTILE_BOT_LOGINS, SCORE_PATTERN, AGENT_SHIN_AUTO_CLOSE_MARKER.
- New helpers: add_label, remove_label, extract_greptile_score,
  parse_iso8601 (the latter two mirrored from close_low_quality_prs.py
  so the daily sweep and the review gate read the score through the
  same logic).
- New comment formatters: format_ready_for_review_comment,
  format_all_clear_comment, format_regression_comment,
  format_within_grace_comment.
- New entry point: review_gate() implementing the pass/regress/recover
  state machine, with the label itself acting as persisted state so
  transition comments fire only on actual transitions.
- main() learns --review-gate, --grace-days, --min-greptile-score and
  dispatches to review_gate() when the flag is set.

Verified via tests/test_litellm/test_github_review_gate.py (18 tests)
and the existing triage suites (144 more) — all 162 pass.

Co-Authored-By: Claude Opus 4.7 <[email protected]>

* agent_shin: extract shared constants/helpers; cover review_gate.yml in guardrail tests

Bug 1: `triage_with_llm.py` and `close_low_quality_prs.py` each defined
their own copies of `extract_greptile_score`, `parse_iso8601`,
`GREPTILE_BOT_LOGINS`, `SCORE_PATTERN`, `GRACE_COMMENT_MARKER`,
`GRACE_PERIOD_SECONDS`, `IMMEDIATE_CLOSE_LOGINS`, and
`AGENT_SHIN_DEFAULT_BOT_LOGIN`. The comments explicitly said the two
copies had to stay in sync, but nothing enforced it. A future change to
one (e.g. extending `SCORE_PATTERN` for a new Greptile output format)
would silently diverge from the other and the daily sweep and the LLM
judge would disagree on which PRs have low scores.

Extract these to `.github/scripts/agent_shin_shared.py` and re-export
them from each script so the existing test attribute access
(`triage_module.GRACE_COMMENT_MARKER`, etc.) keeps working without
any test changes.

Bug 2: `review_gate.yml` is a destructive workflow (close PRs, add/remove
labels, post comments) with the same gating philosophy as the others
(`AGENT_SHIN_ENABLED = "true"` + a per-run `CLOSE_FLAG = "true"`),
but it was missing from `DESTRUCTIVE_GATE_ENV` in the guardrail tests.
Add it so a future regression (e.g. flipping to `!= "false"`) is
caught by the same parameterized invariants as every other workflow.

Co-authored-by: Yassin Kortam <[email protected]>

* agent_shin: fix bug bundle (gated LLM key, author-filtered marker dedup, dedup gh/grace helpers)

Co-authored-by: Yassin Kortam <[email protected]>

* agent_shin: fix review_gate close-after-regression and case-insensitive label match

Co-authored-by: Yassin Kortam <[email protected]>

* feat(triage): add one-shot 7-day heads-up sweep for Agent Shin rollout

Adds a rollout-day workflow that comments on every open external PR/issue
that the new triage bot WOULD auto-close, giving contributors 7 days to
fix their description before any destructive action runs.

Why now: merging this PR enables Agent Shin in dry-run. The follow-up
"enact" PR (next Monday) flips the destructive paths on. Without this
heads-up, contributors would get a close-comment on day 8 with no prior
warning. The heads-up names the cutoff date, lists the rubric, calls out
each PR/issue's specific missing pieces, and explains the recovery paths
(@agent-shin reconsider for PRs, edit + reopen for issues).

Files
- .github/scripts/_agent_shin_actions.py — thin maybe_post_comment /
  maybe_close_* / maybe_add_label / etc. wrappers. Each is a single
  `if dry_run: log; return; else: call_through()` so a dry-run preview
  differs from the real run in exactly one call site per mutation. The
  call-through goes via `triage_with_llm.<name>` (module-qualified) so
  monkeypatching the underlying function in tests is reflected here.
- .github/scripts/triage_rollout_heads_up.py — the sweep. Iterates every
  open PR + issue via `gh pr list` / `gh issue list`, runs the future
  rubric (review_gate for PRs, triage(kind="issue") for issues), and
  posts the heads-up on any item that would be auto-closed. Idempotent
  via a `<!-- agent-shin:rollout-heads-up -->` marker. Defaults to dry-
  run; --close opts in to real posts. --close-on overrides the cutoff
  date (defaults to today + 7 days).
- .github/workflows/triage_rollout_heads_up.yml — one-shot workflow.
  Triggers on push to litellm_internal_staging filtered to the script
  path (fires on rollout merge) plus workflow_dispatch with a dry_run
  input that defaults to "true" for safe manual re-runs.
- tests/test_litellm/test_triage_rollout_heads_up.py — 28 unit tests
  covering: the dry-run wrappers (each maybe_* gates correctly), the
  _would_be_closed predicate for PR vs. issue results, the comment
  formatter (cutoff/rubric/marker/recovery wording), per-item dispatch
  (skip-not-open, skip-internal-author, skip-already-notified,
  skip-passing, would-post/posted), and the sweep loop end-to-end.

Local preview (no GitHub mutations):
    python3 .github/scripts/triage_rollout_heads_up.py --repo BerriAI/litellm

Real run (what the workflow does):
    python3 .github/scripts/triage_rollout_heads_up.py --repo BerriAI/litellm --close

TODO: replace the placeholder ROLLOUT_BLOG_URL with the canonical
docs URL once the litellm-docs PR ships.

Co-Authored-By: Claude Opus 4.7 <[email protected]>

* fix: gate reconsider workflow OPENAI_API_KEY + remove dead actions wrappers

- Mirror sibling Agent Shin workflows by only exposing OPENAI_API_KEY in
  triage_reconsider.yml when vars.AGENT_SHIN_ENABLED == 'true'. Previously
  the secret was unconditionally exposed, so any PR/issue author could
  trigger paid LLM calls by commenting '@agent-shin reconsider' even while
  the bot was supposed to be in dry-run.
- Remove the six unused dry-run wrappers (maybe_close_pr, maybe_close_issue,
  maybe_reopen_pr, maybe_reopen_issue, maybe_add_label, maybe_remove_label)
  from _agent_shin_actions.py — only maybe_post_comment is used by rollout
  scripts. Drop the associated tests that exercised the now-removed
  functions.

Co-authored-by: Yassin Kortam <[email protected]>

* fix: address triage script edge cases

- triage_rollout_heads_up.py: replace %-d strftime specifier (GNU-only)
  with portable day formatting so the script doesn't crash on Windows.
- close_low_quality_prs.py: skip malformed JSON lines in fetch_pr_comments
  instead of letting one bad line abort the daily sweep, matching the
  pattern in triage_with_llm._iter_paginated_json.
- triage_with_llm.py: move has_linked_issue short-circuit before
  build_pr_prompt to avoid unnecessary prompt construction on PRs that
  link an issue.

Co-authored-by: Yassin Kortam <[email protected]>

* fix(scripts): per-PR error isolation and limit grace warnings in close_low_quality_prs

- Wrap per-PR processing in try/except so a transient GitHub API failure
  on one PR no longer aborts the entire daily sweep (mirrors the pattern
  already used in triage_rollout_heads_up.py).
- Have --limit bound *all* destructive write actions (closures and grace
  warnings combined), not just closures. Prevents a backlog of newly
  failing PRs from flooding contributors with comments in a single run.

Co-authored-by: Yassin Kortam <[email protected]>

* fix(agent-shin): remove 1000-PR cap on bulk sweeps; sweep entire backlog

Both bulk-sweep scripts hardcoded `gh {pr,issue} list --limit 1000`, and gh
lists newest-first — so the OLDEST ~900 PRs and ~380 issues were silently
dropped. That's exactly the stale backlog the daily closer and one-shot
rollout heads-up exist to catch.

Extract a single `list_open_items(kind, *, repo, fields)` helper into
`agent_shin_shared.py` with `GH_LIST_ALL_LIMIT = 100_000` — a ceiling far
above any realistic open backlog so gh paginates until the queue is
exhausted. `fetch_open_prs` and `_list_open_numbers` both delegate to it,
so the limit lives in exactly one place going forward.

Verified live against BerriAI/litellm:
- `fetch_open_prs` -> 1981 PRs (was 1000)
- `_list_open_numbers(issue)` -> 1382 issues (was 1000)
- `_list_open_numbers(pr)` -> 1981 PRs (was 1000)

Adds 7 regression tests asserting the new limit is passed, the dedicated
`gh {pr,issue} list` command + fields are used per kind, bad kind raises
ValueError, and both callers delegate to the shared helper.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(agent-shin): require non-mocked end-to-end QA proof for PR pass

The PR rubric previously passed any PR with a linked issue, regardless
of whether it showed the fix actually working. Sample spot-check found
21/25 recent external PRs passing, including ones that linked an issue
but provided zero QA evidence.

Tighten the rubric so a pass now requires BOTH:

  (1) CONTEXT — a linked issue OR a clear problem description with
      expected-vs-actual behavior.
  (2) END-TO-END QA PROOF — at least one of:
      (a) screenshot(s) of the fix working,
      (b) screen recording / video,
      (c) specific commands actually run, paired with their real
          output, against the real system.

Mocked unit tests, generic 'I tested it' claims, 'all tests pass'
without output, and the linked issue itself are explicitly excluded
from QA proof.

Also add 'qa_proof_type' to the JSON schema so the per-PR report
surfaces which kind of proof (or 'none') the judge saw.

Re-sample on the same 25 recent external PRs shifts the verdict
distribution from 21 pass / 4 fail to 4 pass / 21 fail, with zero
prior-fails now passing — the stricter rule catches PRs that ship
only with unit-test claims and no real integration evidence.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(agent-shin): link blog explainer from every action-required bot comment

Adds "What's this and why am I getting it?" links to docs.litellm.ai/blog/
agent-shin-triage from the four comments contributors actually read when
something went wrong: PR close, PR grace warning, issue close, issue grace
warning. PR comments also link the rubric section directly from the
QA-proof bullet so contributors can self-serve "what counts as proof"
without pinging a maintainer.

Pins the new guarantees in tests: blog link must appear in all four
comments, and the PR close comment must continue to flag mocked-dependency
unit tests as insufficient proof.

The linked blog post is in BerriAI/litellm-docs PR #240; the URL will 404
until that lands.

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* fix(review_gate): raise sweep limit from 1000 to 100000 to match GH_LIST_ALL_LIMIT

gh lists newest-first, so capping at 1000 silently drops the oldest open
PRs — exactly the stale ones the daily sweep is meant to reconcile. Use
the same ceiling as agent_shin_shared.GH_LIST_ALL_LIMIT so the workflow
sees the entire backlog.

Co-authored-by: Yassin Kortam <[email protected]>

* Fix three Agent Shin triage edge cases

- review_gate: expire the regression-marker short-circuit after grace_days
  so PRs that were regressed and then abandoned can eventually be closed.
- review_gate: when the rubric short-circuits to pass via the linked-issue
  regex but Greptile drags the PR below the bar, replace the synthetic
  'LLM was not called' explanation with the real Greptile shortfall so
  regression / close comments are not misleading.
- triage_rollout_heads_up._comments_have_marker: drop the unused 'kind'
  parameter and filter by bot author so a contributor quoting the
  heads-up via 'Quote reply' cannot trick the idempotency check, matching
  the pattern in triage_with_llm._has_marker.

Co-authored-by: Yassin Kortam <[email protected]>

* fix: pass min_greptile_score through to ready-for-review comment text

Co-authored-by: Yassin Kortam <[email protected]>

* feat(agent-shin): warmer triage comments — bullet-train emoji, 'what you got right' section, softer 'park this for later' framing

User feedback on the auto-triage comments contributors will see:

1. Tone — the previous 'You have 1 day to address this before this PR is
   auto-closed' framing reads as an ultimatum. Replace with: 'If the
   description isn't updated in the next 1 day, I'll auto-close this PR.
   That's not us saying we don't care about the change — we want the
   open-PR list to mirror what a maintainer can act on right now, so
   contributors don't get lost in a backlog. A closed PR is a soft "park
   this for later," not a rejection. Take your time.'

2. Positive feedback — the previous comments only listed what was missing.
   Now every close + grace-warning comment opens with a 'What you got
   right:' section rendered from the judge's per-field flags. Contributors
   see a checkmark for everything they got right (linked issue, problem
   description, expected/actual, QA proof for PRs; runnable repro,
   screenshot/log, expected/actual, motivation+example for issues) before
   the gaps. The block is omitted entirely when nothing is present so
   we never render 'What you got right: (nothing).'

3. Reconsider trigger — the previous grace warning told contributors to
   comment '@agent-shin reconsider' during the grace window. They don't
   need to — the bot re-checks on every sweep. The new copy says 'just
   update the description, no need to ping me' for the grace path, and
   reserves '@agent-shin reconsider' for the post-close recovery path.

4. Bullet-train emoji — replace 👋 with 🚄 (Shinkansen, the symbol of
   Agent Shin) across every action-required comment: PR close, PR grace
   warning, issue close, issue grace warning, within-grace, Greptile-
   closer grace warning, rollout heads-up. Pinned in tests so a future
   refactor can't silently revert.

5. Greptile-post-close — the @greptileai bullet now explicitly says 'a
   low Greptile score isn't a blocker either,' since the previous copy
   buried the fact that @greptileai works after auto-close.

Comment templates updated: format_pr_close_comment,
format_issue_close_comment, format_grace_warning_pr_comment,
format_grace_warning_issue_comment, format_within_grace_comment
(triage_with_llm.py); format_grace_warning_comment
(close_low_quality_prs.py); format_heads_up_comment header
(triage_rollout_heads_up.py).

New helpers: _format_present_for_pr / _format_present_for_issue /
_format_present_block, driven off the existing per-field flags the
LLM judge already emits — no prompt change needed.

New tests pin: bullet-train emoji in every action-required comment;
'What you got right' appears with ✅ bullets when fields are present;
the block is omitted when no fields are present; 'park this for
later' / 'not a rejection' softer framing; grace warnings tell the
contributor 'no need to ping' during the grace window (reconsider is
the post-close path only).

Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]>

* feat(agent-shin): gate triage on a dogfood allowlist

Add ALLOWLIST_LOGINS to agent_shin_shared so Agent Shin only acts on the
named accounts while the set is non-empty. mateo-berri and SwiftWinds are
allowlisted for the dogfood rollout; everyone else is skipped with
skip-not-allowlisted across all four entrypoints (triage, review gate, the
daily low-quality sweep, and the rollout heads-up).

For an allowlisted author the usual internal/external classification is
bypassed, so a maintainer's own org account still gets triaged during
testing. Emptying the set lifts the restriction and restores full triage
for the public rollout. The gate is dependency-injected via an `allowlist`
parameter defaulting to the constant, so the internal/external-skip paths
stay testable.

* feat(agent-shin): tighten QA-proof and issue rubrics, ack reconsider with reactions

Reorder the end-to-end QA proof options to video, then screenshots, then
exact commands with their real output across the PR template, the LLM judge
prompts, and every contributor-facing comment, and spell out that mocked or
stubbed runs (including pytest on the repo's own unit tests, which mock the
provider, DB, and network) never count as proof. QA proof is now required of
all contributors, not just external ones.

Tighten the issue bug-report rubric to require end-to-end evidence of the bug
(the "before" half: a video, screenshot, or command paired with real output)
plus expected vs. actual behavior, drop the bias toward PASS, and collapse the
separate has_repro/has_proof flags into a single has_repro signal.

Standardize the bullet-train emoji and strip em dashes from the bot's
public-facing messages, and route issue recovery through @agent-shin
reconsider since GitHub doesn't let OSS authors reopen an issue a bot closed.

Acknowledge an @agent-shin reconsider the moment it's accepted with an eyes
reaction and a thumbs-up once the run finishes, both gated on
AGENT_SHIN_ENABLED so dry-run leaves no trace.

* fix(agent-shin): shorten auto-close grace to 2 hours and drop the instant-close bypass

Two dogfooding changes to the Agent Shin grace window. First, the warn-then-close
grace (GRACE_PERIOD_SECONDS) drops from a day to 2 hours so the "fix it before it
closes" loop can be exercised in one sitting; the constant carries a note to bump
it back up for the public rollout.

Second, remove IMMEDIATE_CLOSE_LOGINS entirely. SwiftWinds (the external dogfood
account) used to skip the grace window and close on first detection, which also
meant closing real PRs even during a scheduled dry run because the per-PR
override flipped dry_run off. It now follows the same warn-then-close path as
every other author, so a low-quality PR is warned first and only closed once the
2-hour window elapses. This also closes the Greptile finding that the sweep could
mutate real PRs while AGENT_SHIN_ENABLED was still off.

The review gate's separate age-based grace (DEFAULT_GRACE_DAYS) is left unchanged.

Regression tests pin that SwiftWinds now warns-grace instead of closing instantly,
and that a dry-run sweep over a closeable PR reports "would close" without making
any GitHub mutation.

* fix(agent-shin): gate reconsider reopen on an Agent Shin close marker

was_closed_by_agent_shin only checked that the most recent close actor was
the bot identity. That identity defaults to github-actions[bot], which is
shared by every workflow in the repo (stale/duplicate sweeps included), so a
contributor could @agent-shin reconsider an item another workflow closed and,
if the description passed the rubric, get it reopened even though Agent Shin
was never the closer.

Require a second, Agent-Shin-specific signal alongside the actor check: an
auto-close comment stamped with a hidden AGENT_SHIN_CLOSE_MARKER. Both close
paths (the grace-period close and the review-gate close) flow through
format_pr_close_comment / format_issue_close_comment, so stamping the marker
there covers every real close while leaving the grace warnings unmarked. The
guard stays fail-closed: no marker, no reopen.

This also replaces the unused AGENT_SHIN_AUTO_CLOSE_MARKER constant (a visible
phrase the guard never consulted) with the hidden marker the guard now relies
on.

* fix(agent-shin): stamp close marker on sweep closes and disclose regression deadline

The daily Greptile sweep's close comment advertised `@agent-shin reconsider`
but never stamped AGENT_SHIN_CLOSE_MARKER, so the reconsider reopen guard
(was_closed_by_agent_shin), which now also requires that marker, silently
rejected every sweep-closed PR with `skip-not-bot-closed`. Move the marker into
agent_shin_shared so both close paths share one source of truth, extract
format_close_comment so the sweep close comment is unit-testable, and stamp the
marker there.

Also disclose the grace_days deadline in the review-gate regression comment; it
promised "the PR stays open" without mentioning that a still-failing PR is
auto-closed grace_days after the notice, which would surprise contributors with
a close they were never warned about.

* fix(triage): tighten Agent Shin reconsider reopen guards

The bot-closed guard accepted any historical Agent Shin marker comment
on the thread as proof that Agent Shin owned the latest close, so a
post-reopen close by another workflow under the shared
`github-actions[bot]` identity could still satisfy the gate and let
`@agent-shin reconsider` reopen a PR that Agent Shin did not close
this cycle. `fetch_last_close_event` now also returns the latest
`closed` event timestamp, and `was_closed_by_agent_shin` requires
the most recent Agent Shin marker comment to sit at (or just before)
that timestamp, with a small skew window for clock drift between the
events and comments APIs.

In the same path the LLM verdict check used `decision != "fail"` to
choose the reopen branch, which treated a missing, empty, or typo
verdict as a pass. Reopen is destructive, so the check now requires an
explicit `decision == "pass"` and ambiguous verdicts fall through
to the "still failing" branch instead.

* style(agent-shin): black-format reconsider guard hardening

* docs(agent-shin): scope dry-run wrapper docstring to the single existing helper

The module docstring claimed it wrapped every Agent Shin mutation and
referenced post_comment/close_pr/etc., but only maybe_post_comment exists.
Describe the single helper accurately while keeping the dry-run pattern
guidance for any future wrapper.

* chore(agent-shin): defer issue/PR template changes to the rollout PR

The triage and review-gate automation is gated to the allowlisted authors
(mateo-berri, SwiftWinds) and AGENT_SHIN_ENABLED, so during this rollout it
only acts on internal PRs/issues. The issue and PR templates have no such
gate; they change for every contributor on merge and advertise that an LLM
bot auto-closes external submissions, which won't happen while the allowlist
is the sole author gate. Revert bug_report.yml, feature_request.yml, and
pull_request_template.md to base so the public-facing messaging lands with
the rollout flip instead of ahead of it. The scripts embed their own rubric
and never read these files, so triage behavior is unchanged.

* ci(agent-shin): hash-pin the openai install in privileged triage workflows

The triage workflows install the OpenAI client with `pip install
"openai>=1.40.0"`, a floating lower bound that resolves openai and its
whole transitive tree to whatever PyPI serves at run time. These jobs run
under pull_request_target with a write-scoped GITHUB_TOKEN, and the
install plus the triage run happen on every PR open regardless of the
AGENT_SHIN_ENABLED dry-run gate (that gate only withholds the LLM key and
the destructive --close path), so a compromised release would execute
during install or import while the token is in scope.

Install instead from a new .github/scripts/triage-requirements.txt that
pins openai==2.33.0 and every transitive dependency to an exact version
with sha256 hashes, via pip --require-hashes. The workflows already
sparse-checkout .github/scripts from the base repo (never fork code), so
the pinned file is trusted. Add static guardrails to
test_github_triage_workflows.py that fail if any installer workflow
reverts to a floating openai install or if the requirements file loses
its exact pins or hashes.

* ci(agent-shin): gate rollout heads-up real run behind manual dispatch

The rollout heads-up workflow fired its real `--close` sweep on every push
to litellm_internal_staging that touched the script, and exposed
OPENAI_API_KEY unconditionally, unlike every sibling triage workflow which
only exposes the key on an enabled or dispatched run. That made merging the
script post real heads-up comments (bounded only by the dogfood allowlist),
which contradicts the inert-by-default safety invariant; once the allowlist
is cleared for the public rollout, any later edit to the file would sweep
the whole open backlog with real writes.

The heads-up cannot be gated on AGENT_SHIN_ENABLED: its whole job is to warn
contributors before that flag flips on, so it has to run while the flag is
still off. Instead the automatic push trigger now stays dry-run, and the
real one-shot sweep is a deliberate manual workflow_dispatch with
dry_run=false, the sole path that adds `--close`. OPENAI_API_KEY is exposed
only on that dispatch, matching the sibling workflows.

Add static guardrails that fail if the push path regains a `--close`, if the
dispatch gate stops fail-closing on the exact string "false", or if the key
is exposed unconditionally again.

---------

Co-authored-by: Cursor Agent <[email protected]>
Co-authored-by: Mateo Wang <[email protected]>
Co-authored-by: Yassin Kortam <[email protected]>
Co-authored-by: Claude <[email protected]>
Co-authored-by: Mateo <[email protected]>

* fix(proxy): allow non-admin virtual keys to call GA Realtime WebRTC HTTP routes (#30089)

* fix(proxy): allow non-admin virtual keys to call GA Realtime WebRTC HTTP routes

Add the realtime WebRTC HTTP sub-routes (/realtime/client_secrets,
/realtime/calls and their /v1 + /openai/v1 variants) to
LiteLLMRoutes.openai_routes so is_llm_api_route() classifies them as
LLM API routes. Without this, non-admin virtual keys received
401 'Only proxy admin can be used to generate, delete, update info
for new keys/users/teams' when calling these endpoints.

Fixes #29923

* fix(proxy): validate session.model for realtime routes in model-access check

The GA Realtime WebRTC HTTP routes resolve the effective model from the
nested session.model (falling back to the top-level model), but the auth
layer's get_model_from_request() only extracted the top-level model. A
model-restricted virtual key could therefore place a disallowed model in
session.model, leave the top-level model unset, and skip can_key_call_model()
entirely - obtaining an ephemeral token for a model it is not allowed to use.

Extract session.model for the realtime client_secrets/calls routes so the
model-access check runs against the model the request will actually use.
Legitimate callers are unaffected; their permitted model still validates.

Relates to #29923

* fix(proxy): classify realtime transcription_sessions routes as LLM API routes

Add the GA Realtime WebRTC transcription_sessions HTTP routes to
openai_routes so is_llm_api_route() returns True for them, matching the
client_secrets and calls routes already fixed. These endpoints are
registered with user_api_key_auth in realtime_endpoints/endpoints.py, so
without this a non-admin virtual key calling
POST /v1/realtime/transcription_sessions would hit the admin-only 401
branch. Extends the regression test parametrization accordingly.

---------

Co-authored-by: habonlaci <[email protected]>

* feat(proxy): surface max_input_tokens/max_output_tokens on /v1/models (#30272)

* feat(proxy): surface max_input_tokens/max_output_tokens on /v1/models

* fix(proxy): degrade /v1/models gracefully when model-group lookup fails

---------

Co-authored-by: Sameer Kankute <[email protected]>

* fix: sort tiered token-cost thresholds numerically (#30375)

* fix: sort tiered token-cost thresholds numerically

_get_token_base_cost iterated input_cost_per_token_above_<N>_tokens keys with a
lexicographic sort, so for tiers whose thresholds have different digit lengths
(e.g. 90k vs 128k) a request crossing both was billed at the lower tier that
sorted first. Sort by the parsed numeric threshold instead, so the highest tier
the request actually crosses is applied.

* refactor: reuse _parse_above_token_threshold for inline threshold parse

---------

Co-authored-by: Eric (GabiDevFamily) <[email protected]>

* fix(openai): preserve cache_control for openai-compatible custom endpoints (#30387)

* fix(openai): preserve cache_control for openai-compatible custom endpoints

* fix(openai): use parsed hostname to detect real OpenAI for cache_control preservation

* fix(proxy): drain all daily-spend batches per flush cycle (#30281) (#30505)

* fix(types): prevent internal parallel_request_limiter fields from leaking to upstream providers (#30545)

* fix(types): add internal parallel_request_limiter fields to all_litellm_params to prevent forwarding to upstream providers

* test(types): add regression test for internal rate-limit fields in all_litellm_params

* fix(init): add bool type annotation to suppress_debug_info (#30531)

Module-level `suppress_debug_info = False` had no annotation, so strict
type checkers (e.g. ty) infer it as `Literal[False]`. Reassigning it to
`True` (as done in proxy_server.py and router.py) then fails with an
invalid-assignment error. Annotate it as `bool` to match every other
flag in this module.

* fix: coalesce null aggregates in update_metrics for no-spend keys (#29945)

* feat(team_endpoints): add query parameter `key_limit` to `/team/info` endpoint (#30006)

* feat(team_endpoints): Add query parameter key_limit to /team/info

* feat(team_endpoints): update schema.d.ts to include the new query parameter

* feat(team_endpoints): add tests for limitting key count in /team/info response

* feat(team_endpoints): Apply suggestions from greptile

* Set greater-than constraint on key-limit
* Fix type

* fix(router): release aiohttp connection when stream iteration ends abnormally (#30271)

* fix(router): release aiohttp connection when stream iteration ends abnormally

A streaming response that terminates with a mid-stream read timeout, a task
cancellation (client disconnect), or GeneratorExit never closed the underlying
aiohttp ClientResponse. aiohttp only auto-releases the connector slot at body
EOF, so each abnormally terminated stream permanently leaked one slot from the
shared TCPConnector pool. During a backend traffic spike the pool drains; once
exhausted every subsequent request to that host waits for a slot, times out
and surfaces as a 408, indefinitely, even after the backend recovers. Only a
proxy restart cleared the in-memory sessions, which matched the reported
symptom of a router stuck returning 408 for a healthy vLLM backend.

Close the response in a finally clause when iteration ends. On a fully read
response the connection was already released at EOF and close() is a no-op,
so keep-alive reuse for normal requests is unchanged.

Fixes #30192

* test(aiohttp): cover GeneratorExit path with a mock instead of a live socket

The previous slot-release test started a real aiohttp TCP server, which can
flake in offline CI and does not exercise this fix's code path directly.
Replace it with a dependency-injected mock that closes the stream generator
(GeneratorExit) and asserts the response is closed, covering the third
abnormal-exit path the finally block handles

* feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery (#30273)

* feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery

* refactor(proxy): move Anthropic model-list formatter into llms/anthropic/common_utils

* fix(proxy): make model_list request param optional for direct callers

* feat(dashscope): add Responses API support (#30286)

* feat(dashscope): add Responses API support

DashScope's OpenAI-compatible endpoint serves /responses, so register a
DashScopeResponsesAPIConfig that routes dashscope/* responses calls to
{api_base}/responses without rewriting the upstream model id, instead of
falling back to the chat-completions -> responses emulation pipeline.

Closes #29780

* feat(dashscope): mark responses API as not supporting native websocket

Matches the hosted_vllm/perplexity/openrouter responses configs, which all
override supports_native_websocket() to False since the OpenAI-compatible
endpoint has no native wss:// responses transport.

---------

Co-authored-by: Sameer Kankute <[email protected]>

* fix(spend-logs): preserve error_message on ProxyException failures (#30381)

* fix(spend-logs): preserve error_message on ProxyException failures

`StandardLoggingPayloadSetup.get_error_information` used
`str(original_exception)` to populate the human-readable error message
stored in `spend_logs.metadata.error_information.error_message`.

`ProxyException` (litellm/proxy/_types.py:3453) sets `self.message` in
its constructor but does NOT call `super().__init__(message)` and does
NOT define `__str__`. As a result, `str(ProxyException(...))` returns
the empty string, and every auth/budget/quota rejection was landing
in spend_logs with `error_message=""` despite a fully populated
traceback.

Operator impact: dashboard "LLM Failure" rows became untriageable —
the only way to tell a 401 from a 429 was to manually unpack the
traceback JSON via psql. Burst failure patterns (e.g. a UI session
polling with a stale token) produced 20-30 indistinguishable
`error_code=401` rows per second.

Fix: prefer the `.message` attribute (set by ProxyException and every
litellm.exceptions.* class) over `str(exc)`. The `str(exc)` fallback
is retained for non-litellm exception types, preserving prior behavior.

Test plan:
  - 2 new unit tests in tests/test_litellm/litellm_core_utils/
    test_litellm_logging.py:
    * test_get_error_information_prefers_message_attribute_over_str
    * test_get_error_information_falls_back_to_str_when_no_message_attr
  - Existing test_get_error_information_error_code_priority still passes
  - End-to-end verified: bad-key 401 now stores full
    "Authentication Error, Invalid proxy server token passed..."
    message in spend_logs.metadata.error_information.error_message

* fix(spend-logs): preserve explicit empty .message + drop dead reference

Greptile P2 on #30381. The truthiness check `if message_attr:`
silently skipped an explicit empty-string `.message` and fell
through to `str(original_exception)`. For ProxyException-shaped
objects both produce empty, so the bug was latent; for other
exception types it would inject a different string into
error_information.error_message and corrupt the signal.

Use `is not None` so an empty string survives verbatim.

Also drop the stale `See e2e/cases/11.` comment reference — that
path does not exist anywhere in the repo and confuses future
readers.

Regression test added: an exception with `.message=""` and a
non-empty `super().__init__()` arg must yield error_message == "".

* ci: retrigger workflows after base branch change to litellm_internal_staging

* fix(anthropic): strip LiteLLM-injected total_tokens from /v1/messages response (#30382)

* fix(anthropic): strip LiteLLM-injected total_tokens from /v1/messages response

The non-streaming /v1/messages response carries a LiteLLM-injected
usage.total_tokens = input_tokens + output_tokens that is not part of
the Anthropic API spec. This caused three problems:

1. Shape divergence with streaming on the same endpoint.
   message_delta.usage in the SSE path never carries total_tokens.
   Clients parsing both paths get two different schemas from one endpoint.

2. Shape divergence with upstream. Direct calls to
   https://api.anthropic.com/v1/messages return no total_tokens field,
   so clients using the official Anthropic SDK couldn't rely on it,
   and clients that did rely on the LiteLLM-injected one broke when
   bypassing the proxy.

3. Numerical misuse. total = input + output undercounts when
   cache_read_input_tokens and cache_creation_input_tokens are
   non-zero, because cache tokens are reported in their own fields.
   A 100k-token cached prompt with 1 non-cache input token + 200
   output tokens reports total_tokens = 201, off by ~99.8% from any
   reasonable definition of "total."

Fix: add _strip_total_tokens_from_anthropic_response in
litellm/proxy/anthropic_endpoints/endpoints.py and invoke it in the
success path of anthropic_response right before returning. Only mutates
dict-shaped responses; streaming (which already lacks the field) is
left untouched.

spend_logs / Prometheus continue to compute total_tokens internally
for billing — this fix only strips the field from the wire response.

Scope: only the Anthropic passthrough endpoint /v1/messages. The
OpenAI-shape /v1/chat/completions is unaffected.

* fix(anthropic): gate total_tokens strip behind flag + handle Pydantic .usage

Two P1 greptile threads on #30382:

P1 — **Backwards-incompatible removal without a feature flag**
  Stripping `usage.total_tokens` unconditionally breaks any client
  currently reading the LiteLLM-shaped non-streaming /v1/messages
  response. Per the codebase's policy (mirrors #30418), gate behind
  a new flag.

  - `litellm.strip_anthropic_total_tokens: bool = False` (default —
    backward-compat: clients keep seeing total_tokens).
  - Env override: `LITELLM_STRIP_ANTHROPIC_TOTAL_TOKENS=true`.
  - Docstring: planned to flip to True in a future major release;
    opt in early.

P1 — **Silent no-op if `result` is a Pydantic model**
  `base_process_llm_request` may return a Pydantic-style object
  whose `.usage` is a plain dict (the most common shape — e.g.
  objects wrapping raw upstream JSON). The original
  `isinstance(response, dict)` guard skipped strip on those, so
  `total_tokens` would still hit the wire. Helper now also reads
  `getattr(response, "usage", None)` and strips when that's a dict.

  Strongly-typed Pydantic `Usage` sub-models with required
  `total_tokens` fields are still skipped — those impose type
  constraints the helper doesn't try to subvert.

Tests:
- `test_strips_total_tokens_on_pydantic_model_with_dict_usage`
- `test_flag_defaults_off`
8/8 pass locally.

* fix(anthropic): drop env var for strip flag (docs CI)

Mirrors #30418's pattern (`expose_router_debug_in_errors: bool = True`,
no `os.getenv`). The `LITELLM_STRIP_ANTHROPIC_TOTAL_TOKENS` env var
introduced in the prior commit was flagged by
`tests/documentation_tests/test_env_keys.py` because the documentation
file `docs/my-website/docs/proxy/config_settings.md` lives in
`BerriAI/litellm-docs` (separate repo) and registering a new env key
requires a parallel docs PR — a friction we avoid here by exposing
the flag only as a Python attribute + `litellm_settings` config key,
both of which load through the existing proxy config plumbing without
needing the env-var registry to be updated.

No semantic change: default still False, behavior identical when set
via `litellm.strip_anthropic_total_tokens = True` or
`litellm_settings.strip_anthropic_total_tokens: true` in config.yaml.

Verified locally: env scan no longer surfaces the key; 8/8 tests pass.

* ci: retrigger workflows after base branch change to litellm_internal_staging

* fix(pricing): correct swapped input/output token costs for command-r7b-12-2024 (#30413)

* fix(pricing): correct swapped input/output token costs for command-r7b-12-2024

* test: resolve model prices JSON relative to test file for pip installs

* fix(exception-mapping): map Gemini upstream-error body code 429 to RateLimitError (#30417)

* fix(exception-mapping): map Gemini upstream-error body code 429 to RateLimitError

Some Gemini-compatible gateways (e.g. new-api) wrap a 429 rate-limit
signal from upstream inside an HTTP 500/503 envelope, with the real
code only surfaced in the JSON body:

    {"error":{"message":"...high demand...","type":"upstream_error",
              "param":"","code":429}}

Previously LiteLLM only looked at the HTTP status and mapped this to
InternalServerError, which Router treats as non-retryable for many
configs — so users got hard 500s instead of fallback/retry.

Now the Gemini/Vertex exception mapper parses error.code from the body
and routes code 429 to RateLimitError before falling through to the
HTTP-status branches. Other body codes fall through unchanged.

Tests cover:
- new-api gateway's `code:429` payload now maps to RateLimitError
- Genuine 500-body responses stay InternalServerError
- Non-JSON body strings fall through to status-code mapping unchanged

* fix(exception-mapping): scope body-code 429 promotion to 5xx envelopes

Addresses greptile P1/P2 + @Sameerlite's review on #30417. The new
elif branch was firing for any HTTP status, so a gateway response of
HTTP 400 with body {"error":{"code":429,...}} would be incorrectly
promoted to RateLimitError (retryable) instead of falling through
to BadRequestError. Same trap for 401 -> AuthenticationError.

Scoped the body-code 429 check to `500 <= status_code < 600` —
covers 500/502/503/504 (gateways wrapping upstream 429 in any 5xx
envelope) without inviting the 4xx misclassification.

Tests: parametrized table now covers 5xx (500/502/503), 4xx (400/401),
and the existing fall-through cases, asserting each maps to the
exception type that matches the HTTP status code. 50/50 pass locally.

* ci: retrigger workflows after base branch change to litellm_internal_staging

* feat(router): add expose_router_debug_in_errors flag (default True) to redact internal model_group/fallback names (#30418)

* feat(router)!: redact internal model_group/fallback names from exception messages

The Router was unconditionally appending internal config names onto
exception.message:
  - "Received Model Group=..."
  - "Available Model Group Fallbacks=..."
  - "No fallback model group found... Fallbacks={...}"
  - "context_window_fallbacks={...}"
  - Deployment-timeout messages including model_group
  - Fallback failure detail listing fallback chain

ProxyException forwards .message verbatim to clients, so gateways were
leaking their model_name / fallback wiring in every failed call.

Fix: gate all five mutation sites on a new
`litellm.expose_router_debug_in_errors` flag (default False). Set to
True to restore upstream debug behavior for local debugging.

Why: matches the redaction posture this codebase already has for
upstream model identifiers (cf. _litellm_returned_model_name) and
removes the last common error-path leak of internal model_group names.

Breaking change marker (!): if anything parses "Received Model Group="
out of client error messages, flip the flag on or migrate to the
x-litellm-* response headers instead.

Tests: 7 cases covering each of the 5 redaction sites + the flag-on
inverse path, plus a "default off" sanity check.

* test(router): cover sites 1 + 3 of expose_router_debug_in_errors gate

Addresses Greptile / codecov feedback on #30418: patch coverage was
55.6% with 4 lines uncovered in litellm/router.py. The existing tests
exercised sites 2 (ContextWindowExceededError), 4 (no-fallback-found),
and 5 (Received Model Group) — both default and flag-on. Sites 1 and 3
were declared in the PR description as covered by "site 5 also fires"
but the gate body lines for each (the `e.message +=` inside the
`if litellm.expose_router_debug_in_errors:` branch) only execute when
the flag is on AND the specific exception path is taken, which neither
existing test triggered.

Added 4 new tests (default + flag-on × 2 sites):

  - test_default_does_not_leak_deployment_timeout_debug
  - test_flag_on_leaks_deployment_timeout_debug
  - test_default_does_not_leak_content_policy_fallback_hint
  - test_flag_on_leaks_content_policy_fallback_hint

Trigger details:

  - Site 1 (litellm.Timeout in _acompletion) is reached via the
    Router-supported `mock_timeout=True` + `timeout=0.001` kwargs on
    `acompletion(...)`. Cannot embed a Timeout instance in model_list
    because Router.__init__ deep-copies it and Timeout.__reduce__ does
    not preserve the required positional args.
  - Site 3 (ContentPolicyViolationError without content_policy_fallbacks
    set, in async_function_with_fallbacks_common_utils) is reached by
    passing a `mock_response=litellm.ContentPolicyViolationError(...)`
    instance via the call-site kwarg — same deepcopy-avoidance reason.

11/11 tests pass locally. Patch coverage on litellm/router.py for this
PR's diff should now be 100%.

* chore(router): flip expose_router_debug_in_errors default to True

Addresses @Sameerlite's review on #30418 — maintain backward
compat on the wire. Redact becomes opt-in via setting the flag
to False; the historical behavior (leak internal model_group /
fallback wiring through exception messages) is preserved as the
default.

- litellm/__init__.py: default flipped to True, docstring rewritten
  with deprecation note pointing at a future flip to False (redact
  by default) in a major release.
- tests/test_litellm/test_router_exception_redaction.py: fixture
  resets to True (was False); the "off" tests now explicitly set
  False; the "default_leaks_*" tests rely on the fixture default.
  test_flag_defaults_off -> test_flag_defaults_on.
- No router.py change needed; the gate keys off the same flag,
  only the default changes.
- PR title no longer needs the breaking-change `!` marker — no
  client sees a behavior change at default settings.

11/11 pass locally.

* ci: retrigger workflows after base branch change to litellm_internal_staging

* feat(guardrails): integrate Repelloai Argus guardrail (#30465)

* feat(guardrails): add RepelloAI Argus guardrail integration (#1)

* feat(guardrails): add RepelloAI Argus guardrail integration

Add a new guardrail hook backed by RepelloAI Argus, with dashboard-managed
asset policies enforced via an asset_id and X-API-Key auth.

* fix(guardrails): harden RepelloAI Argus guardrail

- scan streaming responses on output (was bypassing the guardrail)
- log blocked verdicts as guardrail_intervened instead of success
- treat auth/config errors (401/403/404/422) as misconfiguration that
  always blocks, not a fail-open-able unreachable error
- default unreachable_fallback to fail_closed and read it directly;
  block on unknown/malformed verdicts so an API change can't silently
  disable enforcement
- type unreachable_fallback as a Literal, drop the duplicate config model,
  expose unreachable_fallback in the config schema, and stop leaking the
  raw provider response / exception strings to the client

* fix(guardrails): address RepelloAI Argus review feedback

- support ARGUS_API_KEY (with REPELLOAI_API_KEY fallback)
- make asset_id required in the config model
- normalize unreachable_fallback so only fail_open opens; block on 400 misconfig
- correct the shared unreachable_fallback field description

* docs(guardrails): add RepelloAI Argus docs page and dashboard listing

- add docs page covering config, env vars, modes, verdicts, failure semantics
- list RepelloAI Argus in the Guardrail Garden with provider/logo mappings
- add a regression test for the provider logo and display-name resolution

* fix(guardrails): keep RepelloAI asset_id optional in config model

A required asset_id leaked onto the shared LitellmParams (which inherits
RepelloAIGuardrailConfigModel), breaking validation for every other
guardrail. Keep it optional like sibling models; the guardrail __init__
still raises when asset_id is missing, which is the real enforcement.

* Add comment for last user turn scanning

* feat(guardrails): harden repelloai scanning

* feat(guardrails): expand repelloai scanning to include tool definitions

Add extraction of tool definitions and tool call arguments to the RepelloAI
guardrail scanning. Improves detection coverage by including function schemas
and parameters in the prompt sent to the guardrail service. Also captures
detailed error responses in logs and adds guardrail header to streaming responses.

* refactor(guardrails): fix and harden repelloai schema text extraction

- Fix duplicate text in _iter_schema_text: previously all dict values were
  re-queued onto the stack even after scalar/list keys were already extracted
  explicitly, causing names/descriptions to appear twice in the scanned prompt
- Extract schema key frozensets to module-level constants so they are not
  reconstructed on every call
- Change _iter_schema_text from @classmethod to @staticmethod (cls unused)
- Narrow _call_analyze stage param from str to Literal["prompt", "response"]
- Add HttpxResponse type annotation to _raise_for_config_error
- Add LLMResponseTypes annotation to async_post_call_success_hook response param

* fix(guardrails): resolve pyright type errors in repelloai guardrail

- Narrow async_handler.post return from Response|None to Response with
  explicit None guard before calling raise_for_status/json
- Fix list comprehension returning str|None by switching to explicit loop
  with isinstance guard so pyright tracks the narrowing
- Cast model_dump() result to Dict since hasattr does not narrow object
  type in pyright

* fix(guardrails/repello): include Responses API instructions field in prompt scan

The /v1/responses top-level `instructions` field was not included in
_extract_prompt_text, allowing a caller to bypass guardrail policy checks
by putting blocked content in `instructions` while keeping `input` benign.

* feat: add api_key to config model and read prompt from data dict

* fix(guardrails/repello): plug input_text and tool-call response bypass gaps

Responses API input content parts with type 'input_text' were silently
dropped by build_inspection_messages (which only handles type='text'),
allowing callers to send blocked content via that path without triggering
the pre-call scan. Fix: add _extract_input_text_parts to RepelloAIGuardrail
and call it when walking the Responses API input messages.

Post-call scanning skipped responses whose choices contained only tool_calls
or function_call (message.content=None), letting models put blocked output in
function arguments undetected. Fix: _extract_chat_completion_text now calls
_extract_tool_call_args_from_message on each choice message.

Also replace typing.Dict/List with builtin dict/list to clear TID251 strict
ruff violations introduced by this file.

* fix(guardrails/repello): scan Responses API function_call output arguments

Output items with type 'function_call' in a /v1/responses response were
skipped by _extract_responses_api_text; only 'message' items were walked.
A model could return blocked content in function_call.arguments undetected.
Now extract arguments from function_call output items before scanning.

* fix(anthropic): drop orphaned server_tool_use on multi-turn replay from generic OpenAI clients (#30486)

* fix(anthropic): drop orphaned server_tool_use on multi-turn replay from generic OpenAI clients

When an Anthropic server-side tool (web_search, id `srvtoolu_...`) is used, its
result is carried in `provider_specific_fields.web_search_results` — PRs #17746
/ #17798 restore it for callers that round-trip provider_specific_fields. A
generic OpenAI client that does NOT preserve provider_specific_fields (e.g. Open
WebUI talking to a Vertex/Anthropic model over /chat/completions) drops it on
replay and instead sends back an assistant `tool_call` + a `tool` message both
keyed to the `srvtoolu_` id. The transform then produced a bare `server_tool_use`
(with no following *_tool_result) plus a user `tool_result` for the same id —
both invalid, so the next turn 400s:

  messages.N.content.0: unexpected `tool_use_id` found in `tool_result` blocks:
  srvtoolu_... Each `tool_result` block must have a corresponding `tool_use`
  block in the previous message.

This is the commonly-reported vertex_ai symptom where Gemini works but Claude
400s on the 2nd turn of a web-search chat.

Fix (litellm/litellm_core_utils/prompt_templates/factory.py):
- convert_to_anthropic_tool_invoke: only emit a server_tool_use when its matching
  *_tool_result is available to pair with it; otherwise skip it (a bare
  server_tool_use is itself rejected).
- anthropic_messages_pt: drop a replayed `tool`/`function` message whose
  tool_call_id starts with `srvtoolu_` (a server-executed tool produces no client
  result; a user tool_result for it is invalid).

The existing reconstruction path (provider_specific_fields present, e.g. the
litellm SDK) is unchanged, as is regular client tool_use/tool_result.

Tests (tests/llm_translation/test_prompt_factory.py):
- update test_convert_to_anthropic_tool_invoke_server_tool ->
  test_convert_to_anthropic_tool_invoke_server_tool_without_result_is_dropped
- add test_anthropic_messages_pt_generic_client_drops_orphan_server_tool

Follow-up to #17746 / #17798; addresses the generic-client (no
provider_specific_fields) case of #17737.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* test(anthropic): cover the srvtoolu_ round-trip fix in the test_litellm unit suite

The regression tests added in tests/llm_translation/test_prompt_factory.py aren't
run by the coverage CI job (it runs tests/test_litellm), so the new factory.py
branches showed as uncovered (codecov patch coverage). Add equivalent focused
tests in the unit suite so both new branches are exercised there:
- convert_to_anthropic_tool_invoke drops a srvtoolu_ server_tool_use when no
  matching *_tool_result is available.
- anthropic_messages_pt drops the orphaned srvtoolu_ tool message a generic
  OpenAI client replays.

Refs #17737

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* test(anthropic): cover the server_tool_use + result valid-pair path in unit suite

Covers the remaining patch-coverage lines codecov flagged: convert_to_anthropic_tool_invoke
emitting server_tool_use followed by its web_search_tool_result when the matching
result is present (the litellm-SDK round-trip path). Refs #17737

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* style(anthropic): flatten srvtoolu_ tool-message guard to a negated if

Addresses the Greptile style nit: replace the if-pass/else with a single negated
`if not (...)` guard around the tool_result append. Behavior unchanged. Refs #17737

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>

* fix(proxy): require premium only when enabling premium metadata fields (#30285) (#30506)

Co-authored-by: Sameer Kankute <[email protected]>

* fix(perplexity): stop double-billing reasoning tokens in manual cost fallback (#30488)

* fix(perplexity): stop double-billing reasoning tokens in manual cost fallback

When perplexity_cost_per_token cannot use the API-provided usage.cost.total_cost short-circuit and falls back to manual calculation, it multiplies the full usage.completion_tokens by output_cost_per_token and then adds reasoning_tokens * output_cost_per_reasoning_token on top. Per the OpenAI/Perplexity usage convention codified for the central path in PR #18607, completion_tokens already INCLUDES reasoning_tokens, so the manual fallback double-bills reasoning at both the output and reasoning rate.

Concrete impact on perplexity/sonar-deep-research (input 2e-6, output 8e-6, reasoning 3e-6): for the exact usage shape exercised by the live response fixture in tests/llm_translation/test_perplexity_reasoning.py (prompt_tokens=9, completion_tokens=20, reasoning_tokens=15) the current code charges 0.000223 vs the convention-correct 0.000103, a 2.165x overcharge. The bug is reachable whenever Perplexity omits the cost object (streaming chunks, fixture-driven paths, older API versions).

Subtracts reasoning_tokens (clamped at zero) from completion_tokens before applying the output rate, mirroring how dashscope/cost_calculator.py and the central generic_cost_per_token already handle it. Preserves the existing fallback behaviour when output_cost_per_reasoning_token is unset (all completion_tokens stay at the output rate).

Existing tests in tests/test_litellm/llms/perplexity/test_perplexity_cost_calculator.py asserted the buggy math and are updated to the convention-correct math. Adds a focused regression test using the exact usage shape from the live response fixture so this class of bug cannot be silently reintroduced.

* style(perplexity): drop redundant type annotation on else branch to satisfy mypy

mypy [no-redef] flagged 'completion_cost' as declared in both if and else arms; keeping the annotation only on the first declaration matches existing patterns in this file.

* fix(perplexity): update integration test expected costs for non-double-billed math

Three tests in test_perplexity_integration.py asserted the old buggy expectation
that reasoning_tokens are billed in addition to the full completion_tokens
count. After the fix in cost_per_token, reasoning_tokens are billed at the
reasoning rate and the remaining (completion_tokens - reasoning_tokens) at the
standard output rate, matching OpenAI/Perplexity convention (PR #18607).

Updates: test_end_to_end_cost_calculation_with_transformation,
test_main_cost_calculator_integration, test_high_volume_cost_calculation.
The high-volume sanity threshold drops to 0.25 to reflect the corrected total.

* fix(ui): use dynamic proxy base URL in MCP usage examples (#30487)

Replace hardcoded http://localhost:4000 with getProxyBaseUrl() in the
MCP server usage example and copy-to-clipboard snippet so the generated
configuration works for non-local deployments.

Fixes #30466

* feat: add missing UK PII entity types to Presidio guardrail (#30537)

* feat: add missing UK PII entity types to Presidio guardrail

Add UK_PASSPORT, UK_POSTCODE, and UK_VEHICLE_REGISTRATION to PiiEntityType enum and PII_ENTITY_CATEGORIES_MAP. These entity types are supported by Microsoft Presidio but were missing from litellm's type definitions, preventing users from configuring UK-specific PII detection.

* test: remove fragile hardcoded entity count test

Remove test_uk_category_entity_count which hardcodes len() == 5. The test_uk_entities_match_presidio_recognizers test already verifies exact set equality, making the count test redundant and fragile to future Presidio additions.

* style: apply Black formatting to match CI requirements

* fix: route volcengine (Doubao) tiered-pricing models to the tiered cost handler (#30357)

Volcengine (Doubao) models define `tiered_pricing` but no flat per-token cost, so cost_per_token fell through to generic_cost_per_token (which only reads flat costs) and tracked them at $0

Route custom_llm_provider == "volcengine" to the shared tiered-pricing handler in litellm/llms/dashscope/cost_calculator.py, which already computes graduated tier costs. Make that handler provider-agnostic by adding a custom_llm_provider argument (default "dashscope" preserves existing behavior) so get_model_info resolves the correct model map entry

Fixes #30346

* feat(mcp): make MCP gateway name and description configurable via env vars (#30473)

* feat(mcp): make MCP gateway name and description configurable via env vars

* Rename function _restore_env to _apply_env

* docs(mcp): document import-time capture of env-backed identity constants

Address Greptile review feedback: clarify that LITELLM_MCP_SERVER_NAME and
LITELLM_MCP_SERVER_DESCRIPTION are read once at import and require a module
reload to observe env changes after import.

Generated with AI assistance

Co-Authored-By: Claude <[email protected]>

---------

Co-authored-by: Yevhen Luhovtsov <[email protected]>
Co-authored-by: Claude <[email protected]>

* fix(mcp): preserve native tools in semantic filter hook (#26650)

* fix(mcp): preserve native tools in semantic filter hook

The SemanticToolFilterHook.async_pre_call_hook passed ALL tools (MCP +
native) to filter_tools(), which only knows MCP-registered tool names.
Native tools silently failed the name match in _get_tools_by_names()
and were dropped from the request.

Fix: partition tools into native and MCP-registered before filtering.
Run the semantic filter only on MCP tools, then merge native tools
back unconditionally.

Changes:
- Robust _is_mcp_tool() using shape-based detection for OpenAI-format
  dicts, safe regardless of future _extract_tool_info changes
- Single-pass partition loop (no double _is_mcp_tool calls)
- Preserve native tools in MCP expansion path (mixed requests)
- Track MCP expansion to prevent expanded tools bypassing filtering
- filter_stats reports MCP-only counts for accurate metrics
- Extracted _emit_filter_metadata() helper
- Skip spurious filter headers for all-native tool requests

Closes #26212

* remove stale docstring note referencing tools_expanded_from_mcp

* fix: handle Responses API name collision and preserve tool ordering

- Classify Responses API tools ({type: 'function', name: '...'}) as
  native to prevent name collisions with MCP canonical names
- Preserve original request tool ordering using id()-based merge
  instead of naive native+mcp concatenation
- Add 2 regression tests: name collision and ordering preservation

* style: apply black formatting

* fix(mcp): harden semantic filter — preserve all native tool formats, safe metadata access, graceful expansion failure, name-based merge

* lint: suppress PLR0915 on async_pre_call_hook (matches codebase convention)

* ci: retrigger checks after rebase onto litellm_internal_staging

* feat(fireworks): sync Fireworks AI model registry with current platform catalog (#30616)

Adds 12 new Fireworks serverless models and updates 3 existing entries in
model_prices_and_context_window.json and its bundled backup to match the
current Fireworks platform model list. New direct models: glm-5p2,
qwen3p7-plus, minimax-m3, minimax-m2p7, kimi-k2p7-code, kimi-k2p6,
deepseek-v4-pro, deepseek-v4-flash. New router endpoints: glm-5p1-fast,
kimi-k2p6-fast, kimi-k2p7-code-fast. Updated: glm-5p1, gpt-oss-120b, and
gpt-oss-20b now carry correct output token caps, cache-read pricing, and
explicit capability flags

max_tokens is set equal to max_output_tokens (not the full context window)
for models whose generation cap is below their context window. This avoids
the shared input+output budget path in get_modified_max_tokens, which would
otherwise let callers request output sizes the model cannot produce. The
same fix corrects the pre-existing glm-5p1, gpt-oss-120b, and gpt-oss-20b
entries that had max_tokens equal to the full context window

Short-form aliases (fireworks_ai/<model>) are added for every direct
accounts/fireworks/models/ entry so cost attribution works for callers
using bare model names. Router endpoints get short-form aliases too, and
transform_request now routes bare names ending in -fast to the
accounts/fireworks/routers/ path instead of defaulting every bare name to
models/. This keeps the kimi-k2p6-fast router from being misrouted to the
nonexistent models/kimi-k2p6-fast endpoint

kimi-k2p6-turbo is intentionally excluded; kimi-k2p6-fast is its
replacement. Context windows for deepseek-v4 and kimi models use the
power-of-two values (1048576 and 262144) published on the Fireworks model
pages, matching the convention already used by existing entries

Two regression tests in test_utils.py assert the exact per-token costs,
token limits, capability flags, and short-form-to-long-form equality for
all 15 models against both the main and backup cost maps. Two routing
tests in test_fireworks_ai_chat_transformation.py verify bare -fast names
route to routers/ and bare direct-model names route to models/

* fix(bedrock): handle role:"system" inside the messages array on /v1/messages (#29698) (#30443)

* feat(anthropic): hoist leading in-array system to top-level (helper)

* test(anthropic): cover _system_content_to_blocks edge cases; deepcopy cache_control

* test(anthropic): mid-conversation system normalization cases

* feat: add supports_mid_conversation_system flag to Claude Opus 4.8

Add supports_mid_conversation_system: true to all 9 claude-opus-4-8 cost-map
entries (Anthropic-native, Bedrock, Vertex, Azure AI) in both the root cost
map and the bundled package backup, since the runtime helper and tests read
the backup in local/offline mode.

Pin the mid-system passthrough regression test to the local cost map via the
existing local_model_cost_map fixture so it reads the branch-local flag rather
than the network-fetched main copy.

* fix(bedrock): normalize in-array system in /v1/messages handler (#29698)

Wire normalize_system_messages_for_anthropic into anthropic_messages_handler
so all Bedrock /v1/messages paths (Invoke / Mantle / ClaudePlatform /
Converse-bridge) hoist leading in-array system entries (and demote
mid-conversation ones on models lacking supports_mid_conversation_system) into
the top-level system field. The normalized messages/system are written back
into the local_vars snapshot the base_llm branch reads from, otherwise the
Invoke/Mantle fix would silently no-op.

Also fix the helper to resolve supports_mid_conversation_system through the
prefix-aware AnthropicModelInfo._supports_model_capability resolver. The raw
_supports_factory could not see the flag once get_llm_provider left the
invoke/ prefix on the model id, which would have wrongly demoted
mid-conversation system on a Bedrock invoke opus-4-8 path.

* fix(bedrock): resolve mid-conversation-system flag through mantle/invoke/converse route prefixes; drop unused param

* fix(types): widen system param to Union[str, List] for hoisted system blocks

* refactor(bedrock): drop dead local_vars messages writeback

* fix(bedrock/converse): translate in-array system in anthropic->openai adapter (#29698)

* fix(bedrock/converse): preserve cache_control on in-array system; test drop-empty

* fix(bedrock/converse): rename colliding local to satisfy mypy; test handler system-merge branches

* fix(types): register supports_mid_conversation_system in model-info schema

The cost-map JSON-schema validation test (test_aaamodel_prices_and_context_window_json_is_valid)
rejects unknown properties, so adding supports_mid_conversation_system to the opus-4-8
cost-map entries failed CI with 'Additional properties are not allowed'. Register the flag
in the INTENDED_SCHEMA allow-list and in the ProviderSpecificModelInfo TypedDict so it is a
typed, first-class capability flag alongside its peers (supports_output_config, etc.).

---------

Co-authored-by: Sameer Kankute <[email protected]>

* fix(bedrock/agentcore): optionally forward multimodal content blocks in InvokeAgentRuntime payload (#28885)

* fix(bedrock/agentcore): optionally forward multimodal content blocks in InvokeAgentRuntime payload

By default the agentcore provider flattens the last message to a text-only
{"prompt": "..."} payload via convert_content_list_to_str, silently dropping
OpenAI multimodal blocks (image_url, file, input_audio, ...).

This adds an opt-in `forward_multimodal_content` litellm param. When truthy and
the last message's content is a list containing a non-text block, the original
OpenAI content list is forwarded verbatim under a new "content" field so an
attachment-aware AgentCore agent can read it. Default off keeps the payload
byte-identical to the legacy {"prompt": "..."} shape — existing agents are
unaffected.

The flag is read from optional_params (where other AgentCore params land) with a
litellm_params fallback, and accepts a bool or a config/env string ('true', '1', ...).

AgentCore Runtime is schemaless on the agent side — the agent's @app.entrypoint
parses arbitrary JSON up to 100 MB (per
https://docs.aws.amazon.com/bedrock-agentcore/latest/devguide/runtime-invoke-agent.html),
so this is a purely upstream change; no AgentCore-side schema is asserted.

* fix(bedrock/agentcore): shallow-copy forwarded multimodal content list

Address review feedback (Sameerlite): payload["content"] = last_content
aliased the caller's mutable messages[-1]["content"] list. Harmless today
because the payload is JSON-serialized immediately, but a latent footgun if
a future caller mutates the returned payload before serialization. Forward
list(last_content) so the payload owns its own list. Block dicts stay shared
on purpose — a deep copy would clone potentially large base64 media on the
request hot path, and the flagged risk was the shared list, not the blocks.

Update the passthrough tests to assert equality + distinct identity, and add
a regression test that mutating the payload list can't leak back into the
original message content.

* Revert "fix(mcp): preserve native tools in semantic filter hook (#26650)"

This reverts commit 438c825.

* Revert "feat(guardrails): integrate Repelloai Argus guardrail (#30465)"

This reverts commit 54da785.

* Revert "feat(dashscope): add Responses API support (#30286)"

This reverts commit 6766256.

* Revert "fix(bedrock): handle role:"system" inside the messages array on /v1/messages (#29698) (#30443)"

This reverts commit b8a8083.

* Revert "fix(anthropic): drop orphaned server_tool_use on multi-turn replay from generic OpenAI clients (#30486)"

This reverts commit 6e9c0b0.

* Revert "fix: route volcengine (Doubao) tiered-pricing models to the tiered cost handler (#30357)"

This reverts commit 172e302.

* Revert "feat(proxy): serve Anthropic-native /v1/models for Claude Code gateway discovery (#30273)"

This reverts commit 4e31885.

* fix: pass key_limit=None in team_member_update and patch model_cost in pricing test

team_member_update called team_info without key_limit, so the fastapi.Query
default object (not None) was passed through to get_data, which failed when
serializing it. Pass key_limit=None explicitly to avoid this.

test_get_model_info_costs patched litellm.model_cost from the local backup so
the assertion holds before the PR is merged and the remote main URL is updated.

* fix(security): validate resolved model in /realtime/client_secrets for non-transcription sessions (#30710)

Omitting both model and session.model caused the endpoint to default to
gpt-4o-realtime-preview without running can_key_call_resolved_model, so
any key could access that model regardless of its allowed-model list.

The transcription path already called can_key_call_resolved_model; this
adds the same call for the realtime path before returning.

* fix(lint): fix F821 undefined model_info and F841 unused metadata in create_model_info_response

* fix: black formatting and stub get_model_group_info in third team translation test

* fix: reformat utils.py with black 26.3.1 to match CI

* fix: replace Optional[X] with X | None to satisfy UP045 ruff strict gate

---------

Co-authored-by: Habon Laszlo <[email protected]>
Co-authored-by: habonlaci <[email protected]>
Co-authored-by: Armaan Sandhu <[email protected]>
Co-authored-by: santino18727-debug <[email protected]>
Co-authored-by: Eric (GabiDevFamily) <[email protected]>
Co-authored-by: Nitish Agarwal <[email protected]>
Co-authored-by: jho1-godaddy <[email protected]>
Co-authored-by: 安妮的心动录 <[email protected]>
Co-authored-by: Harshith Gujjeti <[email protected]>
Co-authored-by: Tomoya Tabuchi <[email protected]>
Co-authored-by: Vedant Agarwal <[email protected]>
Co-authored-by: Prathamesh Jadhav <[email protected]>
Co-authored-by: songkuan-zheng <[email protected]>
Co-authored-by: Kropiunig <[email protected]>
Co-authored-by: Lavish Bansal <[email protected]>
Co-authored-by: Shane Emmons <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Co-authored-by: Anuj ojha <[email protected]>
Co-authored-by: Nahrin <[email protected]>
Co-authored-by: Nbouyaa <[email protected]>
Co-authored-by: Vineeth Sai <[email protected]>
Co-authored-by: Eugene Lugovtsov <[email protected]>
Co-authored-by: Yevhen Luhovtsov <[email protected]>
Co-authored-by: Ayush Shekhar <[email protected]>
Co-authored-by: Ahmad Shahzad <[email protected]>
Co-authored-by: Kent <[email protected]>
Co-authored-by: Jón Levy <[email protected]>

)

* feat(search): add TinyFish as search provider

Adds TinyFish web search (GET https://api.search.tinyfish.ai) as the
16th search provider in LiteLLM. Follows the BaseSearchConfig pattern
used by other GET-based providers like Brave.

Includes unit tests in tests/test_litellm/ for full patch coverage.

* fix(search/tinyfish): use concrete types to pass any-discipline and ruff UP006/UP045

Replace typing.Dict/List/Optional/Union with modern syntax (dict, list,
X | None) and use concrete type parameters (dict[str, str] for headers,
dict[str, object] for params) to eliminate LIT009 Any-discipline
violations. Move _append_domain_filters to module level to avoid leaking
Any through self.

* fix(search/tinyfish): eliminate Any-typed values for any-discipline gate

Use Pydantic BaseModel and TypeAdapter at httpx/base-class boundaries
to validate untyped inputs (json(), params.get(), bare set). Three
genuine external boundaries annotated with any-ok.

* style: fix black formatting for long line

* fix(search/tinyfish): move any-ok comment to violation line for any-discipline gate

The any-discipline checker matches `# any-ok` comments by line number.
The comment was on the closing-paren line (127) but the violation was
on the call-expression line (126), so the suppression did not apply.

* fix(search/tinyfish): align with approved PR #30158

Drop explicit AND from domain filter query to match the approved
implementation. Set pricing to zero. Rename test to match behavior.

Cut the legacy "Old Usage" report (?page=usage) over from the switch in
(dashboard)/page.tsx to a path route at (dashboard)/old-usage. The segment is
old-usage rather than usage because the modern usage dashboard (new_usage)
already owns /usage. Adding the MIGRATED_PAGES entry repoints the sidebar item
and redirects existing ?page=usage links to /ui/old-usage.

The report was the switch's catch-all else, so removing it means choosing a new
fallback: collapse the now-redundant explicit api-keys arm into the else so the
main dashboard (UserDashboard) is the default. Unknown ?page= values now land on
the dashboard instead of the Old Usage report, which is the sensible default.

The new route sources identity from useAuthorized() and passes keys={null}: the
key-filter dropdown read the parent's keys state, which was already empty on
direct navigation to ?page=usage, so this preserves that rather than wiring a
paginated key fetch into a deprecated report.

…ross-pod counter is unreliable (#30684)

Budget enforcement reads spend from the cross-pod Redis counter via get_current_spend, which trusted the counter whenever Redis returned a value. A Redis instance that restarts and reloads an older RDB snapshot (the customer's logs repeat "Redis is loading the dataset in memory") comes back with a stale-low counter; that read is a hit, not a clean miss, so the existing DB reseed never ran and a key kept getting admitted even though its recorded spend was already over max_budget. The symptom was recorded spend sitting above the limit while requests kept succeeding.

Read-time enforcement: get_current_spend takes an optional max_budget and, when the counter would admit the request but reads below this caller's last-known recorded spend, re-reads the authoritative spend and enforces against the higher value. The authoritative source depends on the counter: key/team/user/org/team-member read the DB row, per-window budgets aggregate spend logs, and end-user/tag have no DB row so the caller's freshly-loaded recorded spend is used. Healthy primary counters and freshly reset keys stay off the DB path, and the value is cached in-process for a few seconds, so a persistently stale counter drives at most one read per counter per window. When the DB value is higher, the counter is repaired with a monotonic, atomic set-max (RedisCache.async_set_max) so every worker reads the corrected total and a concurrent increment is never clobbered.

Reconcile no longer fails open: when the post-call reservation reconcile found the counter missing or an adjustment that would drive it negative, it deleted the counter and continued (the deletion is what left counters nil/unenforced after a Redis reload). It now reseeds from the DB's lagging authoritative floor instead of deleting; the monotonic set-max can only raise a stale-low counter, and the read-time floor converges to the true total as the spend buffer flushes. The pre-call admission resize path keeps its original fail-closed behavior.

Opt-in strict enforcement: general_settings.fail_closed_budget_enforcement (default False) makes the authoritative re-check run for every budgeted entity (closing the gap where a stale-low counter and a stale-low cached fallback would otherwise both pass the cheap guard), and rejects a request with 503 when the spend backing an admit decision can be verified against neither Redis nor the database. Default behavior is unchanged; the re-check stays bounded by the in-process cache.

Resolves LIT-3772

Drop the two Agent Shin workflows that ran on the pull_request_target
trigger: the PR triage workflow and the review gate. Both were dry-run
and gated behind AGENT_SHIN_ENABLED, so no live automation changes.

The shared scripts under .github/scripts stay in place; four other
Agent Shin workflows still depend on them and run on schedule, dispatch,
and issue events rather than pull_request_target

* fix(proxy): bump health-check max_tokens default to 16 for GPT-5 compatibility (#30708)

OpenAI GPT-5 models require max_completion_tokens >= 16.
Health checks were using 5 (proxy/health_check.py) and 10
(health_check_helpers.py), causing failures on GPT-5 models.

Fixes #23836

* fix: increase health check max_tokens from 5 to 16 (#23836) (#26610)

GPT-5 models enforce a minimum of 16 for max_output_tokens. The current
default of 5 still causes health checks to fail for these models. Bump
the non-wildcard default to 16 — the smallest value that satisfies all
known provider minimums while keeping health checks lightweight.

Also tightens the wildcard test assertion from a weak disjunctive check
to strict key-absence.

Co-authored-by: Sameer Kankute <[email protected]>

* fix: ensure checks show gemini-3-flash-preview supports responseJsonS… (#30696)

* fix: ensure checks show gemini-3-flash-preview supports responseJsonSchema.

* fix: remove async keyword from test.

* fix: make Bedrock Mantle Responses routing data-driven per model (#30700)

* Make Bedrock Mantle Responses routing data-driven per model

Route Bedrock Mantle models to the native Responses API based on each
model's price-map capability signal instead of a hardcoded model-name
heuristic, and derive the OpenAI-compatible base path segment per model.

Responses dispatch now selects the native config when the model advertises
responses support (/v1/responses in supported_endpoints, or mode=responses),
both overridable via register_model and proxy model_info. This enables
native Responses for gpt-oss-120b/20b and the gemma-4 family while keeping
chat-only models (gpt-oss safeguard, nvidia, mistral, ...) on the existing
chat-completions emulation. Capability is per-model, so gpt-oss-120b routes
natively while gpt-oss-safeguard-120b does not despite sharing the gpt-oss
substring.

The wire path is a separate concern, driven by the existing
use_openai_responses_path flag rather than a model-name match: gpt-5.x and
gemma-4-* on /openai/v1, everything else (incl. gpt-oss) on /v1. The chat
config now derives its base from the same flag, fixing gemma-4
chat-completions requests that previously went to /v1 instead of /openai/v1.

Cost maps: add supported_endpoints to the gpt-oss entries (responses for the
non-safeguard variants, chat-only for safeguard) and supported_endpoints +
use_openai_responses_path to all three gemma-4 entries.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

* Address review: move capability helper into bedrock_mantle package

Move the Responses capability check out of utils.py into
litellm/llms/bedrock_mantle/common_utils.py as mantle_supports_responses,
alongside its companion wire-path helper mantle_base_segment. Both are now
pure functions of (model, model_cost): the price-map mode/supported_endpoints
read replaces the get_model_info call, so the rules are unit-testable without
patching global state and the Bedrock Mantle package is self-contained.

Use str | None instead of Optional[str] on the new signatures to satisfy the
ruff UP045 strict-rule gate. Add direct unit tests for both helpers.

Fix test_register_model_restore_undoes_existing_key_overwrite: gpt-oss-120b
now legitimately supports Responses, so it can no longer be the
"None after restore" vehicle; use the chat-only safeguard variant, which
isolates the register/restore effect from the model's own capability.

Co-Authored-By: Claude Opus 4.8 (1M context) <[email protected]>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Co-authored-by: Sameer Kankute <[email protected]>

* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (#30366)

* fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup

LiteLLM's Prisma datasource is pinned to provider = 'postgresql', so a sqlite:// or mysql:// DATABASE_URL can never connect.

Today that surfaces as an opaque startup stall where the port never binds, and a separate 'DB not connected' 500 on /key/generate when no DATABASE_URL is set at all leaves operators guessing what to configure.

Validate the DATABASE_URL / DIRECT_URL scheme in run_server before any Prisma call and exit with an actionable message naming the unsupported scheme.

Also reword CommonProxyErrors.db_not_connected_error to tell the operator to set DATABASE_URL to a postgresql:// connection string.

Add regression tests covering postgres acceptance and sqlite/mysql/mssql rejection.

* fix: resolve CI failures and proxy DB URL typing issue

* fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (#30653)

The tiered cost calculator resolved a tier's per-token cost with
`tier.get(cost_key) or tier.get(fallback_cost_key, 0)`. Because `or`
short-circuits on any falsy value, a tier that legitimately prices a
component at 0.0 (e.g. a free-cache-read tier with
cache_read_input_token_cost: 0.0, or a free-reasoning tier) is treated
as missing and silently billed at the full fallback rate
(input_cost_per_token / output_cost_per_token).

The flat-pricing path in the same module already handles this correctly
with an `is None` guard. Resolve tier costs through a small helper that
mirrors it, so 0.0 is honored at both the in-range and overflow sites.

No shipped model currently has a 0.0 tier cost, so this is a latent
defect; the fix makes the tiered path consistent with the flat path and
prevents over-charging the first time such a tier appears. Adds unit
tests covering the in-range and overflow paths, and drops an unused
import flagged by ruff in the touched test file.

* feat(proxy): show session-aggregate cost and duration in request logs (#25708) (#30507)

* fix(anthropic): don't leak tool 'type' into OpenAI function parameters schema (#30618)

In the messages->chat/completions bridge, translate_anthropic_tools_to_openai
merged every non-mapped tool key into the function parameters dict. The
Anthropic tool 'type' (e.g. 'custom') thus overwrote parameters.type ('object'
-> 'custom'), and providers reject it ('custom' is not a valid JSON-Schema type).
Exclude 'type' from the passthrough. Fixes #30557.

* fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183)

An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the
running query-engine and spawns a new one. That planned kill was
indistinguishable from a crash, and three reconnect paths used two
uncoordinated locks, so a single refresh triggered a cascade of engine
kill/respawn cycles:

  1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old
     engine, spawn new one.
  2. The engine-death watcher sees that kill, assumes a crash, and calls
     `attempt_db_reconnect(force=True)` (a different lock,
     `_db_reconnect_lock`) -> recreate again -> kills the fresh engine.
  3. In-flight queries failing during the swap are classified as transport
     errors and trigger their own `attempt_db_reconnect` -> recreate again.

Fix coordinates planned restarts across the wrapper and the watcher:

  - PrismaWrapper records the old engine PID in `_expected_engine_deaths`
    before killing it; all four watcher death-detectors (waitpid thread,
    pidfd, already-dead probe, os.kill poll) consume that PID and skip the
    reconnect instead of treating it as a crash.
  - `recreate_prisma_client` now serializes through `_reconnection_lock` and
    bumps a monotonic `_engine_generation`. Callers pass `expected_generation`
    as an optimistic-lock token, so racing/cascading recreates collapse into a
    single restart (losers no-op). This closes the two-lock gap.
  - The direct reconnect path probes the writer with SELECT 1 before
    recreating; a healthy connection (e.g. engine already replaced by a
    refresh) skips the recreate entirely.
  - `_safe_refresh_token` coalesces: it skips when the current token still has
    more than the refresh buffer of runway, so stacked triggers (proactive
    loop + __getattr__ fallback) don't each restart the engine. An
    `on_engine_replaced` hook re-arms the watcher on the new PID.

RoutingPrismaWrapper forwards `expected_generation` and skips recreating the
reader when the writer recreate was skipped.

* feat(bedrock): support file content retrieval for batch output files (#30595)

Implements transform_file_content_request and transform_file_content_response
in BedrockFilesConfig so GET /v1/files/{id}/content works for Bedrock batch
files. The request transform resolves the file id (direct s3:// URI or base64
unified id) to its S3 object, validates bucket and key prefix against the
server-configured bucket, and SigV4-signs an S3 GetObject using the same
credential and region resolution as the existing upload path. The credential
and region params are validated into a typed model at the boundary, so the only
untyped values left are the botocore signing primitives.

Also fixes the proxy managed-files path: CredentialLiteLLMParams now carries
s3_bucket_name (previously dropped when building deployment credentials) and
the managed-files hook passes the deployment credential snapshot when routing
afile_content, so unified-id content retrieval works with per-model bucket
config instead of only the AWS_S3_BUCKET_NAME env var.

Preserves managed-file access control: the proxy file-content endpoint now
rejects raw cloud-storage ids (s3://, gs://), which would otherwise skip the
owner/team check that only runs for unified ids and let a caller read another
tenant's batch output by its object key. Managed outputs are reachable only
through their unified file id. The afile_content "not found" error now reports
the caller's unified id rather than the resolved internal S3 URI.

Fixes #16186, #15563

* fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (#30646)

* fix(oci): map Cohere tool array/object params to lowercase builtins

OCI's Cohere backend returns HTTP 500 on a tool parameter typed as a bare
"List", which is what OCI_JSON_TO_PYTHON_TYPES produced for JSON-schema
arrays. MLflow {{trace}} judges trip this: their tools (get_root_span,
get_span) take an attributes_to_fetch array. The lowercase builtins list/dict
are accepted; only the bare "List" 500s ("Dict" happens to be tolerated, but
both are lowercased for consistency).

Verified live against us-chicago-1 (cohere.command-a-03-2025 and
command-latest). Adds a unit regression on the transformed parameterDefinitions
plus a gated integration test exercising an array-param tool end to end.

* fix(oci): make Cohere agentic tool-calling continuation work

Two bugs broke the OCI Cohere tool-calling loop that MLflow {{trace}} judges
drive once a tool has been executed and its result is fed back.

Request side: litellm pulled the last user message into the top-level `message`
and emitted the tool result as a TOOL entry in chatHistory. OCI rejects that
("cannot specify message if the last entry in chat history contains tool
results"), and an empty message alone is rejected too ("message must be at least
1 token long or tool results must be specified"). OCI carries the current turn's
results in a dedicated top-level `toolResults` field. The Cohere transform now
sends an empty message, keeps the user turn in chatHistory, and puts the results
in `toolResults`, matching the langchain-oracle reference. Tool results are no
longer represented as chatHistory entries.

Response side: tool-grounded answers come back with citations carrying
`documentIds` (camelCase) and no `document_ids`, which made the required
`CohereCitation.document_ids` field fail validation and sink the whole response
parse. Those citations are never surfaced, so the field (and CohereSearchQuery's
generation_id) is now optional.

Verified live against us-chicago-1 (cohere.command-a-03-2025 and command-latest),
single and multi-round tool loops. Adds unit regressions on the transformed
request shape and on citation parsing, plus gated integration tests for the
continuation.

* feat: integrate Repelloai Argus guardrail (#30673)

* feat(guardrails): add RepelloAI Argus guardrail integration (#1)

* feat(guardrails): add RepelloAI Argus guardrail integration

Add a new guardrail hook backed by RepelloAI Argus, with dashboard-managed
asset policies enforced via an asset_id and X-API-Key auth.

* fix(guardrails): harden RepelloAI Argus guardrail

- scan streaming responses on output (was bypassing the guardrail)
- log blocked verdicts as guardrail_intervened instead of success
- treat auth/config errors (401/403/404/422) as misconfiguration that
  always blocks, not a fail-open-able unreachable error
- default unreachable_fallback to fail_closed and read it directly;
  block on unknown/malformed verdicts so an API change can't silently
  disable enforcement
- type unreachable_fallback as a Literal, drop the duplicate config model,
  expose unreachable_fallback in the config schema, and stop leaking the
  raw provider response / exception strings to the client

* fix(guardrails): address RepelloAI Argus review feedback

- support ARGUS_API_KEY (with REPELLOAI_API_KEY fallback)
- make asset_id required in the config model
- normalize unreachable_fallback so only fail_open opens; block on 400 misconfig
- correct the shared unreachable_fallback field description

* docs(guardrails): add RepelloAI Argus docs page and dashboard listing

- add docs page covering config, env vars, modes, verdicts, failure semantics
- list RepelloAI Argus in the Guardrail Garden with provider/logo mappings
- add a regression test for the provider logo and display-name resolution

* fix(guardrails): keep RepelloAI asset_id optional in config model

A required asset_id leaked onto the shared LitellmParams (which inherits
RepelloAIGuardrailConfigModel), breaking validation for every other
guardrail. Keep it optional like sibling models; the guardrail __init__
still raises when asset_id is missing, which is the real enforcement.

* Add comment for last user turn scanning

* feat(guardrails): harden repelloai scanning

* feat(guardrails): expand repelloai scanning to include tool definitions

Add extraction of tool definitions and tool call arguments to the RepelloAI
guardrail scanning. Improves detection coverage by including function schemas
and parameters in the prompt sent to the guardrail service. Also captures
detailed error responses in logs and adds guardrail header to streaming responses.

* refactor(guardrails): fix and harden repelloai schema text extraction

- Fix duplicate text in _iter_schema_text: previously all dict values were
  re-queued onto the stack even after scalar/list keys were already extracted
  explicitly, causing names/descriptions to appear twice in the scanned prompt
- Extract schema key frozensets to module-level constants so they are not
  reconstructed on every call
- Change _iter_schema_text from @classmethod to @staticmethod (cls unused)
- Narrow _call_analyze stage param from str to Literal["prompt", "response"]
- Add HttpxResponse type annotation to _raise_for_config_error
- Add LLMResponseTypes annotation to async_post_call_success_hook response param

* fix(guardrails): resolve pyright type errors in repelloai guardrail

- Narrow async_handler.post return from Response|None to Response with
  explicit None guard before calling raise_for_status/json
- Fix list comprehension returning str|None by switching to explicit loop
  with isinstance guard so pyright tracks the narrowing
- Cast model_dump() result to Dict since hasattr does not narrow object
  type in pyright

* fix(guardrails/repello): include Responses API instructions field in prompt scan

The /v1/responses top-level `instructions` field was not included in
_extract_prompt_text, allowing a caller to bypass guardrail policy checks
by putting blocked content in `instructions` while keeping `input` benign.

* feat: add api_key to config model and read prompt from data dict

* fix(guardrails/repello): plug input_text and tool-call response bypass gaps

Responses API input content parts with type 'input_text' were silently
dropped by build_inspection_messages (which only handles type='text'),
allowing callers to send blocked content via that path without triggering
the pre-call scan. Fix: add _extract_input_text_parts to RepelloAIGuardrail
and call it when walking the Responses API input messages.

Post-call scanning skipped responses whose choices contained only tool_calls
or function_call (message.content=None), letting models put blocked output in
function arguments undetected. Fix: _extract_chat_completion_text now calls
_extract_tool_call_args_from_message on each choice message.

Also replace typing.Dict/List with builtin dict/list to clear TID251 strict
ruff violations introduced by this file.

* fix(guardrails/repello): scan Responses API function_call output arguments

Output items with type 'function_call' in a /v1/responses response were
skipped by _extract_responses_api_text; only 'message' items were walked.
A model could return blocked content in function_call.arguments undetected.
Now extract arguments from function_call output items before scanning.

* refactor(guardrails/repello): clean up typing and remove lint-any workarounds

- Replace Optional[X]/Union[X,Y] with X|None/X|Y union syntax throughout
- Use dict[str, object] instead of bare dict in all signatures
- Remove **kwargs from __init__; declare guardrail_name, event_hook, default_on explicitly
- Replace getattr(litellm_params, ...) with direct attribute access now that LitellmParams inherits RepelloAIGuardrailConfigModel
- Add _event_hook_from_mode() to convert str|list[str]|Mode to typed GuardrailEventHooks
- Use TypeAdapter.validate_json() instead of response.json() + manual dict construction
- Add _is_object_dict/_is_object_list TypeGuard helpers to narrow object types without Any
- Remove cast() workarounds and typed intermediate variables that existed only for the now-removed lint-any CI check
- Drop _AddLiteLLMCallback Protocol; budget has sufficient slack for the one reportUnknownMemberType
- Fix GuardrailConfigModel missing type arg: GuardrailConfigModel[BaseModel]

* fix(guardrails/repello): suppress LIT007 on TypeGuard helpers and add streaming scan-skip warning

- Add guard-ok suppressions to _is_object_dict and _is_object_list to satisfy the LIT007 hard-zero budget gate
- Emit verbose_proxy_logger.warning when the streaming hook finds no inspectable text after assembly, matching observability of pre/post hooks

* refactor: modifications for lint check

* feat: add Pinstripes as an OpenAI-compatible provider (#30567)

* feat: add Pinstripes as an OpenAI-compatible provider

Pinstripes (https://pinstripes.io) is an OpenAI-compatible inference
provider serving open-source models (GLM-4.5-Air, Qwen3, DeepSeek, etc.)
with per-token pricing and no subscriptions.

Changes:
- `litellm/llms/openai_like/providers.json`: register pinstripes with
  base_url, api_key_env, and max_completion_tokens→max_tokens mapping
- `litellm/types/utils.py`: add `PINSTRIPES = "pinstripes"` to LlmProviders
- `litellm/constants.py`: add to openai_compatible_providers and
  openai_compatible_endpoints lists
- `litellm/litellm_core_utils/get_llm_provider_logic.py`: auto-detect
  provider when api_base is "https://pinstripes.io/v1"
- `provider_endpoints_support.json`: document supported endpoints
- `tests/`: 7 unit tests covering provider registration, resolution,
  URL auto-detection, api_base override, and Router config

Usage:
    import litellm
    response = litellm.completion(
        model="pinstripes/ps/glm-4.5-air",
        messages=[{"role": "user", "content": "Hello"}],
        api_key=os.environ["PINSTRIPES_API_KEY"],
    )

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix(pinstripes): resolve Greptile P1 review comments

- Add api_base_env: PINSTRIPES_API_BASE to providers.json so env var override works
- Set responses: false in provider_endpoints_support.json — not actually wired up
- Remove docs/my-website/docs/providers/pinstripes.md — belongs in litellm-docs repo

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix(pinstripes): add api_base_env and correct responses capability

- Add api_base_env: PINSTRIPES_API_BASE to providers.json
- Set responses: false in provider_endpoints_support.json

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix(pinstripes): wire up Responses API — add supported_endpoints

Adds supported_endpoints: ["/v1/chat/completions", "/v1/responses"] so
JSONProviderRegistry.supports_responses_api returns true correctly,
matching what provider_endpoints_support.json advertises.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* feat(pinstripes): enable embeddings endpoint

Pinstripes serves nomic-embed-text-v1.5 and bge-m3 via /v1/embeddings.
Add /v1/embeddings to supported_endpoints and set embeddings: true.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix(pinstripes): use 4-space indentation in model_prices_and_context_window.json

Matches the file's existing convention. Flagged by Greptile review.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

* fix(pinstripes): set a2a: false — A2A protocol not implemented

All comparable JSON-configured providers (tensormesh, parasail, empiriolabs,
libertai, neosantara) have a2a: false. Pinstripes does not implement the
Google A2A protocol, so this should be false to match.

Co-Authored-By: Claude Sonnet 4.6 <[email protected]>

---------

Co-authored-by: inference_provider <[email protected]>
Co-authored-by: Claude Sonnet 4.6 <[email protected]>

* fix(rag): attach existing OpenAI file ids (#30628)

* fix(rag): attach existing OpenAI file ids

* chore: use modern typing in rag ingest fix

* chore: retrigger ci

* fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (#30341)

cache_control_injection_points was only consumed by the chat/completions
prompt-management hook; on the native Anthropic /v1/messages path it was
forwarded unused, so deployment-level cache injection was silently dropped
(cache_creation_input_tokens stayed 0 for Anthropic-native clients).

Add AnthropicCacheControlHook.apply_to_anthropic_messages_request to inject
cache_control at block level for system / tools / message locations (the only
forms /v1/messages accepts), wire it into the native anthropic_messages
handler, and pop the param so it does not leak upstream as an unknown field.
A {location: message, role: system} config is redirected to the top-level
system prompt so the same YAML works on both endpoints.

Injection respects Anthropic's 4-block cache_control limit shared across
system, tools, and messages: client-supplied markers count toward the cap and
are never overwritten, a slot is reserved per Bedrock tool_config point, and
injection stops once the budget is exhausted. Locations this path cannot
represent (tool_config) are forwarded downstream instead of being silently
consumed, mirroring get_chat_completion_prompt's remaining_points pass-through.

Built on litellm_internal_staging. Refs #30293

* fix(proxy): release budget reservation when a request is cancelled mid-flight (#30522)

* fix(proxy): release budget reservation on cancel when no chunk was delivered

The pre-call budget reservation increments the cross-pod spend counter by a
request's worst-case cost, then reconciles it on success (cost callback) or
error (failure hook). A client disconnect or timeout cancels the request and
surfaces as CancelledError / GeneratorExit, which neither path catches, so the
reservation leaks. Under a retry storm the leaked holds accumulate, pin the
counter above real spend, and return spurious 429 "Budget has been exceeded" to
keys whose spend is far below budget; the counter only recovers when its TTL
lapses, so the failure is intermittent and self-healing.

Release the reservation in async_streaming_data_generator (which the Anthropic
and Google SSE generators delegate to) on the (CancelledError, GeneratorExit)
path, alongside the existing max_parallel_requests release. release_budget_
reservation_on_cancel runs under asyncio.shield so it completes despite the
in-progress cancellation, is guarded by the reservation's finalized flag, and
swallows a failing release so it cannot replace the in-flight cancellation.

The refund is gated on whether a chunk reached the client. The flag is set
immediately before the yield, after the slow-path hook await: an async generator
suspends at the yield, so a GeneratorExit on disconnect after a delivered chunk
sees it True (keep the hold), while a cancellation during the slow-path await
leaves it False (refund, nothing sent). A non-streaming cancellation delivers
nothing and a completed non-streaming response is reconciled by the success
callback, so neither needs a release here.

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* fix(proxy): reconcile a cancelled reservation to input cost, not zero

A streaming request cancelled before the first chunk previously reconciled its
reservation to zero and finalized it. But by the time the generator is
consuming the response the provider call was already dispatched, so the input
tokens were billed even though no chunk reached the client, and the
success/failure cost callbacks are skipped on cancellation. Refunding to zero
let a caller send an expensive request and abort pre-token to dodge the input
charge.

Compute the request's input-token cost at reservation time and reconcile the
cancelled reservation to it instead of zero. The worst-case output portion of
the reservation is still released (so a legitimate mid-flight cancellation no
longer pins the counter and 429s the key), while the input the provider already
processed is charged.

---------

Co-authored-by: Bytechoreographer <[email protected]>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* fix(caching): encode object name in GCS cache GET path (#30378)

GCS cache reads always missed when gcs_path was set. The GET methods
interpolated the object name directly into the URL path, while the GCS
JSON API requires it to be URL-encoded (a "/" must be sent as %2F).

With gcs_path configured the object name is "<prefix>/<sha256>", so the
raw slash produced a malformed object path and GCS returned 404. httpx
does not raise on 4xx, so the status_code == 200 check fell through and
get/async_get returned None, silently missing on every read. Without
gcs_path the key has no slash, which is why this went unnoticed.

Wrap the object name with urllib.parse.quote(..., safe="") in get_cache
and async_get_cache. Apply the same encoding to the name= query
parameter in set_cache and async_set_cache so the key written matches
the key read back.

Adds regression tests asserting the GET path and SET query are encoded
(%2F) when gcs_path is set, for both sync and async paths; these fail on
the unpatched code.

Fixes #30377

* chore: add soniox stt-async-v5 model (#30672)

* fix(proxy): include model group aliases in v1 model info (#30626)

* Include model group aliases in v1 model info

* Fix model info alias implementation

* removed extra blank line

* chore: rerun CI

* fix(lint): remove redundant noqa directive in proxy_cli.py

* fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme

* Revert "fix: address greptile review - restore bedrock_mantle auth symbols, guard OCI empty message list, validate DIRECT_URL scheme"

This reverts commit 52c7a07.

* Revert "fix(anthropic-messages): apply cache_control_injection_points on /v1/messages path (#30341)"

This reverts commit c9e8a17.

* Revert "fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183)"

This reverts commit 85828da.

* fix(proxy): stop IAM-refresh engine restart from cascading reconnects (#29176) (#30183)

An RDS IAM token refresh recreates the Prisma client, which SIGKILLs the
running query-engine and spawns a new one. That planned kill was
indistinguishable from a crash, and three reconnect paths used two
uncoordinated locks, so a single refresh triggered a cascade of engine
kill/respawn cycles:

  1. `_safe_refresh_token` (holds `_reconnection_lock`) -> recreate -> kill old
     engine, spawn new one.
  2. The engine-death watcher sees that kill, assumes a crash, and calls
     `attempt_db_reconnect(force=True)` (a different lock,
     `_db_reconnect_lock`) -> recreate again -> kills the fresh engine.
  3. In-flight queries failing during the swap are classified as transport
     errors and trigger their own `attempt_db_reconnect` -> recreate again.

Fix coordinates planned restarts across the wrapper and the watcher:

  - PrismaWrapper records the old engine PID in `_expected_engine_deaths`
    before killing it; all four watcher death-detectors (waitpid thread,
    pidfd, already-dead probe, os.kill poll) consume that PID and skip the
    reconnect instead of treating it as a crash.
  - `recreate_prisma_client` now serializes through `_reconnection_lock` and
    bumps a monotonic `_engine_generation`. Callers pass `expected_generation`
    as an optimistic-lock token, so racing/cascading recreates collapse into a
    single restart (losers no-op). This closes the two-lock gap.
  - The direct reconnect path probes the writer with SELECT 1 before
    recreating; a healthy connection (e.g. engine already replaced by a
    refresh) skips the recreate entirely.
  - `_safe_refresh_token` coalesces: it skips when the current token still has
    more than the refresh buffer of runway, so stacked triggers (proactive
    loop + __getattr__ fallback) don't each restart the engine. An
    `on_engine_replaced` hook re-arms the watcher on the new PID.

RoutingPrismaWrapper forwards `expected_generation` and skips recreating the
reader when the writer recreate was skipped.

* fix(lint): modernize type annotations in IAM-refresh prisma client files (UP006/UP045)

* Revert "feat(proxy): show session-aggregate cost and duration in request logs (#25708) (#30507)"

This reverts commit f530b22.

* Revert "fix(dashscope): treat an explicit 0.0 tier cost as a real price, not missing (#30653)"

This reverts commit 4f58bd0.

* Revert "fix(oci): make Cohere {{trace}} judges work (tool param types + agentic tool-calling continuation) (#30646)"

This reverts commit 50f34e0.

* Revert "fix(proxy): fail fast on non-PostgreSQL DATABASE_URL instead of hanging on startup (#30366)"

This reverts commit 0544eed.

* fix(bedrock_mantle): restore BedrockMantleAuthMixin and constants removed by routing rewrite

* fix(key management): restore exact /key/list user_id & key_alias matching by default (#30593)

Before substring search was added (commit 33bd570), /key/list matched user_id
and key_alias exactly. That change made admin-authenticated calls substring-match
by default, breaking the prior contract: a caller passing an exact user_id as an
access filter (e.g. an integration scoping to one user with an admin key) then
received other users' keys -- user_id="alice" also returned "alice2",
"alice-test", etc. This is a cross-user key disclosure.

Make substring matching opt-in via a new admin-only substring_matching=true query
param; default to exact, restoring the prior behavior. The dashboard search box
(keyListCall) passes the flag so partial search still works. Non-admins remain
exact and scoped to their own keys.

Updates the proxy-behavior key_alias test to opt in and adds an exact-by-default
guard; adds list_keys unit coverage for the opt-in gate.

---------

Co-authored-by: perseus <[email protected]>
Co-authored-by: Hannah Smith <[email protected]>
Co-authored-by: Charlie Patterson <[email protected]>
Co-authored-by: Matthew Lapointe <[email protected]>
Co-authored-by: Claude Opus 4.8 (1M context) <[email protected]>
Co-authored-by: KRISH SONI <[email protected]>
Co-authored-by: Yash Raj Pandey <[email protected]>
Co-authored-by: Nitish Agarwal <[email protected]>
Co-authored-by: hcl <[email protected]>
Co-authored-by: tushar8408 <[email protected]>
Co-authored-by: AD Mohanraj <[email protected]>
Co-authored-by: Fede Kamelhar <[email protected]>
Co-authored-by: Lavish Bansal <[email protected]>
Co-authored-by: max-amos <[email protected]>
Co-authored-by: inference_provider <[email protected]>
Co-authored-by: NK <[email protected]>
Co-authored-by: 安妮的心动录 <[email protected]>
Co-authored-by: Rick <[email protected]>
Co-authored-by: Bytechoreographer <[email protected]>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Burak Ömür <[email protected]>
Co-authored-by: Dan Lemon <[email protected]>
Co-authored-by: Vanika Dangi <[email protected]>
Co-authored-by: Jay Gowdy <[email protected]>

* chore(ci): remove Agent Shin pull_request_target workflows

Drop the two Agent Shin workflows that ran on the pull_request_target
trigger: the PR triage workflow and the review gate. Both were dry-run
and gated behind AGENT_SHIN_ENABLED, so no live automation changes.

The shared scripts under .github/scripts stay in place; four other
Agent Shin workflows still depend on them and run on schedule, dispatch,
and issue events rather than pull_request_target

* ci(zizmor): also run on litellm_internal_staging

PR #30784 deleted .github/workflows/review_gate.yml and
triage_pr_with_llm.yml, but test_github_triage_workflows.py still
listed both in its parametrize tables, so _load_workflow raised
FileNotFoundError for every case naming them.

Remove the two stale entries from DESTRUCTIVE_GATE_ENV and
LLM_CLIENT_INSTALLER_WORKFLOWS; the remaining four workflows that still
exist keep their guardrail coverage.

Delete the in-product survey and Claude Code feedback prompts end to end.

Frontend: remove the src/components/survey/ module, the index page's nudge
state/effects/handlers, the getInProductNudgesCall helper, and the orphaned
"Disable UI nudges" toggle in the admin UI Settings page; prune the stale
eslint-suppressions entries.

Backend: remove the now-dead /in_product_nudges route, the InProductNudgeResponse
type, and the disable_ui_nudges UI setting (Field + allowlist). Nothing read it
for logic and the UISettings model is extra="allow", so existing stored configs
are unaffected (the value is just no longer surfaced). schema.d.ts is regenerated
and the two tests covering the removed route/setting are dropped.

Cut the default "Virtual Keys" landing (?page=api-keys) over to a path route at
(dashboard)/api-keys. The dashboard is extracted into a shared ApiKeysDashboard
component used by both the new route and the index's inline render, so there's
no duplication. Adding the MIGRATED_PAGES entry repoints the sidebar item and
redirects ?page=api-keys to /ui/api-keys.

The index is the post-login landing and still hosts the legacy switch for the
not-yet-migrated pages (models, pass-through, usage) plus the invitation flow,
so it stays. The auto-redirect now fires only for an explicit ?page= param,
leaving the bare /ui/ landing to render inline; this keeps the return-URL
handling and the invitation_id flow (both of which run at the bare landing)
intact, where a blanket redirect would have dropped them. The new route uses
useAuthorized for the login gate, matching every other migrated route.

)

Switch the zizmor check to fail on any finding at medium severity or
above (advanced-security off, min-severity medium, annotations on) so it
can be promoted to a required check, and pin the engine to zizmor 1.24.1
through zizmor-action v0.5.6 for deterministic runs.

Clear the findings that were outstanding so the check passes: correct
mismatched action pin version comments, scope the proxy endpoint
workflow's id-token and pull-requests permissions to the jobs that use
them, and mark the server-root-path docker build as non-publishing while
dropping its shared gha build cache.

…tion streaming passthrough (#30800)

* fix(proxy): use e.request_data for logging_obj in ModifyResponseException streaming passthrough

When a guardrail blocks a streaming request pre-call by raising
ModifyResponseException (or RejectedRequestError), chat_completion streams the
violation message back as a 200 by building a CustomStreamWrapper. It read the
logging object from the outer request body (`data.get("litellm_logging_obj")`),
but that dict never carries litellm_logging_obj -- it diverges from the
processor's data at function_setup, and only the processor copy (exposed as
e.request_data, already bound to `_data` here) gets the logging object
attached. CustomStreamWrapper.__init__ then dereferences
`logging_obj.model_call_details` on None and 500s the request with
"AttributeError: 'NoneType' object has no attribute 'model_call_details'".

Read logging_obj from `_data` (= e.request_data) in both streaming
passthrough handlers so the refusal streams correctly. The non-streaming and
the anthropic/responses passthrough paths were unaffected.

Adds a regression test asserting the wrapper receives the logging object from
e.request_data rather than None.

* test(proxy): cover RejectedRequestError streaming passthrough

The streaming logging_obj fix was applied to both the ModifyResponseException
and RejectedRequestError handlers, but only the former had a regression test.
Extract a shared helper and add a parallel test for the RejectedRequestError
streaming path so both handlers stay guarded against the None-logging_obj crash.

---------

Co-authored-by: Joseph Barker <[email protected]>

…AWS SDK (#30815)

…creds per exporter (#30590)

* fix(otel): one v2 logger owns the global provider; scope tenant creds per exporter

The proxy published the OTel global TracerProvider before callbacks were
initialized, so no preset logger existed yet and a second generic logger was
built that won the global provider. Server spans then exported through a
different provider than the preset's gen-ai spans, orphaning the LLM span on
the preset backend. Publish after callback init and reuse the already-built
logger instead.

Separately, per-request tenant OTLP credentials were stamped onto every OTLP
exporter, leaking one backend's key onto a co-configured backend. Tag each
exporter with the preset that contributed it and apply dynamic credentials
only to the matching owner.

* fix(otel): satisfy Any-discipline on changed lines

Type the logger-selection parameter as Sequence[object] (isinstance narrows
it), cast the list[Any] global at the single call site, and pass model_copy a
typed dict[str, str] update so no changed line carries an Any value.

* fix(otel): annotate the untyped-global boundary with any-ok

select_global_otel_v2_logger consumes litellm._in_memory_loggers, a shared
List[Any] global this change does not own. A cast doesn't satisfy the
Any-discipline checker (it inspects the inner expression), and re-annotating the
global is out of scope, so mark the single boundary line any-ok.

* test(otel): cover the startup global-provider publish via injectable helper

The publish step lived inline in proxy_startup_event (a FastAPI lifespan unit
tests do not execute), so its lines were uncovered though the selection logic
was tested. Extract publish_global_otel_v2_provider, which selects the single v2
logger and publishes its provider through an injected setter, and unit-test that
the published provider is the selected logger's. proxy_server delegates to it.

* refactor(otel): select global provider from the registered owner, not a list scan

The startup publish picked the global TracerProvider by scanning
_in_memory_loggers for the first OpenTelemetryV2, re-deriving an answer the
factory already settled: the first logger built registers itself as
proxy_server.open_telemetry_logger, and every other v2 path (guardrail, identity
seeding, phase spans) routes through that owner via _registered_v2_logger. Pass
that owner into select_global_otel_v2_logger so the global provider reuses the
same logger instead of an independent, order-dependent guess; the list scan
remains the SDK-path fallback. The owner is injected at the proxy call site to
keep the helper free of hidden global reads.

* refactor(otel): type ExporterSpec.owner as an ExporterOwner enum

The owner field carried free-form strings that had to match preset callback
names. Introduce a str-based ExporterOwner enum (values equal to the callback
names, so per-request credential routing's owner==callback_name comparison still
holds) and have each preset tag its exporter with the enum member.

* refactor(otel): rename ExporterOwner.ARIZE to ARIZE_AX

Distinguish the hosted Arize AX backend from Arize Phoenix at the member level
while keeping the value 'arize' (the public callback name routing compares
against). Add a comment noting AX and Phoenix are separate backends.

…eams (#30787)

…treams (#30788)

A streaming request that breaks mid-flight, for example on a mid-stream read
timeout, still bills the provider for the chunks already delivered, yet the proxy
recorded that interrupted request as a zero-spend failure. An earlier revision
logged the recovered partial usage through the success path, which mislabeled a
failed request as a success and produced a misleading spend row

This recovers the partial usage where the failure is actually logged. The
streaming handler assembles the usage from the chunks seen so far and stashes it,
with its cost, on the logging object before firing the failure handlers. The
proxy failure hook lifts that usage and cost onto request_data before the
non-serialisable logging object is popped, and the spend-log writer records the
real partial spend on the failure row instead of a hardcoded zero;
get_logging_payload honors the recovered usage for the token columns and
_failure_handler_helper_fn preserves the recovered cost so the non-DB failure
loggers stay consistent

A request that recovers via a successful fallback is unaffected: the failure hook
only fires when the whole request fails, so the fallback's combined-usage success
row stays the single source of truth and there is no double counting

Resolves LIT-3825

Co-authored-by: veria-ai[bot] <224490171+veria-ai[bot]@users.noreply.github.com>

The "View Usage Guide" button on the legacy Usage page (shown when
DISABLE_EXPENSIVE_DB_QUERIES is set, i.e. SpendLogs has 1M+ rows) linked
to docs/proxy/spending_monitoring, which was removed from the docs and
now returns 404. Point it at docs/proxy/cost_tracking, which is live.

Fixes LIT-2724

…29990)

The delete-team confirmation modal warned that a team's keys would be
deleted but said nothing about models. #29977 made team deletion also
delete the team's BYOK models, so the modal copy was understating what
gets removed.

The warning banner now mentions models alongside keys, and the
always-shown confirmation message does too so a team that has models but
no keys (the banner only renders when keys exist) still gets warned.

…he scope keys (#30675)

Adds a "valkey-semantic" cache type so semantic prompt caching can run
against Valkey clusters (for example AWS ElastiCache for Valkey) using the
valkey-search module.

The existing "redis-semantic" backend cannot drive valkey-search. RedisVL
gates the connection on a RediSearch module version that valkey-search does
not report, and its SemanticCache index declares the prompt as a TEXT field,
which valkey-search does not implement. ValkeySemanticCache therefore talks to
valkey-search directly over redis-py: it builds a vector index from the field
types valkey-search supports (TAG for caller scope, VECTOR for the prompt
embedding) and runs KNN queries for retrieval. Prompt extraction, embedding
generation, and cached-response parsing are reused from RedisSemanticCache
since those are backend agnostic. The redis dependency is imported lazily in
the cache dispatch so importing litellm without redis installed still works.

It also fixes semantic-cache scope keys so similarity matching works across
reworded prompts. get_cache_key() hashed messages / prompt / input into the
litellm_cache_key that every semantic backend filters its KNN search on, so a
paraphrase landed in a different bucket and never matched, even far above the
similarity threshold. For semantic cache types the prompt-bearing params are
now excluded from the scope key and the server-set tenant identity
(user_api_key, team, org) is appended instead, restoring embedding matching
within a tenant while keeping cache entries scoped to the authenticated
key / team / org. The three semantic backends share this key, so the same
change fixes redis-semantic and qdrant-semantic.

Connections resolve from VALKEY_HOST / VALKEY_PORT / VALKEY_PASSWORD, falling
back to REDIS_* for drop-in compatibility, and passwordless clusters (IAM or
no-auth) are supported.

Resolves #29121
Fixes #29086

)

The deprecated OldTeams component takes only accessToken, userID, userRole
and premiumUser; it ignores the teams prop these tests passed and instead
populates its table from the mocked teamListCall. The delete-warning block
never set teamListCall, and vi.clearAllMocks clears call history but not
implementations, so the table rendered the "Legacy Team" (keys.length 2)
left behind by the previous block's last test. Both delete tests therefore
ran against that leaked team: the keys-present case passed only because the
leaked count happened to be 2, and the no-keys case rendered the same warning
it asserted should be absent, so it failed.

Seed the team through the channel the component actually reads (teamListCall)
and drop the props it never consumes, so each test renders exactly the team
it declares. The keys-present case now uses a distinctive count so it can no
longer pass on a coincidental leak

… ruff gate (#30877)

* feat: add CI-parity mode and truncation-proof summary to strict ruff gate

* refactor: tolerant worktree cleanup and concrete GateInputs types

* fix: clean up temp dir when git worktree add fails

* fix: align lint-gate with CI by dropping unused --ci-parity path

The lint-gate Makefile target invoked ruff_strict_gate.py with --ci-parity,
which counted violations on a throwaway merge of base into HEAD against base
counts at the base tip. CI in test-linting.yml runs the same script without
--ci-parity on a PR-head checkout, taking the gather_fast path that counts on
the live tree against base counts at the merge-base. A local pass could
therefore disagree with CI.

Drop --ci-parity from the Makefile and remove the now-unused gather_ci_parity
branch and flag so there is one code path that both local and CI exercise.
The docstring claim that CI runs against the synthetic merge ref was also
wrong; the workflow checks out github.event.pull_request.head.sha.

---------

Co-authored-by: Cursor Agent <[email protected]>

….20.2) (#30894)

* bump: version 0.1.42 → 0.1.43

* bump: version 1.89.0 → 1.90.0

* adding uv lock

…30897)

* fix(watsonx): wrap string embedding input in array for WatsonX API

WatsonX text/embeddings expects inputs as []string; OpenAI clients often send a single string.

Co-authored-by: Cursor <[email protected]>

* style(watsonx): format watsonx embed transformation for black

Co-authored-by: Cursor <[email protected]>

* fix(watsonx): avoid UP006 in embed transformation strict lint gate

Use list[str] and branch-based input normalization instead of List and cast
so the watsonx embedding change does not add strict ruff UP006 violations.

Co-authored-by: Cursor <[email protected]>

---------

Co-authored-by: Shivam Rawat <[email protected]>
Co-authored-by: Cursor <[email protected]>

…ndpoint (#30900)

* test: point router/completion/triton tests at the local fake OpenAI endpoint

The shared Railway-hosted mock (exampleopenaiendpoint-production.up.railway.app)
takes down unrelated CI jobs whenever it is unreachable. #30695 moved the mounted
proxy configs onto a job-local fake server but left these in-Python api_base
literals pointing at the dead host, so litellm_router_testing, local_testing_part1,
local_testing_part2 and llm_translation_testing still fail with a 404
"Application not found" when Railway is down

Resolve the api_base from FAKE_OPENAI_API_BASE (default http://127.0.0.1:8190)
through a shared helper, auto-start the canned server from the local_testing and
llm_translation conftests when nothing is already serving, and extend the server
with a Triton embeddings route and a slow-endpoint delay so the triton and
latency-timeout tests run fully offline. The deliberately broken fallback URL is
left as-is so fallback handling still has a failing upstream

* fix: ignore non-loopback FAKE_OPENAI_API_BASE so the local mock is used in CI

* fix: drop 0.0.0.0 from loopback hosts, an unreliable client connect target

* fix(tests): keep fake OpenAI mock alive across xdist workers

ensure_fake_openai_endpoint registered atexit on the worker that spawned
the subprocess, so under -n 4 the first worker to drain its queue would
terminate the shared mock while siblings were still hitting it. Detach
the child via start_new_session and drop the per-worker teardown; reuse
on /health handles re-runs and CI containers clean up themselves

* feat(sandbox): add e2b code execution primitive

Add a provider-agnostic code execution primitive that runs model-generated
code in an isolated sandbox and returns the output, with e2b as the first
backend over raw httpx (no SDK dependency).

Public API: litellm.acode_interpreter_tool (ephemeral create -> run -> delete)
plus the low-level lifecycle litellm.acreate_sandbox / arun_code /
adelete_sandbox. Each is @client-decorated so operations are logged like
litellm.asearch. Backends implement BaseSandboxConfig; resolved via
ProviderConfigManager.get_provider_sandbox_config.

* fix(sandbox): address review feedback and CI gates

- document e2b provider in provider_endpoints_support.json and add a sandbox endpoint definition
- regenerate dashboard CallTypes after the sandbox call-type additions
- guard explicit timeout=0 instead of coercing it to the default
- require a ContainerHandle access token before running code; reject bare-id runs
- return False on a 404 delete now that the shared http handler raises for status
- skip non-JSON NDJSON lines and cap streamed output to bound memory
- move the real-network integration tests out of tests/test_litellm into tests/integration/sandbox

* fix(sandbox): satisfy strict ruff gate and scope star-exports

- modernize annotations in the new sandbox modules to PEP 585/604 (list/dict,
  X | None) and drop the now-unnecessary quoted forward refs so the strict-rule
  budget delta for UP006/UP037/UP045 returns to zero
- add __all__ to litellm/sandbox/main.py so 'import *' only re-exports the four
  public entrypoints instead of leaking module-level imports

* fix(sandbox): drop quotes on sandbox config return annotation

utils.py uses 'from __future__ import annotations', so the quoted forward ref
tripped UP037; the unquoted union is lazily evaluated and keeps the strict-rule
delta at zero

* chore(sandbox): re-trigger automated review after addressing feedback

… is not set" (#30903)

The migrated /ui/api-keys route gates rendering on useAuthorized() but read userID from the AuthContext (useAuth), which hydrates asynchronously. On a hard refresh or deep link the route could render UserDashboard before AuthContext had populated userID, so UserDashboard hit its `userID == null` guard and showed "User ID is not set". The legacy index page avoided this by gating on AuthContext's own authLoading; the migration switched the gate to useAuthorized without aligning the identity source.

Read identity from useAuthorized (a synchronous cookie decode) so userID is populated whenever the route is authorized. useAuth is kept only for the backfill setters UserDashboard still expects, until the planned AuthContext consolidation removes them.

Refs LIT-3687

….20.2) (#30906)

… seam (#30887)

Introduce a single, typed caller identity that is resolved once at the auth
boundary and read by reference downstream, instead of being re-derived from a
50-field key object or rebuilt from request metadata.

What this adds (litellm/proxy/auth/resolvers/), organized by responsibility:
- Principal: a small, frozen, identity-only value type (user / organization /
  teams / project / end-user / roles / scopes / network), with its sub-models
  and the role mapping. No budget or policy state; those stay on the key object.
- DbIdentityStore: the auth flow's resolver, owning both halves of resolving a
  caller. resolve_key does the one combined_view lookup (cache, then DB via the
  shared lower-level helpers, then write-back) and returns the key object, which
  still flows for budget / rate-limit / policy unchanged. principal_from_key
  projects the identity slice of that key object into a Principal, issuing no
  lookup. user_api_key_auth resolves every key through the store rather than
  calling get_key_object directly; auth_checks.get_key_object stays as the legacy
  entrypoint for its other callers until they migrate.
- network: the X-Forwarded-For / trusted-proxy CIDR primitives live here in one
  place. trusted_proxy_utils now imports them rather than keeping a second copy.

At the seam, user_api_key_auth projects one per-request Principal off the
resolved key object and stamps the request network context onto it once
(X-Forwarded-For is trusted only when trusted_proxy_ranges is configured). It is
attached to request.state.principal for the downstream consumers later phases
add. The projection is additive and defensive: a failure never rejects an
already-authenticated request, and a missing principal must be treated as deny
by any future reader. The Principal is always identifiable (credential_ref and a
stable subject off the token), never anonymous.

This is additive and changes no behavior today; it is the identity foundation
the spend-attribution and authorization phases build on.

…ace (#30885)

* feat(fireworks_ai): sync chat completions endpoint with full API surface

Add 23 missing request parameters to get_supported_openai_params():
seed, top_logprobs, min_p, typical_p, repetition_penalty,
mirostat_target, mirostat_lr, logit_bias, echo, echo_last, ignore_eos,
prompt_cache_key, prompt_cache_isolation_key, raw_output,
perf_metrics_in_response, return_token_ids, safe_tokenization,
service_tier, metadata, speculation, prediction, stream_options,
sampling_mask. Also add reasoning_history gated on supports_reasoning.

Fix prompt_truncate_length to prompt_truncate_len to match the actual
API parameter name. The old name was never in
DEFAULT_CHAT_COMPLETION_PARAM_VALUES, so it always went to extra_body
and was rejected by Fireworks; it never actually worked.

Normalize reasoning_effort boolean values to strings: True becomes
"medium", False becomes "none". The Fireworks OpenAPI schema
documents these as accepted types, but the server rejects non-string
values with HTTP 400 in practice. Integers pass through as-is since the
server is expected to validate them.

Auto-inject stream_options.include_usage=true when stream=true and the
user has not explicitly set stream_options. Without this, Fireworks
returns null usage in all streaming chunks, which is inconsistent with
the non-streaming behavior where usage is always present. If the user
explicitly sets include_usage=false, it is preserved.

Capture Fireworks-specific response fields in transform_response():
perf_metrics, prompt_token_ids, raw_output, and token_ids are now
extracted from the response and stored in response._hidden_params
(fireworks_perf_metrics, fireworks_prompt_token_ids,
fireworks_raw_outputs, fireworks_token_ids) so they are accessible to
logging, the proxy, and downstream consumers when the corresponding
request parameters are enabled.

Remove deprecated document inlining logic. Document inlining was
deprecated on 2025-06-30
(https://docs.fireworks.ai/updates/changelog#-document-inlining-deprecation).
This removes _add_transform_inline_image_block(), the file-to-image_url
migration in _transform_messages_helper(), and the
disable_add_transform_inline_image_block lookup. Current models that
support image input do so natively as VLMs. cache_control,
provider_specific_fields, and thinking_blocks stripping is retained.

Update get_provider_info() to look up supports_vision and
supports_pdf_input from the model cost map instead of hardcoding both
to True (which was based on the now-deprecated document inlining).
supports_prompt_caching remains True.

API docs: https://docs.fireworks.ai/api-reference/post-chatcompletions
Reasoning guide: https://docs.fireworks.ai/guides/reasoning
Prompt caching: https://docs.fireworks.ai/guides/prompt-caching

* fix fireworks chat api surface gaps

* Scope Fireworks thinking param to reasoning models

* style: fix black formatting

* fix(test): update minimax-m3 expected_vision to True

* test: cover non-dict content branch in transform_messages_helper

* fix(fireworks_ai): remove metadata from supported params to prevent internal metadata disclosure

* test(fireworks_ai): replace stale document-inlining capability test

The CircleCI-only litellm_utils_tests suite still asserted the old
behavior where document inlining made every Fireworks model report
supports_pdf_input and supports_vision as True. That premise was removed
in this change, so the test now reflects cost-map-driven capabilities:
unmapped models no longer advertise vision/PDF support while mapped VLMs
like minimax-m3 still do.

* test(fireworks_ai): add end-to-end regression for native OpenAI params

The existing coverage for the newly supported OpenAI-native params asserted
list membership in get_supported_openai_params or called map_openai_params
with a hand-built dict, both of which bypass the get_optional_params gate
(DEFAULT_CHAT_COMPLETION_PARAM_VALUES). That gate is what previously raised
UnsupportedParamsError for seed, top_logprobs, logit_bias, prompt_cache_key,
service_tier and prediction when drop_params=False. Assert the full path so a
revert of the supported-params additions fails the test instead of passing a
shallow membership check.

* test(fireworks_ai): fix test isolation in vision/inlining tests

Use monkeypatch in test_fireworks_ai_vision_capability_from_cost_map so the
LITELLM_LOCAL_MODEL_COST_MAP env var and litellm.model_cost are restored after
the test instead of leaking global state into the rest of the process.

Switch the document-inlining integration tests off deepseek-v3p1, whose
supports_vision is null in the cost map, onto minimax-m3 which is explicitly
supports_vision:true. The pass-through assertions no longer depend on a model
incidentally not being marked non-vision.

* fix(fireworks_ai): gate image rejection on exact vision capability

The image_url rejection read supports_vision via _get_model_cost_capability,
which falls back to hyphen-boundary substring matching when no exact cost-map
entry exists. A custom or fine-tuned model id that merely contains a known
non-vision model's short name (e.g. an id ending in -glm-5p2) inherited that
entry's supports_vision:false and hard-failed valid image_url blocks on a
vision-capable deployment.

Split the exact candidate-key lookup into _get_model_cost_capability_exact and
use it for the hard rejection so a fuzzy match can never block images; the
substring fallback stays a soft signal for capability reporting. Also rewrites
the fallback as a comprehension + max instead of an accumulating loop.

* feat(fireworks_ai): surface response fields on streaming responses

The Fireworks-specific response fields (perf_metrics, prompt_token_ids,
per-choice raw_output and token_ids) were only captured into _hidden_params in
transform_response, which runs for non-streaming completions; streaming chat
went through the default OpenAI chunk handler and dropped them.

Add a FireworksAIChatCompletionStreamingHandler that the provider now returns
from get_model_response_iterator. It reuses one extraction helper with
transform_response and attaches the fields to each streamed chunk's
provider_specific_fields, which is the channel litellm preserves when it
rebuilds streamed chunks (per-chunk _hidden_params is not carried through).
Per-choice token_ids/raw_output ride the content chunks; response-level
perf_metrics/prompt_token_ids ride the final usage chunk. Covered by an
end-to-end streaming test through litellm.completion(stream=True).

---------

Co-authored-by: Ahmad Shahzad <[email protected]>
Co-authored-by: Graham Neubig <[email protected]>

* feat: plugin architecture — toggle between AI Gateway and external plugins

Adds a generic plugin system so any external service can register with
litellm and appear as a mode in the UI alongside the AI Gateway.

Backend (litellm/proxy/plugin_routes.py — new):
- GET /api/plugins: returns registered plugins from config; returns
  plugin_key only to authenticated requests
- ANY /plugin-proxy/{name}/{path}: reverse proxies API calls to plugin

Config:
  general_settings:
    plugins:
      - name: my-plugin
        display_name: My Plugin
        url: https://my-plugin.example.com
        plugin_key: sk-...   # plugin auth key, passed to iframe

UI:
- PluginModeContext.tsx: fetches /api/plugins, persists mode to localStorage
- leftnav.tsx: mode switcher dropdown at top of sidebar; plugin mode shows
  plugin-specific nav items
- layout.tsx: renders iframe to plugin URL in plugin mode; passes plugin_key
  as ?token= for auto sign-in

Plugin contract: expose GET /api/plugin-manifest returning
{ name, display_name, nav_items[], capabilities[] }. No litellm changes
needed to add new plugins — config only.

Reference implementation: LiteLLM-Labs/litellm-agent-control-plane

* feat: add Plugins tab to Admin Settings UI

Allows admins to add/edit/delete plugin registrations directly in the
litellm UI under Admin Settings > Plugins, instead of editing config.yaml.

Uses existing /config/field/update API to persist to general_settings.plugins.
Each plugin entry has: name (identifier), display_name, url, plugin_key.

* fix(ci): black, prettier, eslint, async-client violations

- Black: format plugin_routes.py and proxy_server.py
- Prettier: format PluginModeContext.tsx and PluginSettings.tsx
- ESLint: replace raw fetch() with createApiClient in PluginModeContext
- ESLint: use lazy useState initializer to read localStorage instead of
  calling setModeState inside useEffect (react-hooks/set-state-in-effect)
- code-quality: replace httpx.AsyncClient per-request with
  get_async_httpx_client() shared client (avoids +500ms overhead)

* fix(ci): schema.d.ts regen, Black proxy_server.py, ApiClientConfig fix

- Regenerate schema.d.ts for new /api/plugins routes
- Re-run Black 26.3.1 on proxy_server.py (matches CI version)
- Fix PluginModeContext: createApiClient requires getBaseUrl field

* fix: security hardening + CI fixes

Security (Greptile 1/5 → addressing all 3 findings):
- plugin_routes.py: add Depends(user_api_key_auth) to both /api/plugins
  and /plugin-proxy/{name}/{path} — was an unauthenticated open relay
- plugin_routes.py: /api/plugins now returns plugin_key only to callers
  with a valid litellm token (enforced by user_api_key_auth), not just
  any header presence
- layout.tsx: replace ?token= URL param with postMessage(targetOrigin)
  — token no longer exposed in browser history / logs / Referer headers

CI:
- backend/routes/allowlist.py: add /api/plugins and /plugin-proxy/ to
  fix test_gateway_plus_backend_covers_full_app
- schema.d.ts: regenerated with enterprise routes included
- Black + Prettier formatting

* fix: regenerate schema.d.ts with enterprise routes included

Install litellm-enterprise workspace member before gen:api so audit and
other enterprise routes appear in the generated types, matching what CI
produces with uv sync --extra proxy.

* fix: exclude plugin routes from OpenAPI schema, restore upstream schema.d.ts

Both /api/plugins and /plugin-proxy/ are internal infrastructure routes,
not part of the public litellm API surface. Marking include_in_schema=False
prevents Python-version-dependent schema diffs from breaking the schema
sync check across different environments.

* fix: schema.d.ts - passing schema base + exact plugin route types from openapi-typescript

Use the CI-correct schema from a recently passing branch as base, then
inject plugin route entries (paths + operations) generated by
openapi-typescript from the plugin routes' OpenAPI spec. This avoids
Python-version-dependent formatting differences that made local gen:api
produce incorrect output.

* fix: schema.d.ts - insert plugin ops at correct route registration position

Plugin operations belong after delete_memory_v1_memory__key__delete
(memory_router is included immediately before plugin_router in proxy_server.py),
not after list_organization which is alphabetically but not registration-order.

* fix: schema.d.ts - correct op positions from hunk analysis

list_plugins_api_plugins_get goes after event_logging_batch op (hunk 1: line 33583).
plugin_proxy ops go after create_policy_policies_post (hunk 2: line 44634).
Previous location after delete_memory_v1_memory__key__delete was wrong.

* fix: schema.d.ts - proxy ops go before create_policy (after otel_spans)

* fix(security): restrict plugin_key to proxy_admin role only

Veria finding: plugin_key was returned to any authenticated caller.
Now only proxy_admin users receive plugin credentials in /api/plugins
response — regular internal users see plugin name/url but not the key.

* fix: update schema.d.ts docstring for list_plugins

* fix: clear plugin registry on config reload (Greptile medium)

register_plugins_from_config now replaces the registry instead of
merging, so plugins removed from config are unreachable immediately
without requiring a process restart.

* fix(security): encrypted token exchange for plugin iframe — no raw litellm credential exposure

The dashboard was sending the user's litellm bearer token to the plugin
iframe via postMessage, allowing a compromised plugin to act as that user.

Fix:
- GET /api/plugins/auth-token: proxy encrypts caller token with Fernet
  keyed from LITELLM_SALT_KEY, returns ciphertext only
- UI postMessages the ciphertext (not raw token) to the iframe
- Plugin decrypts server-side with same LITELLM_SALT_KEY via POST /api/plugin-auth
- Raw litellm credential never leaves the proxy in plaintext

Additional hardening already in place:
- /plugin-proxy/* strips Authorization header, injects plugin_key instead
- plugin_key only returned to proxy_admin role via /api/plugins
- Plugin registry cleared (not merged) on config reload

Adds docs/plugin_architecture.md with plugin integration guide.

* fix(code-quality): use get_async_httpx_client in plugin_proxy

* fix: add /api/plugins/auth-token to schema.d.ts

* fix: use apiClient for auth-token fetch, copy correct layout.tsx and PluginModeContext

- Replace raw fetch() with createApiClient (fixes no-restricted-syntax ESLint rule)
- Copy correct layout.tsx with encrypted token + postMessage approach
- Copy correct PluginModeContext.tsx with accessToken prop injection
- Update schema.d.ts with auth-token path and operation entries

* fix: add plugin_auth_token operation to schema.d.ts

* fix(security): strip cookie/set-cookie + fix compressed response headers

Veria High: cookie header was forwarded to plugin backends allowing
capture of litellm JWT session cookies. Strip cookie on requests.
Strip set-cookie from responses so plugins cannot overwrite litellm
session cookies.

Greptile P1: httpx decompresses responses but resp.headers still
contained Content-Encoding/Transfer-Encoding/Content-Length from the
wire. Forwarding these caused double-decompression and length errors.
Now filtered via _RESPONSE_STRIP before returning to the browser.

* fix: update plugin_key help text — no more ?token= reference

* fix(security): disable follow_redirects to prevent SSRF

follow_redirects=True allowed a plugin backend to return a 3xx to an
internal URL, causing the proxy to fetch that internal service and relay
the response. Disabled: clients handle their own redirects.

* fix: forward user identity headers to plugin to address confused deputy

Plugins receive X-LiteLLM-User-Id and X-LiteLLM-User-Role so they can
enforce their own per-user access control before acting on requests that
arrive with the shared plugin_key credential.

* fix(security): restrict /plugin-proxy/* to proxy_admin role

Closes the confused deputy gap: regular users could invoke any plugin
endpoint using the shared plugin_key as a bearer credential. Now only
proxy_admin callers can use the plugin proxy route.

Plugin UIs communicate with the plugin service directly via the iframe
(using the encrypted token exchange); this proxy route is for
administrative/server-to-server access only.

* fix: update schema.d.ts for admin-only proxy route docstring

* fix(bug): use PassThroughEndpoint instead of None for get_async_httpx_client

get_async_httpx_client(llm_provider=None) raises TypeError — the function
concatenates the provider string and None is not a str. Use
httpxSpecialProvider.PassThroughEndpoint, the enum value used by other
internal proxy pass-through routes.

* fix(security): add 30s TTL to encrypted plugin auth tokens

Veria medium: encrypted tokens had no expiry, allowing indefinite replay.
Fernet embeds a timestamp; decrypt_token now passes ttl=30 so tokens
older than 30 seconds are rejected even with a valid HMAC.

Plugin's /api/plugin-auth must call litellm within 30s of the iframe
receiving the postMessage — normal browser behavior, tight enough to
close the replay window.

* feat(ui): topnav plugin switcher, embed plugins at their root

Builds on the plugin architecture already on this branch (encrypted-token
postMessage handshake, /api/plugins, PluginSettings) and removes the parts of the
embed that assumed a specific plugin's shape.

The mode switcher moves out of the sidebar into the topnav and lists AI Gateway
plus each registered plugin by its display_name. Selecting a plugin hides
litellm's sidebar entirely and renders the plugin full-bleed at its root url; the
plugin draws its own navigation inside the iframe. This drops the hardcoded
"Agent Control Plane" label and the hardcoded Sessions/Agents/Routines/... nav
groups (agentControlPlaneMenuGroups / acpPagePaths) that only matched the agent
platform and 404'd for a plugin that serves only / (e.g. the chat UI). The
encrypted-token postMessage flow is unchanged.

Note: embedding at root means a plugin must route internally from /; plugins that
previously relied on the /sessions entrypoint should redirect from their root.

* fix(security): audience-scoped identity claim replaces litellm token

Veria: shared LITELLM_SALT_KEY with plugins + encrypting user bearer token
created delegation/impersonation risk.

Architecture change:
- /api/plugins/auth-token now issues a plugin-scoped identity CLAIM
  {user_id, user_role, plugin, exp} encrypted with HMAC(LITELLM_SALT_KEY, plugin_name)
- Each plugin holds only its own HMAC-derived key; cannot forge claims for
  other plugins or recover LITELLM_SALT_KEY
- Claim contains NO litellm bearer token — compromised plugin learns caller
  identity only, cannot act as that user against the proxy
- 30s TTL enforced in both Fernet header and explicit exp field
- LAP /api/plugin-auth verifies claim, returns its own master key to browser
  (LAP key never exposed without valid claim)

* fix(plugins): allow registering plugins from the admin UI

Adding a plugin in the UI POSTs general_settings.plugins to /config/field/update,
which rejected it with "Invalid field=plugins passed in." because `plugins` was
not a field on ConfigGeneralSettings. Add a typed PluginConfig model and a
`plugins` field so the update validates and persists.

The in-memory plugin registry only refreshed at startup, so a plugin added via
the UI did not appear in /api/plugins (the view switcher) until a restart. Refresh
the registry from the new general_settings whenever the plugins field is updated.

While here, type the registry as dict[str, PluginConfig] instead of raw dicts so
list_plugins and plugin_proxy access typed attributes.

Fix the Plugin Key field copy: it is optional and only used to authenticate
litellm's server-side reverse proxy to a plugin's own backend
(/plugin-proxy/<name>/*). It is not involved in iframe auth, which forwards the
user's litellm token. Plugins that use the forwarded token leave it blank.

* fix: regenerate schema.d.ts with PluginConfig type and updated auth-token endpoint

* fix: use CI-compatible schema base for plugin entries

* fix(plugins): load DB-persisted plugins on startup

Plugins added through the admin UI are saved to DB general_settings, but the
registry only initialised from the YAML config at boot, so UI-added plugins
disappeared from the view switcher after a restart (the Plugins table still
listed them since it reads the DB directly). Refresh the registry from the DB
general_settings when it is merged in at startup.

* fix: add PluginConfig schema, plugins field, fix list_plugins return type

* fix: correct PluginConfig and plugins field positions in schema

* fix: correct plugins field position in schema (after pass_through_endpoints)

* fix: update PluginConfig.plugin_key description to match _types.py source

* fix: move plugins field after pass_through_request_timeout (correct alphabetical position)

* fix: redact plugin_key in config/field/info response

Veria medium: proxy_admin_viewer could read plugin_key via
GET /config/field/info?field_name=plugins. Now plugin_key is
replaced with *** in the response regardless of caller role.
The credential is only usable server-side.

* fix(security): correct plugin docs salt-key guidance, drop iframe clipboard-read

Address the two open Veria findings on the plugin architecture.

The plugin docs told external services to decrypt the iframe auth payload
with the proxy's LITELLM_SALT_KEY directly. That is both insecure and wrong:
the running code derives a per-plugin key as HMAC-SHA256(LITELLM_SALT_KEY,
plugin_name) and ships only a short-lived identity claim with no litellm
bearer token. Sharing the master salt would let a compromised plugin decrypt
any litellm secret recovered from a dump or backup. Rewrite the doc to match
the implementation: the proxy computes the per-plugin key once and provisions
it as a dedicated secret, the plugin validates the claim's audience and 30s
TTL, and LITELLM_SALT_KEY never leaves the proxy. Also refresh the now-stale
module and UI comments that still described the old shared-key token flow.

Drop clipboard-read from the plugin iframe's allow attribute so an untrusted
plugin can no longer read the user's clipboard; clipboard-write is retained.

* fix(ci): modernize PluginConfig typing, refresh budget baselines via merge

* fix(plugins): close iframe auth race and empty-plugins mode fallback

Address the two open Greptile behavioral findings.

The iframe auth handshake only posted the encrypted claim on the iframe's
`load` event. When the auth-token fetch resolved after the iframe had already
loaded, that listener never fired again and the plugin never received the
claim. Send the claim immediately as well as on subsequent loads so both
orderings are covered.

The plugin mode fallback guarded on a non-empty plugins list, so removing all
plugins left a user stranded on a stale mode with a blank iframe instead of
returning to the AI Gateway. Track a loaded flag and fall back to ai-gateway
once plugins have loaded whenever the stored mode is no longer registered,
including the empty-list case.

Add a PluginModeContext regression test covering the empty-list fallback and
the still-registered path.

* chore: re-trigger CI (GH Actions missed the prior head; re-run flaky live-API suites)

* fix(plugins): scope iframe auth claim to the active plugin

The iframe auth-token fetch omitted plugin_name, so the proxy always issued a
claim encrypted under the default plugin's per-plugin key. For any other active
plugin the iframe received a claim it could not decrypt and sign-in silently
broke, and because the cached claim was posted to whichever plugin was mounted,
a compromised iframe could replay the default plugin's claim. The active
plugin's name was also missing from the fetch effect's dependencies, so
switching plugins never refreshed the claim.

Request the claim with the active plugin's name, re-fetch when the active
plugin changes, and only deliver a claim while it still matches the mounted
plugin so one plugin's claim is never replayed to another.

* fix(plugins): never overwrite a stored plugin_key with its redaction placeholder

/config/field/info redacts every plugin_key to "***", so an admin editing a
plugin in the settings UI posted that placeholder straight back and the update
handler persisted "***" as the real credential, permanently destroying the key.

Preserve the stored credential on update: a blank or redacted plugin_key now
sources the existing key from the saved config, only a real value replaces it,
and a placeholder with no stored key is dropped rather than written. The edit
modal also starts the key field blank so an untouched save keeps the current
key, with the field labelled accordingly.

* fix(security): sandbox proxied plugin responses on the dashboard origin

The /plugin-proxy reverse proxy returned the plugin's body and content-type on
the litellm dashboard origin, so a compromised plugin could serve an HTML/JS
document that a proxy_admin navigates to and have it execute with the admin's
session against same-origin management APIs.

Force every proxied response inert: set Content-Security-Policy: sandbox (opaque
origin, scripts disabled) and X-Content-Type-Options: nosniff, applied after the
plugin's own headers so they cannot be overridden. The header construction moves
to a pure helper with a unit test covering the sandbox enforcement and the
existing wire/cookie header stripping.

* fix(plugins): recover to ai-gateway when the plugins fetch fails

The loaded flag was only set on a successful /api/plugins response, so when the
fetch failed a user with a plugin mode stored in localStorage stayed on the
blank plugin placeholder with no switcher to escape. Mark loaded in a finally
so the stored mode still falls back to ai-gateway on failure, and add a
regression test for the failed-fetch path.

* fix(security): never return plugin_key from /api/plugins

The plugin list endpoint returned the plaintext plugin_key to proxy_admin
callers, and the dashboard fetches /api/plugins on every load into React state,
so the credential was exposed to DevTools, memory snapshots, and any same-origin
script. The browser never uses the key; the proxy injects it server-side from
the registry and admin key management runs through the redacted
/config/field/info path. Drop plugin_key from the response for every caller and
update the regression test to assert it is never returned.

* chore(ui): regenerate schema.d.ts for updated list_plugins docstring

* fix(security): strip every litellm auth header before forwarding to plugins

The plugin reverse proxy only removed Authorization and x-api-key, but
user_api_key_auth also authenticates a caller via API-Key, x-goog-api-key,
Ocp-Apim-Subscription-Key, x-litellm-api-key, and any configured custom key
header. A malicious plugin could lure a proxy_admin into calling
/plugin-proxy/... with the litellm key in one of those headers; the request
authenticated locally and then forwarded the same key to the plugin, letting it
impersonate the admin.

Add a canonical SpecialHeaders.litellm_credential_header_names() that the auth
header enum is the single source for, and strip that whole set plus the live
general_settings.litellm_key_header_name from every forwarded request. New auth
headers added to SpecialHeaders are now stripped automatically. Regression tests
cover each credential header, the custom configured header, and the canonical
list's contents.

)

* feat(sandbox): code interpreter interceptor on the Responses API

Route OpenAI's code interpreter to a configured sandbox (e2b) instead of
OpenAI's container, with no client change. A client calls /v1/responses with
a code_interpreter tool; the interceptor converts it to a function tool so the
model emits the code, runs that code in the sandbox via the phase 1 primitive,
feeds the result back, and lets the agentic loop continue.

Reuses the existing agentic-loop hooks (no new hook methods). The anthropic
agentic caller _call_agentic_completion_hooks gains an api_surface argument and
a responses execute path (_execute_responses_agentic_plan re-calls aresponses);
the responses handler invokes it after transforming the response. Web search and
compression interceptors are untouched.

Adds an api_base passthrough to the sandbox SDK and a sandbox_tools registry the
proxy parses, so the interceptor resolves a named tool to provider/key/base.

v0 limitation: no file upload or download yet; stdout and inline results flow
back, attaching input files and downloading produced files do not.

* feat(sandbox): re-inject code_interpreter_call so the response matches OpenAI

The native OpenAI Responses code interpreter returns a code_interpreter_call
output item (id, type, status, code, container_id, outputs) alongside the
message. The interceptor now re-injects an equivalent item via
async_post_agentic_loop_response_hook so a client gets the same response shape
whether the code ran in OpenAI's container or the sandbox: build_plan records
the executed code and the container id per call, and the post hook inserts the
code_interpreter_call before the message in the final response output.

* feat(sandbox): support streaming for the code interpreter interceptor

A stream:true /v1/responses request with code_interpreter previously broke,
because the agentic loop only runs on the non-streaming responses path. The
interceptor now forces stream=False in the pre-call hook (so the loop runs in
the sandbox) and the responses handler wraps the completed response back into a
synthetic stream via MockResponsesAPIStreamingIterator, so the caller still gets
SSE. The follow-up call and nested wrapping are guarded by stripping the
converted-stream flag from the follow-up request and only wrapping at the
outermost call (agentic loop depth 0).

* fix(lint): use builtin generics in code interpreter interceptor to satisfy UP006 budget

* fix(code-interpreter): gate sandbox execution, delete sandboxes, harden registry

Gate the agentic loop on a server-set interception marker and re-check
provider scope so an authenticated caller cannot trigger sandbox code
execution by naming their own function tool litellm_code_execution; the
marker is stripped from client requests at the proxy boundary and only
set when the pre-call hook actually converts a native code_interpreter
tool. Delete the sandbox once the final response is assembled instead of
leaking it until its own timeout, and prune expired cache entries by
deleting their containers too. Resolve sandbox params once at create time
and reuse them for run and delete. Clear the sandbox-tool registry before
re-registering so stale tools do not survive a config reload.

* fix(lint): use PEP 604 X | None unions to satisfy UP045 budget

* test(code-interpreter): cover execution-error and unparseable-argument tool-call paths

* fix(code-interpreter): rewrite forced code_interpreter tool_choice to the function tool

* test(sandbox): cover sandbox-tool registry resolution, reload clearing, and secret lookup

* test(code-interpreter): cover dict-shaped responses and object-attribute tool-call detection

* fix(proxy): strip client-supplied _code_interpreter_interception_converted_stream

A client could inject the converted-stream marker to force the completed
response to be re-wrapped as a synthetic SSE stream it never requested.
Add it to the untrusted root control fields alongside the other agentic
loop markers so the proxy strips it at the request boundary.

* fix(code-interpreter): isolate sandboxes by server-minted key and clear registry on tool removal

Key the per-request sandbox cache on a server-minted random token instead
of the caller-controlled litellm_call_id (sourced from the x-litellm-call-id
header). Two concurrent requests that send a colliding call id can no longer
share a sandbox container and read each other's code or files. The token is
minted in the pre-call hook when interception activates, stripped from client
requests at the proxy boundary, and survives the server-driven followups so a
single request still reuses one sandbox across the agentic loop.

Register sandbox tools unconditionally with an empty-list fallback so a config
reload that removes sandbox_tools clears the previously registered credentials
instead of leaving them resolvable in the process.

* refactor(sandbox): swap the tool registry atomically on reload

Build the new registry and rebind it in one assignment instead of clearing
then repopulating in place, so a concurrent resolve_sandbox_tool can never
observe a transiently empty or half-populated registry during a config
reload. clear_sandbox_tools now delegates to register_sandbox_tools([]).

* fix(code-interpreter): cap caller loop limit and emit OpenAI-shaped outputs

Strip max_agentic_loops at the proxy request boundary so an authenticated
caller cannot raise the agentic-loop ceiling to drive many upstream model
calls and sandbox executions from a single request; the loop stays bounded
by the server default.

Populate the re-injected code_interpreter_call.outputs with an OpenAI-shaped
logs array ([{"type": "logs", "logs": stdout}], or [] when there is no
stdout) instead of None, so clients that iterate over outputs or validate the
response through the OpenAI SDK's Pydantic model do not break.

…edpyright can analyze it (#30802)

…an analyze it (#30793)