fix(galileo): support hosted v2 spans API and string output extraction#28771
Merged
Conversation
Use GALILEO_API_KEY with /v2/projects/{id}/spans for Galileo Cloud,
keep legacy observe/ingest for username/password deployments, and
extract assistant content as a string instead of a message dict.
Co-authored-by: Cursor <[email protected]>
Contributor
Greptile SummaryThis PR extends the Galileo Observe integration with two complementary fixes: routing hosted Galileo Cloud logs through the new v2 spans endpoint (
Confidence Score: 5/5Safe to merge — the Galileo integration changes are additive and non-blocking, the management utils fix correctly preserves exception propagation, and all test paths use mocks. The dual-path dispatch logic is straightforward and well-tested. The snapshot-then-count pattern in flush_in_memory_records is correct because both captures happen before any await. No defects found. No files require special attention.
|
| Filename | Overview |
|---|---|
| litellm/integrations/galileo.py | Major refactor: adds v2 cloud spans API path with API-key auth, converts sync login to async, fixes output extraction to return plain string content, caps in-memory buffer at 1000 records, and makes flush non-blocking. |
| litellm/proxy/management_helpers/utils.py | Wraps OTEL span emission in a try/except so failures there cannot swallow the original management-endpoint exception before re-raise. |
| tests/test_litellm/integrations/test_galileo.py | New mock-only test file covering v2 URL/headers, span structure, multi-role input, output extraction variants, 201 clearing, 401 header reset, and end-to-end flush. |
Reviews (5): Last reviewed commit: "test(galileo): expand v2 coverage for co..." | Re-trigger Greptile
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
1 task
Collaborator
Author
|
Documentation CI fix: BerriAI/litellm-docs#218 — merge litellm-docs PR first, then re-run the |
Use async httpx for enterprise login to avoid blocking the event loop, preserve multi-turn messages in v2 span input, and clean up tests. Co-authored-by: Cursor <[email protected]>
Collaborator
Author
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 3 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for all 3 issues found in the latest run.
- ✅ Fixed: Timezone check misses negative UTC offset timestamps
- Replaced the substring check with a regex matching trailing
Zor[+-]HH:MM/[+-]HHMMoffsets, so negative UTC offsets are detected and not double-suffixed withZ.
- Replaced the substring check with a regex matching trailing
- ✅ Fixed: Success check only accepts HTTP 200, ignoring other 2xx
- Changed the success check in
flush_in_memory_recordsfrom== 200to200 <= status_code < 300so any 2xx response (e.g., 201/202) clears the in-memory records.
- Changed the success check in
- ✅ Fixed:
json.dumpsfails on Pydantic ImageObject list- Passed
default=strtojson.dumpsfor the ImageResponse data list so PydanticImageObjectinstances serialize without raisingTypeError.
- Passed
Preview (2a0436419f)
diff --git a/litellm/integrations/galileo.py b/litellm/integrations/galileo.py
--- a/litellm/integrations/galileo.py
+++ b/litellm/integrations/galileo.py
@@ -1,18 +1,25 @@
+import json
import os
-from typing import Any, Dict, List, Optional
+import re
+from typing import Any, Dict, List, Optional, Tuple
from pydantic import BaseModel, Field
import litellm
from litellm._logging import verbose_logger
from litellm.integrations.custom_logger import CustomLogger
+from litellm.litellm_core_utils.prompt_templates.common_utils import (
+ convert_content_list_to_str,
+ get_content_from_model_response,
+)
from litellm.llms.custom_httpx.http_handler import (
get_async_httpx_client,
httpxSpecialProvider,
)
+GALILEO_CLOUD_API_BASE_URL = "https://api.galileo.ai"
-# from here: https://docs.rungalileo.io/galileo/gen-ai-studio-products/galileo-observe/how-to/logging-data-via-restful-apis#structuring-your-records
+
class LLMResponse(BaseModel):
latency_ms: int
status_code: int
@@ -37,65 +44,184 @@
def __init__(self) -> None:
self.in_memory_records: List[dict] = []
self.batch_size = 1
- self.base_url = os.getenv("GALILEO_BASE_URL", None)
- self.project_id = os.getenv("GALILEO_PROJECT_ID", None)
+ self.api_key = os.getenv("GALILEO_API_KEY")
+ self.project_id = os.getenv("GALILEO_PROJECT_ID")
+ self.log_stream_id = os.getenv("GALILEO_LOG_STREAM_ID")
+ self.username = os.getenv("GALILEO_USERNAME")
+ self.password = os.getenv("GALILEO_PASSWORD")
+ self.base_url = self._normalize_base_url(os.getenv("GALILEO_BASE_URL"))
+ if self.api_key and not self.base_url:
+ self.base_url = GALILEO_CLOUD_API_BASE_URL
+ self.use_v2_api = bool(self.api_key)
self.headers: Optional[Dict[str, str]] = None
self.async_httpx_handler = get_async_httpx_client(
llm_provider=httpxSpecialProvider.LoggingCallback
)
- pass
- def set_galileo_headers(self):
- # following https://docs.rungalileo.io/galileo/gen-ai-studio-products/galileo-observe/how-to/logging-data-via-restful-apis#logging-your-records
+ @staticmethod
+ def _normalize_base_url(base_url: Optional[str]) -> Optional[str]:
+ if base_url:
+ return base_url.rstrip("/")
+ return None
- headers = {
- "accept": "application/json",
- "Content-Type": "application/x-www-form-urlencoded",
- }
- galileo_login_response = litellm.module_level_client.post(
+ def _is_configured(self) -> bool:
+ if not self.project_id or not self.base_url:
+ return False
+ if self.use_v2_api:
+ return bool(self.api_key)
+ return bool(self.username and self.password)
+
+ async def async_set_galileo_headers(self) -> None:
+ galileo_login_response = await self.async_httpx_handler.post(
url=f"{self.base_url}/login",
- headers=headers,
+ headers={
+ "accept": "application/json",
+ "Content-Type": "application/x-www-form-urlencoded",
+ },
data={
- "username": os.getenv("GALILEO_USERNAME"),
- "password": os.getenv("GALILEO_PASSWORD"),
+ "username": self.username,
+ "password": self.password,
},
)
-
+ galileo_login_response.raise_for_status()
access_token = galileo_login_response.json()["access_token"]
-
self.headers = {
"accept": "application/json",
"Content-Type": "application/json",
"Authorization": f"Bearer {access_token}",
}
- def get_output_str_from_response(self, response_obj, kwargs):
- output = None
- if response_obj is not None and (
- kwargs.get("call_type", None) == "embedding"
- or isinstance(response_obj, litellm.EmbeddingResponse)
+ async def _ensure_headers(self) -> bool:
+ if self.headers is not None:
+ return True
+
+ if self.use_v2_api:
+ if not self.api_key:
+ return False
+ self.headers = {
+ "accept": "application/json",
+ "Content-Type": "application/json",
+ "Galileo-API-Key": self.api_key,
+ }
+ return True
+
+ if not (self.username and self.password and self.base_url):
+ return False
+
+ try:
+ await self.async_set_galileo_headers()
+ return True
+ except Exception as e:
+ verbose_logger.debug("Galileo Logger: failed to authenticate: %s", e)
+ return False
+
+ @staticmethod
+ def _galileo_input_messages(
+ messages: Optional[List[Any]], input_text: str
+ ) -> List[Dict[str, str]]:
+ if not messages:
+ return [{"role": "user", "content": input_text}]
+
+ galileo_messages: List[Dict[str, str]] = []
+ for message in messages:
+ if not isinstance(message, dict):
+ continue
+ role = message.get("role")
+ if not role:
+ continue
+ galileo_messages.append(
+ {
+ "role": str(role),
+ "content": convert_content_list_to_str(message=message),
+ }
+ )
+
+ if galileo_messages:
+ return galileo_messages
+ return [{"role": "user", "content": input_text}]
+
+ @staticmethod
+ def _record_to_v2_span(record: Dict[str, Any]) -> Dict[str, Any]:
+ created_at = record.get("created_at", "")
+ if created_at and not re.search(r"(Z|[+-]\d{2}:?\d{2})$", created_at):
+ created_at = f"{created_at}Z"
+
+ span: Dict[str, Any] = {
+ "type": "llm",
+ "name": record.get("node_type", "litellm"),
+ "created_at": created_at,
+ "input": GalileoObserve._galileo_input_messages(
+ record.get("messages"), record.get("input_text", "")
+ ),
+ "output": {
+ "role": "assistant",
+ "content": record.get("output_text", ""),
+ },
+ "status_code": record.get("status_code", 200),
+ "model": record.get("model"),
+ "metrics": {
+ "duration_ns": int(record.get("latency_ms", 0)) * 1_000_000,
+ "num_input_tokens": record.get("num_input_tokens"),
+ "num_output_tokens": record.get("num_output_tokens"),
+ },
+ }
+ if record.get("tags"):
+ span["tags"] = record["tags"]
+ return span
+
+ def _get_ingest_request(self) -> Optional[Tuple[str, Dict[str, Any]]]:
+ if not self.base_url or not self.project_id:
+ return None
+
+ if self.use_v2_api:
+ payload: Dict[str, Any] = {
+ "spans": [
+ self._record_to_v2_span(record) for record in self.in_memory_records
+ ],
+ "reliable": False,
+ }
+ if self.log_stream_id:
+ payload["log_stream_id"] = self.log_stream_id
+ return (
+ f"{self.base_url}/v2/projects/{self.project_id}/spans",
+ payload,
+ )
+
+ return (
+ f"{self.base_url}/projects/{self.project_id}/observe/ingest",
+ {"records": self.in_memory_records},
+ )
+
+ def get_output_str_from_response(
+ self, response_obj: Any, kwargs: Dict[str, Any]
+ ) -> Optional[str]:
+ if response_obj is None:
+ return None
+ if kwargs.get("call_type", None) == "embedding" or isinstance(
+ response_obj, litellm.EmbeddingResponse
):
- output = None
- elif response_obj is not None and isinstance(
- response_obj, litellm.ModelResponse
- ):
- output = response_obj["choices"][0]["message"].json()
- elif response_obj is not None and isinstance(
- response_obj, litellm.TextCompletionResponse
- ):
- output = response_obj.choices[0].text
- elif response_obj is not None and isinstance(
- response_obj, litellm.ImageResponse
- ):
- output = response_obj["data"]
+ return None
+ if isinstance(response_obj, litellm.TextCompletionResponse):
+ return response_obj.choices[0].text
+ if isinstance(response_obj, litellm.ImageResponse):
+ return json.dumps(response_obj["data"], default=str)
+ if isinstance(response_obj, (litellm.ModelResponse, dict)):
+ return get_content_from_model_response(response_obj)
+ return None
- return output
-
async def async_log_success_event(
self, kwargs: Any, response_obj: Any, start_time: Any, end_time: Any
):
verbose_logger.debug("On Async Success")
+ if not self._is_configured():
+ verbose_logger.debug(
+ "Galileo Logger: skipping flush — set GALILEO_PROJECT_ID and "
+ "either GALILEO_API_KEY (hosted) or GALILEO_USERNAME/GALILEO_PASSWORD "
+ "(enterprise Observe)."
+ )
+ return
+
_latency_ms = int((end_time - start_time).total_seconds() * 1000)
_call_type = kwargs.get("call_type", "litellm")
input_text = litellm.utils.get_formatted_prompt(
@@ -125,25 +251,49 @@
), # timestamp str constructed in "%Y-%m-%dT%H:%M:%S" format
)
- # dump to dict
request_dict = request_record.model_dump()
+ messages = kwargs.get("messages")
+ if messages:
+ request_dict["messages"] = messages
self.in_memory_records.append(request_dict)
if len(self.in_memory_records) >= self.batch_size:
await self.flush_in_memory_records()
async def flush_in_memory_records(self):
- verbose_logger.debug("flushing in memory records")
- response = await self.async_httpx_handler.post(
- url=f"{self.base_url}/projects/{self.project_id}/observe/ingest",
- headers=self.headers,
- json={"records": self.in_memory_records},
- )
+ if not self.in_memory_records:
+ return
- if response.status_code == 200:
+ ingest_request = self._get_ingest_request()
+ if ingest_request is None:
verbose_logger.debug(
- "Galileo Logger:successfully flushed in memory records"
+ "Galileo Logger: missing GALILEO_BASE_URL or GALILEO_PROJECT_ID"
)
+ return
+
+ if not await self._ensure_headers():
+ verbose_logger.debug("Galileo Logger: could not set request headers")
+ return
+
+ url, payload = ingest_request
+ verbose_logger.debug("flushing in memory records to %s", url)
+
+ try:
+ response = await self.async_httpx_handler.post(
+ url=url,
+ headers=self.headers,
+ json=payload,
+ )
+ except Exception as e:
+ verbose_logger.debug(
+ "Galileo Logger: failed to flush in memory records: %s", e
+ )
+ return
+
+ if 200 <= response.status_code < 300:
+ verbose_logger.debug(
+ "Galileo Logger: successfully flushed in memory records"
+ )
self.in_memory_records = []
else:
verbose_logger.debug("Galileo Logger: failed to flush in memory records")
diff --git a/tests/test_litellm/integrations/test_galileo.py b/tests/test_litellm/integrations/test_galileo.py
new file mode 100644
--- /dev/null
+++ b/tests/test_litellm/integrations/test_galileo.py
@@ -1,0 +1,115 @@
+import os
+import sys
+from unittest.mock import AsyncMock, patch
+
+import pytest
+
+sys.path.insert(0, os.path.abspath("../.."))
+
+from litellm.integrations.galileo import GalileoObserve
+from litellm.types.utils import Choices, Message, ModelResponse
+
+
+@pytest.fixture
+def galileo_v2_env(monkeypatch):
+ monkeypatch.setenv("GALILEO_API_KEY", "test-api-key")
+ monkeypatch.setenv("GALILEO_PROJECT_ID", "86ff8ebe-a297-4134-b167-748bdd8d2c20")
+ monkeypatch.setenv("GALILEO_LOG_STREAM_ID", "76c4ea50-8aa3-4771-a0d7-8567b112210f")
+ monkeypatch.setenv("GALILEO_BASE_URL", "https://api.galileo.ai")
+
+
+@pytest.mark.asyncio
+async def test_galileo_v2_ingest_url_and_headers(galileo_v2_env):
+ logger = GalileoObserve()
+ logger.in_memory_records = [
+ {
+ "latency_ms": 100,
+ "status_code": 200,
+ "input_text": "hi",
+ "output_text": "hello",
+ "node_type": "acompletion",
+ "model": "gpt-5.2",
+ "num_input_tokens": 1,
+ "num_output_tokens": 2,
+ "created_at": "2026-05-25T12:00:00",
+ }
+ ]
+
+ url, payload = logger._get_ingest_request()
+ assert (
+ url
+ == "https://api.galileo.ai/v2/projects/86ff8ebe-a297-4134-b167-748bdd8d2c20/spans"
+ )
+ assert payload["log_stream_id"] == "76c4ea50-8aa3-4771-a0d7-8567b112210f"
+ assert payload["spans"][0]["type"] == "llm"
+ assert payload["spans"][0]["output"]["content"] == "hello"
+
+ assert await logger._ensure_headers() is True
+ assert logger.headers["Galileo-API-Key"] == "test-api-key"
+
+
+def test_galileo_v2_span_preserves_message_roles(galileo_v2_env):
+ record = {
+ "latency_ms": 1,
+ "status_code": 200,
+ "input_text": "fallback",
+ "output_text": "ok",
+ "node_type": "acompletion",
+ "model": "gpt-5.2",
+ "num_input_tokens": 0,
+ "num_output_tokens": 0,
+ "created_at": "2026-05-25T12:00:00",
+ "messages": [
+ {"role": "system", "content": "be helpful"},
+ {"role": "user", "content": "hello"},
+ ],
+ }
+ span = GalileoObserve._record_to_v2_span(record)
+ assert span["input"] == [
+ {"role": "system", "content": "be helpful"},
+ {"role": "user", "content": "hello"},
+ ]
+
+
+def test_galileo_output_text_from_model_response(galileo_v2_env):
+ logger = GalileoObserve()
+ response = ModelResponse(
+ choices=[
+ Choices(
+ message=Message(
+ content="assistant reply",
+ role="assistant",
+ annotations=[],
+ )
+ )
+ ]
+ )
+
+ output = logger.get_output_str_from_response(response, {"call_type": "acompletion"})
+ assert output == "assistant reply"
+
+
+@pytest.mark.asyncio
+async def test_galileo_flush_swallows_http_errors(galileo_v2_env):
+ logger = GalileoObserve()
+ logger.in_memory_records = [
+ {
+ "latency_ms": 1,
+ "status_code": 200,
+ "input_text": "a",
+ "output_text": "b",
+ "node_type": "acompletion",
+ "model": "gpt-5.2",
+ "num_input_tokens": 0,
+ "num_output_tokens": 0,
+ "created_at": "2026-05-25T12:00:00",
+ }
+ ]
+
+ with patch.object(
+ logger.async_httpx_handler, "post", new_callable=AsyncMock
+ ) as mock_post:
+ mock_post.side_effect = Exception("404 Not Found")
+ await logger.flush_in_memory_records()
+
+ assert len(logger.in_memory_records) == 1You can send follow-ups to the cloud agent here.
…mageObject serialization Co-authored-by: Yassin Kortam <[email protected]>
|
|
Use response.is_success so 201 Created clears in_memory_records and avoids duplicate span submissions on subsequent flushes. Co-authored-by: Cursor <[email protected]>
Collaborator
Author
Co-authored-by: Cursor <[email protected]>
* fix(proxy): Bedrock Knowledge Base pass-through: preserve SigV4 headers and signed request body (#27526) * Fix Bedrock KB pass-through SigV4 headers and signed body Coerce botocore HeadersDict to a dict for pass-through routes. When forward_headers is true, drop request headers that collide case-insensitively with signed headers so client Bearer auth does not shadow AWS SigV4. Send prepped.body as raw content so the outbound payload matches the signature after logging hooks mutate the parsed dict. Co-authored-by: Cursor <[email protected]> * Simplify pass-through raw body handling Read the SigV4-signed bytes directly from request.state inside pass_through_request instead of threading a custom_raw_body argument through three functions. Helper methods are restored to their original signatures, and the new branch lives in one place at each httpx call site. Co-authored-by: Cursor <[email protected]> * Harden pass-through raw body read from request.state Guard missing request.state (test fixtures) and ignore non-bytes/str values so MagicMock does not trigger the SigV4 raw-body path. Co-authored-by: Cursor <[email protected]> * Test pass_through_request state_raw_body uses httpx content= Cover non-streaming (async_client.request) and streaming (build_request) paths so SigV4 bytes on request.state are not replaced by json= of a hook-mutated dict. Co-authored-by: Cursor <[email protected]> --------- Co-authored-by: Cursor <[email protected]> * chore(tests): migrate Bedrock CI to AWS account 941277531214 (#28728) * chore(tests): migrate Bedrock CI from AWS account 888602223428 to 941277531214 The original account (888602223428) was put under a security restriction by AWS after a root access key leaked in a PR comment. While that account works its way through the AWS Support unlock process, Bedrock-touching CI tests have been migrated to a fresh account (941277531214). Changes: - Replace 26 hardcoded references to 888602223428 with 941277531214 across 8 files (provisioned-model ARNs, imported-model ARNs, AgentCore runtime ARNs, batch execution role ARN, and example proxy config). - The provisioned-model and imported-model ARNs are referenced only from mocked unit tests — no AWS resources to recreate. - The batch execution IAM role has been recreated in the new account with the same name and equivalent permissions. - The two AgentCore runtimes (hosted_agent_r9jvp-3ySZuRHjLC, hosted_agent_13sf6-cALnp38iZD) are being recreated in the new account under the same names — see tools/agentcore-deploy/ in a follow-up. CircleCI env vars AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION_NAME were updated separately via the CircleCI API to point at the new account. Smoke-tested locally against the new account: aws bedrock-runtime converse --region us-west-2 \ --model-id us.anthropic.claude-sonnet-4-5-20250929-v1:0 \ --messages '[{"role":"user","content":[{"text":"ping"}]}]' → 200, model returned 'pong' Co-Authored-By: Claude Opus 4.7 <[email protected]> * chore(tests): refresh AgentCore ARN suffixes to match newly-deployed runtimes The first migration commit replaced just the account ID, but AgentCore auto-assigns a random 10-char suffix to every runtime on creation — we can't reuse the original suffixes (`3ySZuRHjLC`, `cALnp38iZD`) in the new account. Updated the AgentCore-runtime ARNs in the three files that reference real runtime IDs (not the mock-based unit-test ARNs). Deployed runtimes: arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_r9jvp-Rq79QFC2fp arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_13sf6-4046UzHSwy Both runtimes are status=READY and pass a smoke invoke: $ aws bedrock-agentcore invoke-agent-runtime --agent-runtime-arn ... --payload '{"prompt":"ping"}' → 200, {"result": "echo: ping"} The agent is a minimal echo (see /tmp/agentcore_deploy/agent.py for the deploy artifacts). Tests that only verify the SDK wiring will pass; if any test asserts on agent output content, swap the echo for the real agent. Co-Authored-By: Claude Opus 4.7 <[email protected]> * chore(tests): point Bedrock batch tests at new-account S3 bucket The account migration (888602223428 -> 941277531214) was a flat account-ID swap, which only rewrites ARNs that embed the account number. S3 bucket names carry no account ID, so the live Bedrock batch tests still uploaded to `litellm-proxy` — a bucket that lives in the old account. S3 names are globally unique, and the old account still holds that name, so it can't be recreated in the new account. Rename to `litellm-proxy-941277531214` (account-ID suffix guarantees global uniqueness). The bucket must be created in 941277531214 and the batch execution role granted s3:GetObject/PutObject/ListBucket on it before this job is run in CI. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * chore(tests): point live S3 logging test at new-account bucket Same account-ID-free blind spot as the batch bucket: `load-testing-oct` lives in the old account and its name can't be reused globally. The `logging_testing` CI job is wired into the workflow and runs test_basic_s3_logging, which uploads to this bucket with the CI env creds, then lists and deletes objects — a live dependency. Rename to `load-testing-oct-941277531214`. The bucket must exist in the new account with the CI IAM principal granted s3:PutObject/GetObject/ListBucket/DeleteObject before this job runs. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * chore(tests): repoint Bedrock guardrail IDs to new-account guardrails The migration left guardrail IDs untouched (no account ID in them), so all live guardrail tests failed with "guardrail identifier or version does not exist" against 941277531214. Recreated both guardrails in the new account and updated the hardcoded IDs: - wf0hkdb5x07f -> zgkmukebruil (PII mask: PHONE + CREDIT_DEBIT_CARD, with explicit inputAction=ANONYMIZE so masking applies to INPUT, which is the source litellm's moderation hook sends) - ff6ujrregl1q -> 4w3d1di3snt5 (blocks "coffee"; blocked message set to the exact string the tests assert on) Updated test_bedrock_guardrails.py, otel_test_config.yaml, and the guardrailConfig in test_bedrock_completion.py. Verified locally: the 5 previously-failing guardrail tests now pass. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * test(bedrock): migrate legacy models to current inference profiles The new CI account (941277531214) cannot invoke legacy Bedrock models (AWS gates them: "marked by provider as Legacy... not actively using in the last 30 days"). Migrated the live-call tests: - anthropic.claude-3-sonnet-20240229 -> us.anthropic.claude-sonnet-4-5-20250929-v1:0 - anthropic.claude-3-haiku-20240307 -> us.anthropic.claude-haiku-4-5-20251001-v1:0 Current Claude models on Bedrock require the us. inference-profile prefix (bare on-demand ids are rejected). cohere.command-r-plus has no working replacement (all Cohere is legacy- gated in the new account): swapped to claude-haiku-4-5 in provider- agnostic param lists. amazon.titan-image-generator skipped (no working replacement). Mocked/transformation/cost tests that reference the legacy strings are intentionally left unchanged. Verified live against the new account. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * test(bedrock): repoint SageMaker + Knowledge Base to new-account resources These referenced account-scoped resources by hardcoded id that only existed in the old account, so the migration's account-ID swap missed them. Recreated in 941277531214 and repointed: - SageMaker endpoint jumpstart-dft-hf-textgeneration1-mp-20240815-185614 -> litellm-ci-textgen (gpt2 on a TGI container, ml.g5.xlarge) - Bedrock Knowledge Base T37J8R4WTM -> LCYXFBR2TU (OpenSearch Serverless vector store + titan-embed-text-v2, seeded with a LiteLLM doc) Verified live: test_sagemaker.py (12 passed) and test_bedrock_knowledgebase_hook.py (12 passed). Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * test(reasoning_effort_grid): skip bedrock claude-opus-4-7 cells (not entitled on 941277531214) claude-opus-4-7 is listed in the new Bedrock CI account's foundation models but invoke is denied (AccessDeniedException: "not available for this account"). Bedrock access to the flagship Opus requires an AWS Sales request, not the self-serve model-access toggle, so it can't be enabled inline with the rest of the account migration. Add an optional `skip_reason` to ModelEntry and set it on the bedrock-claude-opus-4-7 entry; the grid test honors it via pytest.skip. Cell count (231) and route coverage are unchanged, so the structural asserts still pass. Restore coverage by deleting the one skip_reason line once access is granted. Co-Authored-By: Claude Opus 4.7 (1M context) <[email protected]> * test(bedrock): swap/skip legacy-gated models unavailable on new CI account The migrated AWS account (941277531214) cannot access several models that the old account could, so the remaining red CI jobs were hitting real Bedrock "Access denied / Legacy" and "account not authorized" errors: - image_gen: skip both Nova Canvas test classes (amazon.nova-canvas-v1:0 is legacy-gated), matching the existing titan skip. - batches: skip test_async_file_and_batch (Bedrock batch inference is not authorized on the new account; requires an AWS support case). - litellm_overhead: swap legacy claude-3-5-haiku for the active us.anthropic.claude-haiku-4-5 inference profile. - test_completion_claude_3_function_call: swap legacy claude-3-sonnet for the active us.anthropic.claude-sonnet-4-5 inference profile. https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa * test(bedrock): fix remaining e2e legacy-model + batch failures on new CI account - e2e_openai_endpoints: skip test_bedrock_batches_api (Bedrock batch inference is not authorized on account 941277531214) and migrate the missed s3_bucket_name in oai_misc_config.yaml to litellm-proxy-941277531214. - build_and_test: swap legacy bedrock claude-3-sonnet for the active us.anthropic.claude-sonnet-4-5 inference profile in the proxy structured output e2e test. https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa * test(bedrock): make opus-4-7 + batch cells fail loudly and mock image-gen (#28791) Replace the silent skips added for the new CI account with noisier behavior: - reasoning-effort grid: opus-4-7 cells now fail (when AWS creds are present) instead of skipping, so the missing entitlement stays visible in CI; they still skip when AWS creds are absent (local dev) - Bedrock batch inference tests: drop the skip so they run and fail until batch access is granted - Titan + Nova Canvas image-gen tests: mock the Bedrock HTTP call so the transform + cost-tracking path stays under test without live model access https://claude.ai/code/session_01MT7SWDnXUjv6e6EPG7BDjT Co-authored-by: Claude <[email protected]> * test(bedrock): use pytest.xfail for known-failing opus-4-7 cells Replace pytest.fail with pytest.xfail when a model has a fail_reason, so known-broken cells stay visible as XFAIL without keeping CI red. Co-authored-by: Yassin Kortam <[email protected]> --------- Co-authored-by: Mateo <[email protected]> Co-authored-by: Claude Opus 4.7 <[email protected]> Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: Yassin Kortam <[email protected]> * fix(otel): export SERVER span on management-endpoint success without http_request (#28794) Co-authored-by: Yassin Kortam <[email protected]> * chore(ci): merge dev branch (#28801) * chore(proxy): route path-dependent call sites through get_request_route Replace direct ``request.url.path`` reads in auth, ACL, routing, and audit-log decisions with ``get_request_route(request)`` — the helper already added in ``auth/auth_utils.py`` that returns the ASGI ``scope["path"]`` with ``root_path`` stripped. Starlette reconstructs ``url.path`` from the Host header; ``scope["path"]`` is uvicorn's parse of the request line and matches what FastAPI dispatches on, so it's the authoritative route for any decision that should agree with the actual handler. Sites: - _experimental/mcp_server/auth/user_api_key_auth_mcp.py - management_endpoints/mcp_management_endpoints.py - vector_store_endpoints/utils.py - pass_through_endpoints/pass_through_endpoints.py - auth/route_checks.py - litellm_pre_call_utils.py - spend_tracking/spend_management_endpoints.py - common_utils/http_parsing_utils.py - management_helpers/utils.py - health_endpoints/_health_endpoints.py Adds regression tests in tests/proxy_unit_tests/test_proxy_routes.py that construct a Request with scope["path"] set to a benign route and the Host header crafted so url.path would resolve differently; each site's decision is asserted against scope["path"]. * chore(proxy): make get_request_route imports lazy at call sites Move the ``from litellm.proxy.auth.auth_utils import get_request_route`` imports added in the prior commit back to the function bodies that use them. The module-level form participates in a long-standing import cycle through ``auth_utils -> _types -> ...`` and was flagged by CodeQL on the PR; the lazy form matches the pattern the proxy already uses for ``user_api_key_auth`` and related helpers elsewhere in these files. Also drop the ``RouteChecks._is_assistants_api_request`` delegation in ``_get_metadata_variable_name`` introduced in the prior commit — the delegation pulled ``RouteChecks`` into the same cycle, and the call site reuses the resolved route for its other branches, so inlining the substring check is both cycle-free and avoids a redundant second ``get_request_route`` call. Comment in test_proxy_routes.py acknowledges that the two MCP table entries exercise ``get_request_route`` directly rather than the full production handler (which needs ASGI scope + MCP state to invoke). --------- Co-authored-by: shin-berri <[email protected]> Co-authored-by: user <[email protected]> * chore(ci): merge dev branch (#28657) * feat(dashboard): navbar hierarchy + Agent Platform notifications (#27543) * feat(dashboard): refine navbar zones and Agent Platform notice Restructure the admin navbar for production users: clear product vs community vs personal columns with vertical dividers, icon-only Slack/GitHub in a shared chip, and Docs/Blog typography aligned on an 8px rhythm. Add a notifications bell with popover linking to the LiteLLM Agent Platform repo and optional mark-as-read persistence. Promote the account control with initials avatar, single-line display name, and navDisplayName mapping for placeholder user ids (e.g. default_user_id). Co-authored-by: Cursor <[email protected]> * fix(dashboard): address PR review — AntD buttons, public page guard, dedupe regex - Replace raw <button> with AntD Button in BlogDropdown, NotificationsBell, UserDropdown, and test mock - Guard NotificationsBell + container behind !isPublicPage to avoid rendering on public pages - Remove redundant equality checks in navDisplayName (regex already covers them) - Remove unused `lower` variable after simplification Co-Authored-By: Claude Opus 4.6 <[email protected]> --------- Co-authored-by: Cursor <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]> Co-authored-by: yuneng-jiang <[email protected]> * fix(dashboard): drop dead useHealthReadiness import in navbar The module was removed in #27896 (replaced by useHealthReadinessDetails), but the import survived the rebase. The symbol is unused — only useHealthReadinessDetails is consumed in the file. Removing the dead import unblocks the UI TypeScript build. * fix(dashboard): align CommunityEngagementButtons test with icon-only aria-labels The component was refactored to an icon-only chip with aria-label='LiteLLM on GitHub' (squash #27543), but the test still asserted /star us on github/i. Update the query to match the rendered accessible name. * refactor(dashboard): drop unused props from NavbarProps The navbar refactor moved user identity + dark-mode state to internal hooks (useAuthorized, useWorker), but the NavbarProps interface still declared userID, userEmail, userRole, premiumUser, isDarkMode, and toggleDarkMode as required, forcing every caller to thread them through. Drop them from the interface and all four call sites (page.tsx, (dashboard)/layout.tsx, public_model_hub.tsx, navbar.test.tsx). Also shrinks the destructure in layout.tsx so the now-unused locals stop being pulled out of useAuthorized(). * refactor(dashboard): use useSyncExternalStore for NotificationsBell dismiss flag Reads/writes of the litellmHideAgentPlatformBanner key were done directly inside NotificationsBell via a useEffect + useState pair. Every other localStorage-backed flag in the dashboard (Disable ShowPrompts, DisableBouncingIcon, DisableShowNewBadge, DisableUsageIndicator, DisableBlogPosts) is wrapped in a useSyncExternalStore hook over localStorageUtils so all mounted components stay in sync. Extract useHideAgentPlatformBanner to follow the same shape, swap NotificationsBell to consume it, and add a regression test that two sibling bells stay in sync without a remount when one is dismissed. * refactor: mask credential fields in proxy settings GET responses (#28682) * refactor: mask credential fields in proxy settings GET responses Brings SSO settings, cache settings, and the email/Slack alerting view in /get/config/callbacks in line with the HashiCorp Vault config-override pattern, so persisted credentials are not transported back to the UI in plaintext. * refactor: harden short-value masking and hoist alerting var constant Closes two review observations: - mask_sensitive_keys now replaces short values (below the visible prefix+suffix length) with an all-mask string instead of returning them unchanged, so a 1-7 character credential is no longer round-tripped verbatim. - _ALERTING_SENSITIVE_VARS is moved out of get_config() to a module-level constant, matching the analogous _SSO_SENSITIVE_FIELDS and _CACHE_SENSITIVE_FIELDS in the SSO and cache endpoint files. --------- Co-authored-by: Krrish Dholakia <[email protected]> Co-authored-by: Cursor <[email protected]> Co-authored-by: Claude Opus 4.6 <[email protected]> * fix(ui): show 2-decimal precision for max_budget on key overview (#28809) The Key Info Overview tab's Spend card truncated sub-dollar budgets to "$0" because formatNumberWithCommas defaults to 0 decimals. The Settings tab passes 2; align the overview so a $0.10 budget renders as "$0.10". Resolves LIT-2845 * feat(proxy): allow `llm_api_routes` virtual keys to list MCP servers (#28442) * feat(proxy): allow llm_api_routes virtual keys to list MCP servers Add a new `mcp_discovery_routes` group (GET /v1/mcp/server and GET /v1/mcp/server/{server_id}) and include it in `llm_api_routes` so that virtual keys configured with `allowed_routes=["llm_api_routes"]` can discover the MCP servers they have access to. Previously these calls failed with 'Virtual key is not allowed to call this route. Only allowed to call routes: [llm_api_routes]'. The GET handlers already sanitize the response for restricted virtual keys via `_sanitize_mcp_server_list_for_virtual_key`, stripping credential-bearing fields (url, headers, env). Write methods (POST/PUT/DELETE) on the same paths remain gated by the existing handler-level admin role checks. The new discovery list is intentionally kept OUT of `mcp_inference_routes`, so `is_llm_api_route()` still returns False for these paths — this preserves the existing contract that DISABLE_LLM_API_ENDPOINTS must not block the Admin UI from listing MCP servers. Co-authored-by: ryan-crabbe-berri <[email protected]> * refactor(proxy): make MCP discovery carve-out method-aware Replace the `mcp_discovery_routes` group in `llm_api_routes` with a method-aware special case inside `is_virtual_key_allowed_to_call_route`. Virtual keys with allowed_routes=["llm_api_routes"] are now permitted to call only GET /v1/mcp/server and GET /v1/mcp/server/{server_id} — non-GET methods and multi-segment admin sub-paths fall through to the existing 403. This keeps the general llm_api_routes list free of management paths and avoids accidentally exposing POST/PUT/DELETE writes through the route-check layer. --------- Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: ryan-crabbe-berri <[email protected]> * chore(ci): merge dev branch (#28807) * chore(proxy): route path-dependent call sites through get_request_route Replace direct ``request.url.path`` reads in auth, ACL, routing, and audit-log decisions with ``get_request_route(request)`` — the helper already added in ``auth/auth_utils.py`` that returns the ASGI ``scope["path"]`` with ``root_path`` stripped. Starlette reconstructs ``url.path`` from the Host header; ``scope["path"]`` is uvicorn's parse of the request line and matches what FastAPI dispatches on, so it's the authoritative route for any decision that should agree with the actual handler. Sites: - _experimental/mcp_server/auth/user_api_key_auth_mcp.py - management_endpoints/mcp_management_endpoints.py - vector_store_endpoints/utils.py - pass_through_endpoints/pass_through_endpoints.py - auth/route_checks.py - litellm_pre_call_utils.py - spend_tracking/spend_management_endpoints.py - common_utils/http_parsing_utils.py - management_helpers/utils.py - health_endpoints/_health_endpoints.py Adds regression tests in tests/proxy_unit_tests/test_proxy_routes.py that construct a Request with scope["path"] set to a benign route and the Host header crafted so url.path would resolve differently; each site's decision is asserted against scope["path"]. * chore(proxy): make get_request_route imports lazy at call sites Move the ``from litellm.proxy.auth.auth_utils import get_request_route`` imports added in the prior commit back to the function bodies that use them. The module-level form participates in a long-standing import cycle through ``auth_utils -> _types -> ...`` and was flagged by CodeQL on the PR; the lazy form matches the pattern the proxy already uses for ``user_api_key_auth`` and related helpers elsewhere in these files. Also drop the ``RouteChecks._is_assistants_api_request`` delegation in ``_get_metadata_variable_name`` introduced in the prior commit — the delegation pulled ``RouteChecks`` into the same cycle, and the call site reuses the resolved route for its other branches, so inlining the substring check is both cycle-free and avoids a redundant second ``get_request_route`` call. Comment in test_proxy_routes.py acknowledges that the two MCP table entries exercise ``get_request_route`` directly rather than the full production handler (which needs ASGI scope + MCP state to invoke). --------- Co-authored-by: shin-berri <[email protected]> Co-authored-by: user <[email protected]> * fix(team): keep team_alias cache in sync on _cache_team_object writes (#28737) * fix(team): keep team_alias cache in sync on _cache_team_object writes _cache_team_object wrote only to the team_id:<id> cache key, but the JWT auth path that uses team_alias_jwt_field reads from a separate team_alias:<alias> key (get_team_object_by_alias caches under both keys on miss, but reads only the alias-keyed one). After any team-mutation endpoint (team_model_add, team_model_delete, update_team, the two access-group writes) the team_id cache was refreshed but the team_alias cache stayed stale until TTL — JWT callers using team_alias_jwt_field kept seeing the pre-mutation team for the full cache window. Mirror the write under the alias key inside _cache_team_object so every existing caller stays in sync without further changes. Skip the alias write when team_alias is None/empty so we don't collide across alias-less teams. Surfaced testing the LIT-3244 cherry-pick on patch/1.86.0: the LIT-3244 fix correctly invalidated the team_id cache but the customer's JWT used team_alias_jwt_field, so they kept hitting the stale alias-keyed entry. * fix(team): delete (not overwrite) team_alias cache on _cache_team_object The prior shape of this PR wrote both team_id:<id> AND team_alias:<alias> from _cache_team_object. team_alias is NOT unique in the schema (no @unique on LiteLLM_TeamTable.team_alias), and get_team_object_by_alias enforces uniqueness on its own DB-fetch path (len(teams) > 1 raises). Writing the alias-keyed cache from the generic refresh path bypassed that check: a team admin renaming their team to collide with another team's alias could silently overwrite the cached team for JWT-by-alias auth, swapping the resolved team under that alias for the cache window. Switch the alias-keyed operation from a write to a delete (mirroring the dual-cache delete pattern in _delete_cache_key_object). After every team write, the next JWT-by-alias reader cache-misses and falls through to get_team_object_by_alias, which (a) re-fetches the fresh team from DB, closing the LIT-3244 staleness gap that motivated this PR, and (b) enforces alias uniqueness before populating either cache key. team_id:<id> writes are unchanged — team_id is the table PK and is guaranteed unique. Surfaced in veria-ai review on #28739. * fix(managed-files): anchor model_id regex so it doesn't match llm_output_file_model_id extract_model_id_from_unified_id used `re.search(r"model_id,([^;]+)", ...)` which substring-matches the `model_id,` inside the file-ID encoding's `llm_output_file_model_id,<deployment_uuid>` field. parse_unified_id then fed that deployment UUID back into the auth path as a model candidate via _extract_models_from_managed_resource_id, and every team-BYOK file attach 403'd with: team not allowed to access model. This team can only access models=['openai/*']. Tried to access <deployment-uuid> The team's models list correctly contains the public name (`openai/*`) that target_model_names matches, but the bogus UUID candidate fails the wildcard check first. Anchor the regex to a field boundary (`(?:^|;)model_id,`) so it matches the legitimate top-level `model_id,<value>` field on vector_store unified IDs and skips substring matches inside other fields. File-IDs (which have no top-level `model_id` field) now return None and contribute no spurious UUID candidate. Surfaced reproducing LIT-3244 on patch/1.86.0 with the customer's exact flow: team with openai/* BYOK deployment, JWT-scoped user, POST /v1/vector_stores/{id}/files attaching a file uploaded with target_model_names=openai/gpt-4o. * fix(proxy): hydrate wildcard discovery credentials (#28284) (#28822) * fix(proxy): hydrate wildcard discovery credentials * fix(proxy): constrain wildcard credential hydration Co-authored-by: Dibyo Mukherjee <[email protected]> * ci: add daily oss-agent-shin branch creation workflow (#28829) Creates litellm_oss_agent_shin_MM_DD_YYYY from main every day at 00:00 UTC. Lets us retarget oss-agent-shin fork PRs onto a canonical branch so CircleCI runs with secrets, without granting the agent write access. Co-authored-by: shin-berri <[email protected]> Co-authored-by: yuneng-jiang <[email protected]> Co-authored-by: Ishaan Jaffer <[email protected]> * test(proxy): add harness for proxy_server.py behavior-pinning (#28827) * test(proxy): add harness for proxy_server.py behavior-pinning Creates tests/test_litellm/proxy/proxy_server/ with: - conftest.py: 11 shared fixtures (app, client, mock_prisma, auth_as, mock_router with parametrized response builders, normalize, etc.) - _coverage_check.py: per-PR coverage gate (line + branch) against a baseline, self-selects target by inspecting which placeholder files have been filled - _pin_check.py: AST-based gate that verifies every pin-list item has >=1 happy + >=1 error test with a real assertion (no status-only) - test_harness_smoke.py: 19 smoke tests covering every fixture + both scripts end-to-end - 26 placeholder test files (one docstring each) reserved for follow-up PRs per the directory ownership in the Notion plan - .coverage_baseline pinned at 0% so future PRs measure deltas against new-tests-only and aren't entangled with the broader scattered test suite Adds a dedicated proxy-server job to test-unit-proxy-endpoints.yml so this directory's runtime + coverage are tracked independently. Plan: https://www.notion.so/36c43b8acdab81ee845fd5365128a2fc * ci(proxy-endpoints): allow workflow_dispatch Lets the workflow be triggered manually on a branch via `gh workflow run`, which is needed for the verify-first flow on workflow changes before opening a PR. * test(proxy): address review feedback on proxy_server harness - conftest.py: anchor sys.path insert to __file__ (Path(__file__).resolve().parents[4]) instead of CWD-relative os.path.abspath("../../../../") which resolved to the wrong directory when pytest is launched from the repo root. - _coverage_check.py: actually read .coverage_baseline and use it as the floor (line_min = max(target, baseline)). Closes the gap between the PR description's "delta semantics" and what the script was doing. With baseline=0.0 today this is a no-op; future PRs that update the baseline cause regressions (test deletions etc.) to trip the gate even if the static PR target is still met. - _pin_check.py: drop unreachable startswith("_") guard (test_*.py glob never yields underscore-prefixed names) and read each test file once instead of twice. * feat(openai): apply regional-processing cost uplift for EU/US data residency (#28626) * feat(openai): apply regional-processing cost uplift for EU/US data residency OpenAI charges a 10% uplift on the latest GPT models when requests are served from a regionalized hostname (eu./us.api.openai.com). Infer the region from `api_base`, expose it on `kwargs["litellm_params"]["data_residency"]`, and multiply the computed cost by a per-model `regional_processing_uplift_multiplier_<region>` field. https://claude.ai/code/session_012ebH44s7ohYxjoix5CXzTW * test: allow regional_processing_uplift_multiplier_{eu,us} in model_prices schema * fix(cost): tighten data_residency inference and restore model_cost in tests - Only infer OpenAI data_residency when custom_llm_provider == "openai"; drop the implicit None fallback so non-OpenAI callers can't accidentally pick up a regional tag from a stray OpenAI hostname. - _local_model_cost_map fixture now snapshots and restores litellm.model_cost and LITELLM_LOCAL_MODEL_COST_MAP so tests don't leak state across the session. * refactor(openai): move data_residency helper under llms/openai * fix: thread data_residency through realtime stream cost calculation Co-authored-by: Yassin Kortam <[email protected]> * fix(cost): thread data_residency through batch_cost_calculator Apply the OpenAI regional-processing uplift multiplier to retrieve_batch cost paths so Batch API requests served via eu./us.api.openai.com are priced at the same uplifted token rates as completions/transcriptions. * refactor(openai): encapsulate provider check inside infer_openai_data_residency Move the custom_llm_provider == "openai" guard from get_litellm_params into the helper itself so the core utility no longer carries provider-specific dispatch logic. Callers pass through the provider unconditionally; the helper returns None for any non-OpenAI provider. * fix(responses): thread data_residency through Responses logging params The Responses API paths build their logging litellm_params dict after provider resolution but did not include data_residency, so cost calc saw None even when the effective api_base was a regional OpenAI host. --------- Co-authored-by: Claude <[email protected]> Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: Yassin Kortam <[email protected]> --------- Co-authored-by: milan-berri <[email protected]> Co-authored-by: Cursor <[email protected]> Co-authored-by: Mateo Wang <[email protected]> Co-authored-by: Mateo <[email protected]> Co-authored-by: Claude Opus 4.7 <[email protected]> Co-authored-by: Yassin Kortam <[email protected]> Co-authored-by: Yassin Kortam <[email protected]> Co-authored-by: yuneng-jiang <[email protected]> Co-authored-by: shin-berri <[email protected]> Co-authored-by: user <[email protected]> Co-authored-by: Krrish Dholakia <[email protected]> Co-authored-by: ryan-crabbe-berri <[email protected]> Co-authored-by: ryan-crabbe-berri <[email protected]> Co-authored-by: Dibyo Mukherjee <[email protected]> Co-authored-by: ishaan-berri <[email protected]> Co-authored-by: Ishaan Jaffer <[email protected]>
- Remove duplicate _CREDENTIAL_LITELLM_PARAM_FIELDS assignment in model_checks - Restore response=dict(result) in _emit_management_endpoint_otel_span so OTEL spans for successful management endpoint calls include response data Co-authored-by: Yassin Kortam <[email protected]>
- Wrap _emit_management_endpoint_otel_span in try/except on the failure path of management_endpoint_wrapper so OTEL errors cannot swallow the original management-endpoint exception. - Bound GalileoObserve.in_memory_records at GALILEO_MAX_IN_MEMORY_RECORDS to prevent unbounded memory growth when flushes persistently fail. Co-authored-by: Yassin Kortam <[email protected]>
…s under concurrency - Snapshot record count before await so concurrent appends during the network round-trip aren't silently dropped when clearing the buffer. - Build payload from a snapshot list so the legacy path no longer shares a live reference with self.in_memory_records. - On legacy enterprise auth (username/password), drop cached bearer-token headers when the upstream rejects the request (401/403) so the next flush re-authenticates instead of failing forever on a stale token. Co-authored-by: Yassin Kortam <[email protected]>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
Reviewed by Cursor Bugbot for commit 499642a. Configure here.
![veria-ai[bot]](https://avatars.githubusercontent.com/in/1723776?s=60&v=4){kind=small}
Contributor
PR overviewAll previously flagged issues have been addressed. No open security concerns remain on this pull request. Security reviewNo open security issues remain on this pull request. Fixed/addressed: 1 · PR risk: 0/10 |
Collaborator
Collaborator
mateo-berri
merged commit f25648c
into
litellm_internal_staging
116 of 119 checks passed
fzowl
pushed a commit
to fzowl/litellm
that referenced
this pull request
Jun 24, 2026
BerriAI#28771) * fix(galileo): support hosted v2 spans API and string output extraction Use GALILEO_API_KEY with /v2/projects/{id}/spans for Galileo Cloud, keep legacy observe/ingest for username/password deployments, and extract assistant content as a string instead of a message dict. Co-authored-by: Cursor <[email protected]> * fix(galileo): address review — async enterprise auth and message input Use async httpx for enterprise login to avoid blocking the event loop, preserve multi-turn messages in v2 span input, and clean up tests. Co-authored-by: Cursor <[email protected]> * fix(galileo): handle negative TZ offsets, 2xx success, and Pydantic ImageObject serialization Co-authored-by: Yassin Kortam <[email protected]> * fix(galileo): treat any 2xx ingest response as success Use response.is_success so 201 Created clears in_memory_records and avoids duplicate span submissions on subsequent flushes. Co-authored-by: Cursor <[email protected]> * fix(galileo): cast message dict for mypy in convert_content_list_to_str Co-authored-by: Cursor <[email protected]> * merge main (BerriAI#28835) * fix(proxy): Bedrock Knowledge Base pass-through: preserve SigV4 headers and signed request body (BerriAI#27526) * Fix Bedrock KB pass-through SigV4 headers and signed body Coerce botocore HeadersDict to a dict for pass-through routes. When forward_headers is true, drop request headers that collide case-insensitively with signed headers so client Bearer auth does not shadow AWS SigV4. Send prepped.body as raw content so the outbound payload matches the signature after logging hooks mutate the parsed dict. Co-authored-by: Cursor <[email protected]> * Simplify pass-through raw body handling Read the SigV4-signed bytes directly from request.state inside pass_through_request instead of threading a custom_raw_body argument through three functions. Helper methods are restored to their original signatures, and the new branch lives in one place at each httpx call site. Co-authored-by: Cursor <[email protected]> * Harden pass-through raw body read from request.state Guard missing request.state (test fixtures) and ignore non-bytes/str values so MagicMock does not trigger the SigV4 raw-body path. Co-authored-by: Cursor <[email protected]> * Test pass_through_request state_raw_body uses httpx content= Cover non-streaming (async_client.request) and streaming (build_request) paths so SigV4 bytes on request.state are not replaced by json= of a hook-mutated dict. Co-authored-by: Cursor <[email protected]> --------- Co-authored-by: Cursor <[email protected]> * chore(tests): migrate Bedrock CI to AWS account 941277531214 (BerriAI#28728) * chore(tests): migrate Bedrock CI from AWS account 888602223428 to 941277531214 The original account (888602223428) was put under a security restriction by AWS after a root access key leaked in a PR comment. While that account works its way through the AWS Support unlock process, Bedrock-touching CI tests have been migrated to a fresh account (941277531214). Changes: - Replace 26 hardcoded references to 888602223428 with 941277531214 across 8 files (provisioned-model ARNs, imported-model ARNs, AgentCore runtime ARNs, batch execution role ARN, and example proxy config). - The provisioned-model and imported-model ARNs are referenced only from mocked unit tests — no AWS resources to recreate. - The batch execution IAM role has been recreated in the new account with the same name and equivalent permissions. - The two AgentCore runtimes (hosted_agent_r9jvp-3ySZuRHjLC, hosted_agent_13sf6-cALnp38iZD) are being recreated in the new account under the same names — see tools/agentcore-deploy/ in a follow-up. CircleCI env vars AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY / AWS_REGION_NAME were updated separately via the CircleCI API to point at the new account. Smoke-tested locally against the new account: aws bedrock-runtime converse --region us-west-2 \ --model-id us.anthropic.claude-sonnet-4-5-20250929-v1:0 \ --messages '[{"role":"user","content":[{"text":"ping"}]}]' → 200, model returned 'pong' * chore(tests): refresh AgentCore ARN suffixes to match newly-deployed runtimes The first migration commit replaced just the account ID, but AgentCore auto-assigns a random 10-char suffix to every runtime on creation — we can't reuse the original suffixes (`3ySZuRHjLC`, `cALnp38iZD`) in the new account. Updated the AgentCore-runtime ARNs in the three files that reference real runtime IDs (not the mock-based unit-test ARNs). Deployed runtimes: arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_r9jvp-Rq79QFC2fp arn:aws:bedrock-agentcore:us-west-2:941277531214:runtime/hosted_agent_13sf6-4046UzHSwy Both runtimes are status=READY and pass a smoke invoke: $ aws bedrock-agentcore invoke-agent-runtime --agent-runtime-arn ... --payload '{"prompt":"ping"}' → 200, {"result": "echo: ping"} The agent is a minimal echo (see /tmp/agentcore_deploy/agent.py for the deploy artifacts). Tests that only verify the SDK wiring will pass; if any test asserts on agent output content, swap the echo for the real agent. * chore(tests): point Bedrock batch tests at new-account S3 bucket The account migration (888602223428 -> 941277531214) was a flat account-ID swap, which only rewrites ARNs that embed the account number. S3 bucket names carry no account ID, so the live Bedrock batch tests still uploaded to `litellm-proxy` — a bucket that lives in the old account. S3 names are globally unique, and the old account still holds that name, so it can't be recreated in the new account. Rename to `litellm-proxy-941277531214` (account-ID suffix guarantees global uniqueness). The bucket must be created in 941277531214 and the batch execution role granted s3:GetObject/PutObject/ListBucket on it before this job is run in CI. * chore(tests): point live S3 logging test at new-account bucket Same account-ID-free blind spot as the batch bucket: `load-testing-oct` lives in the old account and its name can't be reused globally. The `logging_testing` CI job is wired into the workflow and runs test_basic_s3_logging, which uploads to this bucket with the CI env creds, then lists and deletes objects — a live dependency. Rename to `load-testing-oct-941277531214`. The bucket must exist in the new account with the CI IAM principal granted s3:PutObject/GetObject/ListBucket/DeleteObject before this job runs. * chore(tests): repoint Bedrock guardrail IDs to new-account guardrails The migration left guardrail IDs untouched (no account ID in them), so all live guardrail tests failed with "guardrail identifier or version does not exist" against 941277531214. Recreated both guardrails in the new account and updated the hardcoded IDs: - wf0hkdb5x07f -> zgkmukebruil (PII mask: PHONE + CREDIT_DEBIT_CARD, with explicit inputAction=ANONYMIZE so masking applies to INPUT, which is the source litellm's moderation hook sends) - ff6ujrregl1q -> 4w3d1di3snt5 (blocks "coffee"; blocked message set to the exact string the tests assert on) Updated test_bedrock_guardrails.py, otel_test_config.yaml, and the guardrailConfig in test_bedrock_completion.py. Verified locally: the 5 previously-failing guardrail tests now pass. * test(bedrock): migrate legacy models to current inference profiles The new CI account (941277531214) cannot invoke legacy Bedrock models (AWS gates them: "marked by provider as Legacy... not actively using in the last 30 days"). Migrated the live-call tests: - anthropic.claude-3-sonnet-20240229 -> us.anthropic.claude-sonnet-4-5-20250929-v1:0 - anthropic.claude-3-haiku-20240307 -> us.anthropic.claude-haiku-4-5-20251001-v1:0 Current Claude models on Bedrock require the us. inference-profile prefix (bare on-demand ids are rejected). cohere.command-r-plus has no working replacement (all Cohere is legacy- gated in the new account): swapped to claude-haiku-4-5 in provider- agnostic param lists. amazon.titan-image-generator skipped (no working replacement). Mocked/transformation/cost tests that reference the legacy strings are intentionally left unchanged. Verified live against the new account. * test(bedrock): repoint SageMaker + Knowledge Base to new-account resources These referenced account-scoped resources by hardcoded id that only existed in the old account, so the migration's account-ID swap missed them. Recreated in 941277531214 and repointed: - SageMaker endpoint jumpstart-dft-hf-textgeneration1-mp-20240815-185614 -> litellm-ci-textgen (gpt2 on a TGI container, ml.g5.xlarge) - Bedrock Knowledge Base T37J8R4WTM -> LCYXFBR2TU (OpenSearch Serverless vector store + titan-embed-text-v2, seeded with a LiteLLM doc) Verified live: test_sagemaker.py (12 passed) and test_bedrock_knowledgebase_hook.py (12 passed). * test(reasoning_effort_grid): skip bedrock claude-opus-4-7 cells (not entitled on 941277531214) claude-opus-4-7 is listed in the new Bedrock CI account's foundation models but invoke is denied (AccessDeniedException: "not available for this account"). Bedrock access to the flagship Opus requires an AWS Sales request, not the self-serve model-access toggle, so it can't be enabled inline with the rest of the account migration. Add an optional `skip_reason` to ModelEntry and set it on the bedrock-claude-opus-4-7 entry; the grid test honors it via pytest.skip. Cell count (231) and route coverage are unchanged, so the structural asserts still pass. Restore coverage by deleting the one skip_reason line once access is granted. * test(bedrock): swap/skip legacy-gated models unavailable on new CI account The migrated AWS account (941277531214) cannot access several models that the old account could, so the remaining red CI jobs were hitting real Bedrock "Access denied / Legacy" and "account not authorized" errors: - image_gen: skip both Nova Canvas test classes (amazon.nova-canvas-v1:0 is legacy-gated), matching the existing titan skip. - batches: skip test_async_file_and_batch (Bedrock batch inference is not authorized on the new account; requires an AWS support case). - litellm_overhead: swap legacy claude-3-5-haiku for the active us.anthropic.claude-haiku-4-5 inference profile. - test_completion_claude_3_function_call: swap legacy claude-3-sonnet for the active us.anthropic.claude-sonnet-4-5 inference profile. https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa * test(bedrock): fix remaining e2e legacy-model + batch failures on new CI account - e2e_openai_endpoints: skip test_bedrock_batches_api (Bedrock batch inference is not authorized on account 941277531214) and migrate the missed s3_bucket_name in oai_misc_config.yaml to litellm-proxy-941277531214. - build_and_test: swap legacy bedrock claude-3-sonnet for the active us.anthropic.claude-sonnet-4-5 inference profile in the proxy structured output e2e test. https://claude.ai/code/session_01Y7zgHYu9GX29YRwV4yiWAa * test(bedrock): make opus-4-7 + batch cells fail loudly and mock image-gen (BerriAI#28791) Replace the silent skips added for the new CI account with noisier behavior: - reasoning-effort grid: opus-4-7 cells now fail (when AWS creds are present) instead of skipping, so the missing entitlement stays visible in CI; they still skip when AWS creds are absent (local dev) - Bedrock batch inference tests: drop the skip so they run and fail until batch access is granted - Titan + Nova Canvas image-gen tests: mock the Bedrock HTTP call so the transform + cost-tracking path stays under test without live model access https://claude.ai/code/session_01MT7SWDnXUjv6e6EPG7BDjT * test(bedrock): use pytest.xfail for known-failing opus-4-7 cells Replace pytest.fail with pytest.xfail when a model has a fail_reason, so known-broken cells stay visible as XFAIL without keeping CI red. Co-authored-by: Yassin Kortam <[email protected]> --------- Co-authored-by: Mateo <[email protected]> Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: Yassin Kortam <[email protected]> * fix(otel): export SERVER span on management-endpoint success without http_request (BerriAI#28794) Co-authored-by: Yassin Kortam <[email protected]> * chore(ci): merge dev branch (BerriAI#28801) * chore(proxy): route path-dependent call sites through get_request_route Replace direct ``request.url.path`` reads in auth, ACL, routing, and audit-log decisions with ``get_request_route(request)`` — the helper already added in ``auth/auth_utils.py`` that returns the ASGI ``scope["path"]`` with ``root_path`` stripped. Starlette reconstructs ``url.path`` from the Host header; ``scope["path"]`` is uvicorn's parse of the request line and matches what FastAPI dispatches on, so it's the authoritative route for any decision that should agree with the actual handler. Sites: - _experimental/mcp_server/auth/user_api_key_auth_mcp.py - management_endpoints/mcp_management_endpoints.py - vector_store_endpoints/utils.py - pass_through_endpoints/pass_through_endpoints.py - auth/route_checks.py - litellm_pre_call_utils.py - spend_tracking/spend_management_endpoints.py - common_utils/http_parsing_utils.py - management_helpers/utils.py - health_endpoints/_health_endpoints.py Adds regression tests in tests/proxy_unit_tests/test_proxy_routes.py that construct a Request with scope["path"] set to a benign route and the Host header crafted so url.path would resolve differently; each site's decision is asserted against scope["path"]. * chore(proxy): make get_request_route imports lazy at call sites Move the ``from litellm.proxy.auth.auth_utils import get_request_route`` imports added in the prior commit back to the function bodies that use them. The module-level form participates in a long-standing import cycle through ``auth_utils -> _types -> ...`` and was flagged by CodeQL on the PR; the lazy form matches the pattern the proxy already uses for ``user_api_key_auth`` and related helpers elsewhere in these files. Also drop the ``RouteChecks._is_assistants_api_request`` delegation in ``_get_metadata_variable_name`` introduced in the prior commit — the delegation pulled ``RouteChecks`` into the same cycle, and the call site reuses the resolved route for its other branches, so inlining the substring check is both cycle-free and avoids a redundant second ``get_request_route`` call. Comment in test_proxy_routes.py acknowledges that the two MCP table entries exercise ``get_request_route`` directly rather than the full production handler (which needs ASGI scope + MCP state to invoke). --------- Co-authored-by: shin-berri <[email protected]> Co-authored-by: user <[email protected]> * chore(ci): merge dev branch (BerriAI#28657) * feat(dashboard): navbar hierarchy + Agent Platform notifications (BerriAI#27543) * feat(dashboard): refine navbar zones and Agent Platform notice Restructure the admin navbar for production users: clear product vs community vs personal columns with vertical dividers, icon-only Slack/GitHub in a shared chip, and Docs/Blog typography aligned on an 8px rhythm. Add a notifications bell with popover linking to the LiteLLM Agent Platform repo and optional mark-as-read persistence. Promote the account control with initials avatar, single-line display name, and navDisplayName mapping for placeholder user ids (e.g. default_user_id). Co-authored-by: Cursor <[email protected]> * fix(dashboard): address PR review — AntD buttons, public page guard, dedupe regex - Replace raw <button> with AntD Button in BlogDropdown, NotificationsBell, UserDropdown, and test mock - Guard NotificationsBell + container behind !isPublicPage to avoid rendering on public pages - Remove redundant equality checks in navDisplayName (regex already covers them) - Remove unused `lower` variable after simplification --------- Co-authored-by: Cursor <[email protected]> Co-authored-by: yuneng-jiang <[email protected]> * fix(dashboard): drop dead useHealthReadiness import in navbar The module was removed in BerriAI#27896 (replaced by useHealthReadinessDetails), but the import survived the rebase. The symbol is unused — only useHealthReadinessDetails is consumed in the file. Removing the dead import unblocks the UI TypeScript build. * fix(dashboard): align CommunityEngagementButtons test with icon-only aria-labels The component was refactored to an icon-only chip with aria-label='LiteLLM on GitHub' (squash BerriAI#27543), but the test still asserted /star us on github/i. Update the query to match the rendered accessible name. * refactor(dashboard): drop unused props from NavbarProps The navbar refactor moved user identity + dark-mode state to internal hooks (useAuthorized, useWorker), but the NavbarProps interface still declared userID, userEmail, userRole, premiumUser, isDarkMode, and toggleDarkMode as required, forcing every caller to thread them through. Drop them from the interface and all four call sites (page.tsx, (dashboard)/layout.tsx, public_model_hub.tsx, navbar.test.tsx). Also shrinks the destructure in layout.tsx so the now-unused locals stop being pulled out of useAuthorized(). * refactor(dashboard): use useSyncExternalStore for NotificationsBell dismiss flag Reads/writes of the litellmHideAgentPlatformBanner key were done directly inside NotificationsBell via a useEffect + useState pair. Every other localStorage-backed flag in the dashboard (Disable ShowPrompts, DisableBouncingIcon, DisableShowNewBadge, DisableUsageIndicator, DisableBlogPosts) is wrapped in a useSyncExternalStore hook over localStorageUtils so all mounted components stay in sync. Extract useHideAgentPlatformBanner to follow the same shape, swap NotificationsBell to consume it, and add a regression test that two sibling bells stay in sync without a remount when one is dismissed. * refactor: mask credential fields in proxy settings GET responses (BerriAI#28682) * refactor: mask credential fields in proxy settings GET responses Brings SSO settings, cache settings, and the email/Slack alerting view in /get/config/callbacks in line with the HashiCorp Vault config-override pattern, so persisted credentials are not transported back to the UI in plaintext. * refactor: harden short-value masking and hoist alerting var constant Closes two review observations: - mask_sensitive_keys now replaces short values (below the visible prefix+suffix length) with an all-mask string instead of returning them unchanged, so a 1-7 character credential is no longer round-tripped verbatim. - _ALERTING_SENSITIVE_VARS is moved out of get_config() to a module-level constant, matching the analogous _SSO_SENSITIVE_FIELDS and _CACHE_SENSITIVE_FIELDS in the SSO and cache endpoint files. --------- Co-authored-by: Krrish Dholakia <[email protected]> Co-authored-by: Cursor <[email protected]> * fix(ui): show 2-decimal precision for max_budget on key overview (BerriAI#28809) The Key Info Overview tab's Spend card truncated sub-dollar budgets to "$0" because formatNumberWithCommas defaults to 0 decimals. The Settings tab passes 2; align the overview so a $0.10 budget renders as "$0.10". Resolves LIT-2845 * feat(proxy): allow `llm_api_routes` virtual keys to list MCP servers (BerriAI#28442) * feat(proxy): allow llm_api_routes virtual keys to list MCP servers Add a new `mcp_discovery_routes` group (GET /v1/mcp/server and GET /v1/mcp/server/{server_id}) and include it in `llm_api_routes` so that virtual keys configured with `allowed_routes=["llm_api_routes"]` can discover the MCP servers they have access to. Previously these calls failed with 'Virtual key is not allowed to call this route. Only allowed to call routes: [llm_api_routes]'. The GET handlers already sanitize the response for restricted virtual keys via `_sanitize_mcp_server_list_for_virtual_key`, stripping credential-bearing fields (url, headers, env). Write methods (POST/PUT/DELETE) on the same paths remain gated by the existing handler-level admin role checks. The new discovery list is intentionally kept OUT of `mcp_inference_routes`, so `is_llm_api_route()` still returns False for these paths — this preserves the existing contract that DISABLE_LLM_API_ENDPOINTS must not block the Admin UI from listing MCP servers. Co-authored-by: ryan-crabbe-berri <[email protected]> * refactor(proxy): make MCP discovery carve-out method-aware Replace the `mcp_discovery_routes` group in `llm_api_routes` with a method-aware special case inside `is_virtual_key_allowed_to_call_route`. Virtual keys with allowed_routes=["llm_api_routes"] are now permitted to call only GET /v1/mcp/server and GET /v1/mcp/server/{server_id} — non-GET methods and multi-segment admin sub-paths fall through to the existing 403. This keeps the general llm_api_routes list free of management paths and avoids accidentally exposing POST/PUT/DELETE writes through the route-check layer. --------- Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: ryan-crabbe-berri <[email protected]> * chore(ci): merge dev branch (BerriAI#28807) * chore(proxy): route path-dependent call sites through get_request_route Replace direct ``request.url.path`` reads in auth, ACL, routing, and audit-log decisions with ``get_request_route(request)`` — the helper already added in ``auth/auth_utils.py`` that returns the ASGI ``scope["path"]`` with ``root_path`` stripped. Starlette reconstructs ``url.path`` from the Host header; ``scope["path"]`` is uvicorn's parse of the request line and matches what FastAPI dispatches on, so it's the authoritative route for any decision that should agree with the actual handler. Sites: - _experimental/mcp_server/auth/user_api_key_auth_mcp.py - management_endpoints/mcp_management_endpoints.py - vector_store_endpoints/utils.py - pass_through_endpoints/pass_through_endpoints.py - auth/route_checks.py - litellm_pre_call_utils.py - spend_tracking/spend_management_endpoints.py - common_utils/http_parsing_utils.py - management_helpers/utils.py - health_endpoints/_health_endpoints.py Adds regression tests in tests/proxy_unit_tests/test_proxy_routes.py that construct a Request with scope["path"] set to a benign route and the Host header crafted so url.path would resolve differently; each site's decision is asserted against scope["path"]. * chore(proxy): make get_request_route imports lazy at call sites Move the ``from litellm.proxy.auth.auth_utils import get_request_route`` imports added in the prior commit back to the function bodies that use them. The module-level form participates in a long-standing import cycle through ``auth_utils -> _types -> ...`` and was flagged by CodeQL on the PR; the lazy form matches the pattern the proxy already uses for ``user_api_key_auth`` and related helpers elsewhere in these files. Also drop the ``RouteChecks._is_assistants_api_request`` delegation in ``_get_metadata_variable_name`` introduced in the prior commit — the delegation pulled ``RouteChecks`` into the same cycle, and the call site reuses the resolved route for its other branches, so inlining the substring check is both cycle-free and avoids a redundant second ``get_request_route`` call. Comment in test_proxy_routes.py acknowledges that the two MCP table entries exercise ``get_request_route`` directly rather than the full production handler (which needs ASGI scope + MCP state to invoke). --------- Co-authored-by: shin-berri <[email protected]> Co-authored-by: user <[email protected]> * fix(team): keep team_alias cache in sync on _cache_team_object writes (BerriAI#28737) * fix(team): keep team_alias cache in sync on _cache_team_object writes _cache_team_object wrote only to the team_id:<id> cache key, but the JWT auth path that uses team_alias_jwt_field reads from a separate team_alias:<alias> key (get_team_object_by_alias caches under both keys on miss, but reads only the alias-keyed one). After any team-mutation endpoint (team_model_add, team_model_delete, update_team, the two access-group writes) the team_id cache was refreshed but the team_alias cache stayed stale until TTL — JWT callers using team_alias_jwt_field kept seeing the pre-mutation team for the full cache window. Mirror the write under the alias key inside _cache_team_object so every existing caller stays in sync without further changes. Skip the alias write when team_alias is None/empty so we don't collide across alias-less teams. Surfaced testing the LIT-3244 cherry-pick on patch/1.86.0: the LIT-3244 fix correctly invalidated the team_id cache but the customer's JWT used team_alias_jwt_field, so they kept hitting the stale alias-keyed entry. * fix(team): delete (not overwrite) team_alias cache on _cache_team_object The prior shape of this PR wrote both team_id:<id> AND team_alias:<alias> from _cache_team_object. team_alias is NOT unique in the schema (no @unique on LiteLLM_TeamTable.team_alias), and get_team_object_by_alias enforces uniqueness on its own DB-fetch path (len(teams) > 1 raises). Writing the alias-keyed cache from the generic refresh path bypassed that check: a team admin renaming their team to collide with another team's alias could silently overwrite the cached team for JWT-by-alias auth, swapping the resolved team under that alias for the cache window. Switch the alias-keyed operation from a write to a delete (mirroring the dual-cache delete pattern in _delete_cache_key_object). After every team write, the next JWT-by-alias reader cache-misses and falls through to get_team_object_by_alias, which (a) re-fetches the fresh team from DB, closing the LIT-3244 staleness gap that motivated this PR, and (b) enforces alias uniqueness before populating either cache key. team_id:<id> writes are unchanged — team_id is the table PK and is guaranteed unique. Surfaced in veria-ai review on BerriAI#28739. * fix(managed-files): anchor model_id regex so it doesn't match llm_output_file_model_id extract_model_id_from_unified_id used `re.search(r"model_id,([^;]+)", ...)` which substring-matches the `model_id,` inside the file-ID encoding's `llm_output_file_model_id,<deployment_uuid>` field. parse_unified_id then fed that deployment UUID back into the auth path as a model candidate via _extract_models_from_managed_resource_id, and every team-BYOK file attach 403'd with: team not allowed to access model. This team can only access models=['openai/*']. Tried to access <deployment-uuid> The team's models list correctly contains the public name (`openai/*`) that target_model_names matches, but the bogus UUID candidate fails the wildcard check first. Anchor the regex to a field boundary (`(?:^|;)model_id,`) so it matches the legitimate top-level `model_id,<value>` field on vector_store unified IDs and skips substring matches inside other fields. File-IDs (which have no top-level `model_id` field) now return None and contribute no spurious UUID candidate. Surfaced reproducing LIT-3244 on patch/1.86.0 with the customer's exact flow: team with openai/* BYOK deployment, JWT-scoped user, POST /v1/vector_stores/{id}/files attaching a file uploaded with target_model_names=openai/gpt-4o. * fix(proxy): hydrate wildcard discovery credentials (BerriAI#28284) (BerriAI#28822) * fix(proxy): hydrate wildcard discovery credentials * fix(proxy): constrain wildcard credential hydration Co-authored-by: Dibyo Mukherjee <[email protected]> * ci: add daily oss-agent-shin branch creation workflow (BerriAI#28829) Creates litellm_oss_agent_shin_MM_DD_YYYY from main every day at 00:00 UTC. Lets us retarget oss-agent-shin fork PRs onto a canonical branch so CircleCI runs with secrets, without granting the agent write access. Co-authored-by: shin-berri <[email protected]> Co-authored-by: yuneng-jiang <[email protected]> Co-authored-by: Ishaan Jaffer <[email protected]> * test(proxy): add harness for proxy_server.py behavior-pinning (BerriAI#28827) * test(proxy): add harness for proxy_server.py behavior-pinning Creates tests/test_litellm/proxy/proxy_server/ with: - conftest.py: 11 shared fixtures (app, client, mock_prisma, auth_as, mock_router with parametrized response builders, normalize, etc.) - _coverage_check.py: per-PR coverage gate (line + branch) against a baseline, self-selects target by inspecting which placeholder files have been filled - _pin_check.py: AST-based gate that verifies every pin-list item has >=1 happy + >=1 error test with a real assertion (no status-only) - test_harness_smoke.py: 19 smoke tests covering every fixture + both scripts end-to-end - 26 placeholder test files (one docstring each) reserved for follow-up PRs per the directory ownership in the Notion plan - .coverage_baseline pinned at 0% so future PRs measure deltas against new-tests-only and aren't entangled with the broader scattered test suite Adds a dedicated proxy-server job to test-unit-proxy-endpoints.yml so this directory's runtime + coverage are tracked independently. Plan: https://www.notion.so/36c43b8acdab81ee845fd5365128a2fc * ci(proxy-endpoints): allow workflow_dispatch Lets the workflow be triggered manually on a branch via `gh workflow run`, which is needed for the verify-first flow on workflow changes before opening a PR. * test(proxy): address review feedback on proxy_server harness - conftest.py: anchor sys.path insert to __file__ (Path(__file__).resolve().parents[4]) instead of CWD-relative os.path.abspath("../../../../") which resolved to the wrong directory when pytest is launched from the repo root. - _coverage_check.py: actually read .coverage_baseline and use it as the floor (line_min = max(target, baseline)). Closes the gap between the PR description's "delta semantics" and what the script was doing. With baseline=0.0 today this is a no-op; future PRs that update the baseline cause regressions (test deletions etc.) to trip the gate even if the static PR target is still met. - _pin_check.py: drop unreachable startswith("_") guard (test_*.py glob never yields underscore-prefixed names) and read each test file once instead of twice. * feat(openai): apply regional-processing cost uplift for EU/US data residency (BerriAI#28626) * feat(openai): apply regional-processing cost uplift for EU/US data residency OpenAI charges a 10% uplift on the latest GPT models when requests are served from a regionalized hostname (eu./us.api.openai.com). Infer the region from `api_base`, expose it on `kwargs["litellm_params"]["data_residency"]`, and multiply the computed cost by a per-model `regional_processing_uplift_multiplier_<region>` field. https://claude.ai/code/session_012ebH44s7ohYxjoix5CXzTW * test: allow regional_processing_uplift_multiplier_{eu,us} in model_prices schema * fix(cost): tighten data_residency inference and restore model_cost in tests - Only infer OpenAI data_residency when custom_llm_provider == "openai"; drop the implicit None fallback so non-OpenAI callers can't accidentally pick up a regional tag from a stray OpenAI hostname. - _local_model_cost_map fixture now snapshots and restores litellm.model_cost and LITELLM_LOCAL_MODEL_COST_MAP so tests don't leak state across the session. * refactor(openai): move data_residency helper under llms/openai * fix: thread data_residency through realtime stream cost calculation Co-authored-by: Yassin Kortam <[email protected]> * fix(cost): thread data_residency through batch_cost_calculator Apply the OpenAI regional-processing uplift multiplier to retrieve_batch cost paths so Batch API requests served via eu./us.api.openai.com are priced at the same uplifted token rates as completions/transcriptions. * refactor(openai): encapsulate provider check inside infer_openai_data_residency Move the custom_llm_provider == "openai" guard from get_litellm_params into the helper itself so the core utility no longer carries provider-specific dispatch logic. Callers pass through the provider unconditionally; the helper returns None for any non-OpenAI provider. * fix(responses): thread data_residency through Responses logging params The Responses API paths build their logging litellm_params dict after provider resolution but did not include data_residency, so cost calc saw None even when the effective api_base was a regional OpenAI host. --------- Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: Yassin Kortam <[email protected]> --------- Co-authored-by: milan-berri <[email protected]> Co-authored-by: Cursor <[email protected]> Co-authored-by: Mateo Wang <[email protected]> Co-authored-by: Mateo <[email protected]> Co-authored-by: Yassin Kortam <[email protected]> Co-authored-by: Yassin Kortam <[email protected]> Co-authored-by: yuneng-jiang <[email protected]> Co-authored-by: shin-berri <[email protected]> Co-authored-by: user <[email protected]> Co-authored-by: Krrish Dholakia <[email protected]> Co-authored-by: ryan-crabbe-berri <[email protected]> Co-authored-by: ryan-crabbe-berri <[email protected]> Co-authored-by: Dibyo Mukherjee <[email protected]> Co-authored-by: ishaan-berri <[email protected]> Co-authored-by: Ishaan Jaffer <[email protected]> * fix: preserve OTEL response payload and remove duplicate constant - Remove duplicate _CREDENTIAL_LITELLM_PARAM_FIELDS assignment in model_checks - Restore response=dict(result) in _emit_management_endpoint_otel_span so OTEL spans for successful management endpoint calls include response data Co-authored-by: Yassin Kortam <[email protected]> * fix: harden OTEL failure path and cap Galileo in-memory buffer - Wrap _emit_management_endpoint_otel_span in try/except on the failure path of management_endpoint_wrapper so OTEL errors cannot swallow the original management-endpoint exception. - Bound GalileoObserve.in_memory_records at GALILEO_MAX_IN_MEMORY_RECORDS to prevent unbounded memory growth when flushes persistently fail. Co-authored-by: Yassin Kortam <[email protected]> * fix(galileo): reset stale bearer token on auth error; preserve records under concurrency - Snapshot record count before await so concurrent appends during the network round-trip aren't silently dropped when clearing the buffer. - Build payload from a snapshot list so the legacy path no longer shares a live reference with self.in_memory_records. - On legacy enterprise auth (username/password), drop cached bearer-token headers when the upstream rejects the request (401/403) so the next flush re-authenticates instead of failing forever on a stale token. Co-authored-by: Yassin Kortam <[email protected]> * test(galileo): expand v2 coverage for config, ingest, headers, and flush paths --------- Co-authored-by: Cursor <[email protected]> Co-authored-by: Yassin Kortam <[email protected]> Co-authored-by: milan-berri <[email protected]> Co-authored-by: Mateo Wang <[email protected]> Co-authored-by: Mateo <[email protected]> Co-authored-by: Yassin Kortam <[email protected]> Co-authored-by: yuneng-jiang <[email protected]> Co-authored-by: shin-berri <[email protected]> Co-authored-by: user <[email protected]> Co-authored-by: Krrish Dholakia <[email protected]> Co-authored-by: ryan-crabbe-berri <[email protected]> Co-authored-by: ryan-crabbe-berri <[email protected]> Co-authored-by: Dibyo Mukherjee <[email protected]> Co-authored-by: ishaan-berri <[email protected]> Co-authored-by: Ishaan Jaffer <[email protected]>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
GALILEO_API_KEYis set (POST /v2/projects/{project_id}/spanswithGalileo-API-Keyheader and optionalGALILEO_LOG_STREAM_ID)/observe/ingest+ username/password) for self-hosted deploymentsoutput_textvalidation by extracting assistant message content as a string instead of serializing the full message dictTest plan
poetry run pytest tests/test_litellm/integrations/test_galileo.py -vNote
Low Risk
Changes are confined to optional Galileo observability logging and non-blocking OTEL error handling in proxy management wrappers, with no auth or request-path behavior changes for LLM calls.
Overview
Galileo logging now supports hosted Galileo Cloud when
GALILEO_API_KEYis set: records are sent as v2 LLM spans (POST /v2/projects/{project_id}/spanswithGalileo-API-Key, optionalGALILEO_LOG_STREAM_ID), including structured chat input frommessagesand assistant output text. Self-hosted Observe still uses/observe/ingestwith async username/password login and bearer auth, including re-auth after 401/403 on stale tokens.Output capture for chat completions now stores plain assistant content (via
get_content_from_model_response) instead of serializing the full message object, fixing validation/API shape issues. Flush is safer: HTTP failures are logged and not raised; successful flushes only remove the batch that was sent; an in-memory cap (1000 records) drops oldest entries on persistent failures; unconfigured env vars skip logging.Proxy management endpoints: OTEL span emission on the failure path is wrapped so observability errors cannot replace the original handler exception.
Adds
tests/test_litellm/integrations/test_galileo.pycovering v2/legacy ingest, headers, spans, flush, and output extraction.Reviewed by Cursor Bugbot for commit 83c6c3d. Bugbot is set up for automated code reviews on this repo. Configure here.