Litellm oss staging 2#28582
Conversation
Co-authored-by: shin-berri <[email protected]> Co-authored-by: yuneng-jiang <[email protected]>
…e via base_model to fix gpt5 model routing (#28490) * feat(azure): decouple deployment ID from model name via base_model Azure OpenAI deployments have arbitrary names (deployment IDs) that may not match the underlying model. Previously, model-type detection (o-series, gpt-5, etc.) relied on substring matching against the deployment name, causing misrouted configs and rejected params when deployment names were non-standard (e.g. 'my-deployment-id' for gpt-5.2). This change extends the existing base_model field to drive model-type detection, config selection, supported param resolution, and param mapping throughout the Azure call path: - _get_azure_config() uses base_model for is_o_series/is_gpt_5 checks - get_provider_chat_config() threads base_model for Azure - get_supported_openai_params() accepts and uses base_model - get_optional_params() accepts base_model and passes it to all Azure config method calls (get_supported_openai_params, map_openai_params) - azure.py completion handler uses base_model for GPT-5 detection - Config internal methods (e.g. is_model_gpt_5_2_model) now receive base_model so features like logprobs are correctly enabled Fully backward compatible - when base_model is unset, behavior is identical. Existing o_series/ and gpt5_series/ prefix workarounds continue to work. Usage in proxy config: model_list: - model_name: my-gpt5 litellm_params: model: azure/my-deployment-id model_info: base_model: azure/gpt-5.2 Fixes: non-standard deployment names like 'prefix-gpt-5.2' rejecting logprobs/top_logprobs despite the underlying model supporting them. * Addressing Greptile comments.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
Greptile SummaryThis PR threads a new
Confidence Score: 5/5Safe to merge — all previously flagged detection-model mismatches have been corrected and the three independent bug fixes are well-scoped with targeted regression tests. The two GPT-5 detection blocks that previously used the raw deployment name instead of No files require special attention — the most impactful change (
|
| Filename | Overview |
|---|---|
| litellm/main.py | Adds _azure_detection_model = base_model or model before all Azure model-type detection branches; threads base_model into get_optional_params and ProviderConfigManager.get_provider_chat_config. The previously-flagged GPT-5 detection block now correctly uses _azure_detection_model. |
| litellm/utils.py | Adds base_model param to get_optional_params (popped from passed_params so it doesn't leak to providers); threads it into Azure detection. Fixes _check_provider_match to treat litellm_provider=None as wildcard. Strips Azure from _PROVIDER_CONFIG_MAP and handles it early in get_provider_chat_config to allow base_model routing. Adds None-guard in register_model. |
| litellm/llms/openai/responses/transformation.py | Adds remove_cache_control_flag_from_input_and_tools and calls it in both transform_responses_api_request and the streaming build path to strip Anthropic-only cache_control markers before sending to OpenAI's Responses API. |
| litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py | Single-line fix: is not None → truthy check for tool_calls, so tool_calls=[] no longer shadows text deltas with empty tool JSON. |
| litellm/llms/azure/azure.py | GPT-5 detection in AzureChatCompletion now uses litellm_params.get("base_model") or model instead of raw model, consistent with the other detection sites. |
| litellm/litellm_core_utils/get_supported_openai_params.py | Adds base_model parameter and uses it for Azure detection and get_supported_openai_params delegation; the early-return path passes base_model or model to the config class. |
| model_prices_and_context_window.json | Corrects gemini-3.1-flash-lite pricing for all three provider variants (vertex_ai, gemini, gemini/) and adds batch/flex/priority tier pricing entries with updated source URL. |
| tests/test_litellm/llms/azure/chat/test_azure_base_model_routing.py | New test file covering _get_azure_config, get_provider_chat_config, get_supported_openai_params, and get_optional_params with base_model overrides for GPT-5, O-series, and default Azure configs. |
| tests/test_litellm/test_register_model_custom_pricing.py | Three new regression tests covering the litellm_provider=None strip in register_model, the _check_provider_match wildcard fix, and an end-to-end Router.add_deployment custom pricing scenario. |
Reviews (4): Last reviewed commit: "fix(openai-responses): strip cache_contr..." | Re-trigger Greptile
* feat(model_prices): add gemini-3.1-flash-lite pricing with standard/batch/flex/priority tiers * fix pricing * add service tier --------- Co-authored-by: shin-berri <[email protected]>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 2 potential issues.
Autofix Details
Bugbot Autofix prepared fixes for both issues found in the latest run.
- ✅ Fixed: Default Azure config receives base_model instead of deployment name
- Updated the default Azure else branch in get_supported_openai_params.py to pass _azure_detection_model so it matches map_openai_params in utils.py.
- ✅ Fixed: Missing base_model for GPT-5 reasoning summary stripping
- Updated the Azure GPT-5 detection in completion() to use
base_model or modelso reasoning summary alias stripping fires for non-standard deployment names.
- Updated the Azure GPT-5 detection in completion() to use
Preview (71cf706444)
diff --git a/litellm/litellm_core_utils/get_supported_openai_params.py b/litellm/litellm_core_utils/get_supported_openai_params.py
--- a/litellm/litellm_core_utils/get_supported_openai_params.py
+++ b/litellm/litellm_core_utils/get_supported_openai_params.py
@@ -11,6 +11,7 @@
request_type: Literal[
"chat_completion", "embeddings", "transcription"
] = "chat_completion",
+ base_model: Optional[str] = None,
) -> Optional[list]:
"""
Returns the supported openai params for a given model + provider
@@ -20,6 +21,11 @@
get_supported_openai_params(model="anthropic.claude-3", custom_llm_provider="bedrock")
```
+ Args:
+ base_model: For Azure, the true underlying model (e.g. ``"azure/gpt-5.2"``)
+ when the deployment name differs. Used for model-type detection so that
+ non-standard deployment names route to the correct config.
+
Returns:
- List if custom_llm_provider is mapped
- None if unmapped
@@ -32,17 +38,21 @@
if custom_llm_provider in LlmProvidersSet:
provider_config = litellm.ProviderConfigManager.get_provider_chat_config(
- model=model, provider=LlmProviders(custom_llm_provider)
+ model=model,
+ provider=LlmProviders(custom_llm_provider),
+ base_model=base_model,
)
elif custom_llm_provider.split("/")[0] in LlmProvidersSet:
provider_config = litellm.ProviderConfigManager.get_provider_chat_config(
- model=model, provider=LlmProviders(custom_llm_provider.split("/")[0])
+ model=model,
+ provider=LlmProviders(custom_llm_provider.split("/")[0]),
+ base_model=base_model,
)
else:
provider_config = None
if provider_config and request_type == "chat_completion":
- return provider_config.get_supported_openai_params(model=model)
+ return provider_config.get_supported_openai_params(model=base_model or model)
if custom_llm_provider == "bedrock":
return litellm.AmazonConverseConfig().get_supported_openai_params(model=model)
@@ -130,16 +140,23 @@
model=model
)
elif custom_llm_provider == "azure":
- if litellm.AzureOpenAIO1Config().is_o_series_model(model=model):
+ _azure_detection_model = base_model or model
+ if litellm.AzureOpenAIO1Config().is_o_series_model(
+ model=_azure_detection_model
+ ):
return litellm.AzureOpenAIO1Config().get_supported_openai_params(
- model=model
+ model=_azure_detection_model
)
- elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=model):
+ elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(
+ model=_azure_detection_model
+ ):
return litellm.AzureOpenAIGPT5Config().get_supported_openai_params(
- model=model
+ model=_azure_detection_model
)
else:
- return litellm.AzureOpenAIConfig().get_supported_openai_params(model=model)
+ return litellm.AzureOpenAIConfig().get_supported_openai_params(
+ model=_azure_detection_model
+ )
elif custom_llm_provider == "openrouter":
return litellm.OpenrouterConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "vercel_ai_gateway":
diff --git a/litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py b/litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
--- a/litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
+++ b/litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
@@ -1476,7 +1476,7 @@
for choice in choices:
if choice.delta.content is not None and len(choice.delta.content) > 0:
text += choice.delta.content
- if choice.delta.tool_calls is not None:
+ if choice.delta.tool_calls:
partial_json = ""
for tool in choice.delta.tool_calls:
if (
diff --git a/litellm/llms/azure/azure.py b/litellm/llms/azure/azure.py
--- a/litellm/llms/azure/azure.py
+++ b/litellm/llms/azure/azure.py
@@ -239,7 +239,9 @@
)
data = {"model": None, "messages": messages, **optional_params}
- elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=model):
+ elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(
+ model=litellm_params.get("base_model") or model
+ ):
data = litellm.AzureOpenAIGPT5Config().transform_request(
model=model,
messages=messages,
diff --git a/litellm/main.py b/litellm/main.py
--- a/litellm/main.py
+++ b/litellm/main.py
@@ -1491,7 +1491,9 @@
provider.value for provider in LlmProviders
]:
provider_config = ProviderConfigManager.get_provider_chat_config(
- model=model, provider=LlmProviders(custom_llm_provider)
+ model=model,
+ provider=LlmProviders(custom_llm_provider),
+ base_model=base_model,
)
if provider_config is not None:
@@ -1550,6 +1552,7 @@
"safety_identifier": safety_identifier,
"service_tier": service_tier,
"allowed_openai_params": kwargs.get("allowed_openai_params"),
+ "base_model": base_model,
}
optional_params = get_optional_params(
**optional_param_args, **non_default_params
@@ -1713,7 +1716,7 @@
and OpenAIGPT5Config.is_model_gpt_5_model(model)
) or (
custom_llm_provider == "azure"
- and litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model)
+ and litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(base_model or model)
):
optional_params, _ = strip_reasoning_summary_aliases_from_optional_params(
optional_params
@@ -1766,7 +1769,12 @@
if max_retries is not None:
optional_params["max_retries"] = max_retries
- if litellm.AzureOpenAIO1Config().is_o_series_model(model=model):
+ # Use base_model (the true underlying model) for model-type
+ # detection when the deployment name differs from the model name.
+ _azure_detection_model = base_model or model
+ if litellm.AzureOpenAIO1Config().is_o_series_model(
+ model=_azure_detection_model
+ ):
## LOAD CONFIG - if set
config = litellm.AzureOpenAIO1Config.get_config()
for k, v in config.items():
diff --git a/litellm/utils.py b/litellm/utils.py
--- a/litellm/utils.py
+++ b/litellm/utils.py
@@ -4019,16 +4019,23 @@
thinking: Optional[AnthropicThinkingParam] = None,
web_search_options: Optional[OpenAIWebSearchOptions] = None,
safety_identifier: Optional[str] = None,
+ base_model: Optional[str] = None,
**kwargs,
):
passed_params = locals().copy()
special_params = passed_params.pop("kwargs")
+ # Remove base_model from passed_params so it doesn't interfere with
+ # non_default_params / _check_valid_arg — it's a routing hint, not an
+ # OpenAI param.
+ passed_params.pop("base_model", None)
provider_config: Optional[BaseConfig] = None
if custom_llm_provider is not None and custom_llm_provider in [
provider.value for provider in LlmProviders
]:
provider_config = ProviderConfigManager.get_provider_chat_config(
- model=model, provider=LlmProviders(custom_llm_provider)
+ model=model,
+ provider=LlmProviders(custom_llm_provider),
+ base_model=base_model,
)
non_default_params = pre_process_non_default_params(
passed_params=passed_params,
@@ -4091,7 +4098,7 @@
sys.modules[__name__], "get_supported_openai_params"
)
supported_params = get_supported_openai_params(
- model=model, custom_llm_provider=custom_llm_provider
+ model=model, custom_llm_provider=custom_llm_provider, base_model=base_model
)
if supported_params is None:
supported_params = get_supported_openai_params(
@@ -4702,22 +4709,27 @@
),
)
elif custom_llm_provider == "azure":
- if litellm.AzureOpenAIO1Config().is_o_series_model(model=model):
+ _azure_detection_model = base_model or model
+ if litellm.AzureOpenAIO1Config().is_o_series_model(
+ model=_azure_detection_model
+ ):
optional_params = litellm.AzureOpenAIO1Config().map_openai_params(
non_default_params=non_default_params,
optional_params=optional_params,
- model=model,
+ model=_azure_detection_model,
drop_params=(
drop_params
if drop_params is not None and isinstance(drop_params, bool)
else False
),
)
- elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=model):
+ elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(
+ model=_azure_detection_model
+ ):
optional_params = litellm.AzureOpenAIGPT5Config().map_openai_params(
non_default_params=non_default_params,
optional_params=optional_params,
- model=model,
+ model=_azure_detection_model,
drop_params=(
drop_params
if drop_params is not None and isinstance(drop_params, bool)
@@ -4739,7 +4751,7 @@
optional_params = litellm.AzureOpenAIConfig().map_openai_params(
non_default_params=non_default_params,
optional_params=optional_params,
- model=model,
+ model=_azure_detection_model,
api_version=api_version, # type: ignore
drop_params=(
drop_params
@@ -8124,10 +8136,8 @@
# Format: (factory_function, needs_model_parameter: bool)
LlmProviders.OPENAI: (lambda: litellm.OpenAIGPTConfig(), False),
LlmProviders.ANTHROPIC: (lambda: litellm.AnthropicConfig(), False),
- LlmProviders.AZURE: (
- lambda model: ProviderConfigManager._get_azure_config(model),
- True,
- ),
+ # AZURE is handled as a special case in get_provider_chat_config()
+ # so that base_model can be threaded through for model-type detection.
LlmProviders.AZURE_AI: (
lambda model: ProviderConfigManager._get_azure_ai_config(model),
True,
@@ -8267,11 +8277,19 @@
}
@staticmethod
- def _get_azure_config(model: str) -> BaseConfig:
- """Get Azure config based on model type."""
- if litellm.AzureOpenAIO1Config().is_o_series_model(model=model):
+ def _get_azure_config(model: str, base_model: Optional[str] = None) -> BaseConfig:
+ """Get Azure config based on model type.
+
+ When *base_model* is provided (e.g. ``"azure/gpt-5.2"``), it is used
+ for model-type detection instead of *model* (the deployment name).
+ This allows non-standard deployment names like ``"azure/foo"`` to be
+ routed through the correct config when the user specifies the true
+ underlying model via ``base_model``.
+ """
+ detection_model = base_model or model
+ if litellm.AzureOpenAIO1Config().is_o_series_model(model=detection_model):
return litellm.AzureOpenAIO1Config()
- if litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=model):
+ if litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=detection_model):
return litellm.AzureOpenAIGPT5Config()
return litellm.AzureOpenAIConfig()
@@ -8329,13 +8347,18 @@
@staticmethod
def get_provider_chat_config( # noqa: PLR0915
- model: str, provider: LlmProviders
+ model: str,
+ provider: LlmProviders,
+ base_model: Optional[str] = None,
) -> Optional[BaseConfig]:
"""
Returns the provider config for a given provider.
Uses O(1) dictionary lookup for fast provider resolution.
Python classes take priority over JSON (they have custom overrides).
+
+ For Azure, *base_model* (when set) drives model-type detection so that
+ non-standard deployment names still route to the correct config.
"""
# Handle OpenAI special cases (O-series and GPT-5 models)
if provider == LlmProviders.OPENAI:
@@ -8344,6 +8367,12 @@
if litellm.OpenAIGPT5Config.is_model_gpt_5_model(model=model):
return litellm.OpenAIGPT5Config()
+ # Handle Azure before the generic map so base_model can be threaded through
+ if provider == LlmProviders.AZURE:
+ return ProviderConfigManager._get_azure_config(
+ model=model, base_model=base_model
+ )
+
# Initialize provider config map lazily (avoids circular imports)
if ProviderConfigManager._PROVIDER_CONFIG_MAP is None:
ProviderConfigManager._PROVIDER_CONFIG_MAP = (
diff --git a/tests/test_litellm/llms/anthropic/experimental_pass_through/adapters/test_anthropic_experimental_pass_through_adapters_transformation.py b/tests/test_litellm/llms/anthropic/experimental_pass_through/adapters/test_anthropic_experimental_pass_through_adapters_transformation.py
--- a/tests/test_litellm/llms/anthropic/experimental_pass_through/adapters/test_anthropic_experimental_pass_through_adapters_transformation.py
+++ b/tests/test_litellm/llms/anthropic/experimental_pass_through/adapters/test_anthropic_experimental_pass_through_adapters_transformation.py
@@ -1203,6 +1203,51 @@
assert content_block_start["id"] == "toolu_bdrk_013xRVejhv3ybmLEGCoZib2b"
+def test_streaming_chunk_with_text_and_empty_tool_calls_returns_text_delta():
+ """
+ Some OpenAI-compatible providers emit `tool_calls: []` on regular text chunks.
+
+ Empty tool_calls should be treated as no tool call so the Anthropic adapter
+ does not shadow text with an empty input_json_delta.
+ """
+ choices = [
+ StreamingChoices(
+ finish_reason=None,
+ index=0,
+ delta=Delta(
+ provider_specific_fields=None,
+ content="Hello from vLLM",
+ role="assistant",
+ function_call=None,
+ tool_calls=[],
+ audio=None,
+ ),
+ logprobs=None,
+ )
+ ]
+
+ adapter = LiteLLMAnthropicMessagesAdapter()
+
+ (
+ type_of_content,
+ content_block_delta,
+ ) = adapter._translate_streaming_openai_chunk_to_anthropic(choices=choices)
+
+ assert type_of_content == "text_delta"
+ assert content_block_delta["type"] == "text_delta"
+ assert content_block_delta["text"] == "Hello from vLLM"
+
+ (
+ block_type,
+ content_block_start,
+ ) = adapter._translate_streaming_openai_chunk_to_anthropic_content_block(
+ choices=choices
+ )
+
+ assert block_type == "text"
+ assert content_block_start == {"type": "text", "text": ""}
+
+
# ============================================================================
# Cache Control Transformation Tests
# ============================================================================
diff --git a/tests/test_litellm/llms/azure/chat/test_azure_base_model_routing.py b/tests/test_litellm/llms/azure/chat/test_azure_base_model_routing.py
new file mode 100644
--- /dev/null
+++ b/tests/test_litellm/llms/azure/chat/test_azure_base_model_routing.py
@@ -1,0 +1,274 @@
+"""Tests for decoupling Azure deployment IDs from underlying model names.
+
+When users name their Azure deployment something non-standard (e.g. "my-deployment-id"),
+setting ``base_model`` should drive model-type detection (o-series, gpt-5,
+etc.) so the correct config, supported params, and param mapping are used.
+"""
+
+import pytest
+
+import litellm
+from litellm.llms.azure.chat.gpt_5_transformation import AzureOpenAIGPT5Config
+from litellm.llms.azure.chat.o_series_transformation import AzureOpenAIO1Config
+from litellm.utils import ProviderConfigManager, get_optional_params
+
+
+# ---------------------------------------------------------------------------
+# _get_azure_config — routes to the correct config based on base_model
+# ---------------------------------------------------------------------------
+class TestGetAzureConfigWithBaseModel:
+ """ProviderConfigManager._get_azure_config should use base_model for detection."""
+
+ def test_should_return_gpt5_config_when_base_model_is_gpt5(self):
+ config = ProviderConfigManager._get_azure_config(
+ model="my-deployment-id", base_model="azure/gpt-5.2"
+ )
+ assert isinstance(config, AzureOpenAIGPT5Config)
+
+ def test_should_return_o_series_config_when_base_model_is_o_series(self):
+ config = ProviderConfigManager._get_azure_config(
+ model="my-deployment-id", base_model="azure/o4-mini"
+ )
+ assert isinstance(config, AzureOpenAIO1Config)
+
+ def test_should_return_default_config_when_base_model_is_regular(self):
+ config = ProviderConfigManager._get_azure_config(
+ model="my-deployment-id", base_model="azure/gpt-4o"
+ )
+ assert type(config).__name__ == "AzureOpenAIConfig"
+
+ def test_should_fallback_to_model_when_base_model_is_none(self):
+ config = ProviderConfigManager._get_azure_config(
+ model="gpt-5.2", base_model=None
+ )
+ assert isinstance(config, AzureOpenAIGPT5Config)
+
+ def test_should_return_default_config_when_both_are_non_standard(self):
+ config = ProviderConfigManager._get_azure_config(
+ model="my-deployment-id", base_model=None
+ )
+ assert type(config).__name__ == "AzureOpenAIConfig"
+
+
+# ---------------------------------------------------------------------------
+# get_provider_chat_config — threads base_model through for Azure
+# ---------------------------------------------------------------------------
+class TestGetProviderChatConfigWithBaseModel:
+ """get_provider_chat_config should pass base_model to Azure config selection."""
+
+ def test_should_return_gpt5_config_for_custom_deployment_with_base_model(self):
+ from litellm.types.utils import LlmProviders
+
+ config = ProviderConfigManager.get_provider_chat_config(
+ model="my-deployment-id",
+ provider=LlmProviders.AZURE,
+ base_model="azure/gpt-5",
+ )
+ assert isinstance(config, AzureOpenAIGPT5Config)
+
+ def test_should_return_o_series_config_for_custom_deployment_with_base_model(self):
+ from litellm.types.utils import LlmProviders
+
+ config = ProviderConfigManager.get_provider_chat_config(
+ model="my-other-deployment",
+ provider=LlmProviders.AZURE,
+ base_model="azure/o3-mini",
+ )
+ assert isinstance(config, AzureOpenAIO1Config)
+
+
+# ---------------------------------------------------------------------------
+# get_supported_openai_params — base_model drives Azure param detection
+# ---------------------------------------------------------------------------
+class TestGetSupportedOpenAIParamsWithBaseModel:
+ """get_supported_openai_params should use base_model for Azure detection."""
+
+ def test_should_return_gpt5_params_for_custom_deployment_with_gpt5_base_model(
+ self,
+ ):
+ params = litellm.get_supported_openai_params(
+ model="my-deployment-id",
+ custom_llm_provider="azure",
+ base_model="azure/gpt-5",
+ )
+ assert params is not None
+ assert "reasoning_effort" in params
+ # gpt-5 maps max_tokens -> max_completion_tokens, verifying we got GPT-5 config
+ assert "max_completion_tokens" in params
+
+ def test_should_return_o_series_params_for_custom_deployment_with_o_series_base_model(
+ self,
+ ):
+ params = litellm.get_supported_openai_params(
+ model="my-other-deployment",
+ custom_llm_provider="azure",
+ base_model="azure/o4-mini",
+ )
+ assert params is not None
+ assert "reasoning_effort" in params
+
+ def test_should_return_regular_params_when_no_base_model(self):
+ """When base_model is not set and model is non-standard, default Azure config."""
+ params = litellm.get_supported_openai_params(
+ model="my-deployment-id",
+ custom_llm_provider="azure",
+ )
+ assert params is not None
+ # Default Azure config supports temperature
+ assert "temperature" in params
+
+
+# ---------------------------------------------------------------------------
+# get_optional_params — base_model drives Azure param mapping
+# ---------------------------------------------------------------------------
+class TestGetOptionalParamsWithBaseModel:
+ """get_optional_params should use base_model for Azure model-type detection."""
+
+ def test_should_map_max_tokens_for_custom_deployment_with_gpt5_base_model(self):
+ """A non-standard deployment name + gpt-5 base_model should map max_tokens -> max_completion_tokens."""
+ params = get_optional_params(
+ model="my-deployment-id",
+ custom_llm_provider="azure",
+ max_tokens=100,
+ base_model="azure/gpt-5",
+ )
+ assert params.get("max_completion_tokens") == 100
+ assert "max_tokens" not in params
+
+ def test_should_keep_max_tokens_for_custom_deployment_without_base_model(self):
+ """A non-standard deployment name without base_model should use default Azure config."""
+ params = get_optional_params(
+ model="my-deployment-id",
+ custom_llm_provider="azure",
+ max_tokens=100,
+ api_version="2024-05-01-preview",
+ )
+ # Default AzureOpenAIConfig keeps max_tokens as-is (or maps based on api_version)
+ assert "max_tokens" in params or "max_completion_tokens" in params
+
+ def test_should_support_reasoning_effort_for_custom_deployment_with_o_series_base_model(
+ self,
+ ):
+ """A non-standard deployment name + o-series base_model should accept reasoning_effort."""
+ params = get_optional_params(
+ model="my-other-deployment",
+ custom_llm_provider="azure",
+ reasoning_effort="low",
+ base_model="azure/o4-mini",
+ )
+ assert params.get("reasoning_effort") == "low"
+
+ def test_should_reject_temperature_for_custom_deployment_with_gpt5_base_model(
+ self,
+ ):
+ """A non-standard deployment + gpt-5 base_model should reject temperature."""
+ with pytest.raises(litellm.UnsupportedParamsError):
+ get_optional_params(
+ model="my-deployment-id",
+ custom_llm_provider="azure",
+ temperature=0.5,
+ base_model="azure/gpt-5",
+ )
+
+
+# ---------------------------------------------------------------------------
+# Backward compatibility — existing patterns still work
+# ---------------------------------------------------------------------------
+class TestBackwardCompatibility:
+ """Existing model-name-based and prefix-based patterns must keep working."""
+
+ def test_should_detect_gpt5_from_model_name(self):
+ config = ProviderConfigManager._get_azure_config(model="gpt-5.2")
+ assert isinstance(config, AzureOpenAIGPT5Config)
+
+ def test_should_detect_gpt5_from_gpt5_series_prefix(self):
+ config = ProviderConfigManager._get_azure_config(
+ model="gpt5_series/my-deployment"
+ )
+ assert isinstance(config, AzureOpenAIGPT5Config)
+
+ def test_should_detect_o_series_from_model_name(self):
+ config = ProviderConfigManager._get_azure_config(model="o4-mini")
+ assert isinstance(config, AzureOpenAIO1Config)
+
+ def test_should_detect_o_series_from_o_series_prefix(self):
+ config = ProviderConfigManager._get_azure_config(model="o_series/my-deployment")
+ assert isinstance(config, AzureOpenAIO1Config)
+
+ def test_should_handle_gpt5_chat_model_correctly(self):
+ """gpt-5-chat models should NOT be routed to GPT-5 config."""
+ config = ProviderConfigManager._get_azure_config(model="gpt-5-chat")
+ assert type(config).__name__ == "AzureOpenAIConfig"
+
+ def test_base_model_overrides_model_detection(self):
+ """base_model should take priority over model for type detection."""
+ # model looks like o-series, but base_model says gpt-5
+ config = ProviderConfigManager._get_azure_config(
+ model="o3-mini", base_model="azure/gpt-5.2"
+ )
+ assert isinstance(config, AzureOpenAIGPT5Config)
+
+
+# ---------------------------------------------------------------------------
+# Deep config method awareness — base_model flows into config internals
+# ---------------------------------------------------------------------------
+class TestBaseModelFlowsIntoConfigInternals:
+ """base_model should be used by config internal methods (e.g. is_model_gpt_5_2_model)."""
+
+ def test_should_support_logprobs_for_prefixed_deployment_with_gpt52_base_model(
+ self,
+ ):
+ """Deployment 'my-gpt-5.2' with base_model='azure/gpt-5.2' should support logprobs."""
+ params = litellm.get_supported_openai_params(
+ model="gpt5_series/my-gpt-5.2",
+ custom_llm_provider="azure",
+ base_model="azure/gpt-5.2",
+ )
+ assert params is not None
+ assert "logprobs" in params
+ assert "top_logprobs" in params
+
+ def test_should_support_logprobs_for_plain_deployment_with_gpt52_base_model(self):
+ """Deployment 'my-deployment-id' with base_model='azure/gpt-5.2' should support logprobs."""
+ params = litellm.get_supported_openai_params(
+ model="my-deployment-id",
+ custom_llm_provider="azure",
+ base_model="azure/gpt-5.2",
+ )
+ assert params is not None
+ assert "logprobs" in params
+ assert "top_logprobs" in params
+
+ def test_should_not_support_logprobs_for_gpt5_base_model(self):
+ """Deployment with base_model='azure/gpt-5' (not 5.2) should NOT support logprobs."""
+ params = litellm.get_supported_openai_params(
+ model="my-deployment-id",
+ custom_llm_provider="azure",
+ base_model="azure/gpt-5",
+ )
+ assert params is not None
+ assert "logprobs" not in params
+ assert "top_logprobs" not in params
+
+ def test_should_pass_logprobs_through_get_optional_params(self):
+ """logprobs should pass validation in get_optional_params when base_model is gpt-5.2."""
+ params = get_optional_params(
+ model="gpt5_series/my-gpt-5.2",
+ custom_llm_provider="azure",
+ logprobs=True,
+ top_logprobs=5,
+ base_model="azure/gpt-5.2",
+ )
+ assert params.get("logprobs") is True
+ assert params.get("top_logprobs") == 5
+
+ def test_should_map_max_tokens_for_prefixed_deployment_with_gpt5_base_model(self):
+ """my-gpt-5.2 with base_model should correctly map max_tokens -> max_completion_tokens."""
+ params = get_optional_params(
+ model="gpt5_series/my-gpt-5.2",
+ custom_llm_provider="azure",
+ max_tokens=200,
+ base_model="azure/gpt-5.2",
+ )
+ assert params.get("max_completion_tokens") == 200
+ assert "max_tokens" not in paramsYou can send follow-ups to the cloud agent here.
|
|
…PI requests (#28431) Squash-merged by litellm-agent from cwang-otto's PR.
71cf706 to
0372f6f
Compare
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
Autofix Details
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Non-Azure providers affected by
base_modelin supported params- Restricted the
base_modelsubstitution to only the Azure provider so non-Azure providers always use the actual model name forget_supported_openai_paramslookups.
- Restricted the
Preview (399de6340d)
diff --git a/litellm/litellm_core_utils/get_supported_openai_params.py b/litellm/litellm_core_utils/get_supported_openai_params.py
--- a/litellm/litellm_core_utils/get_supported_openai_params.py
+++ b/litellm/litellm_core_utils/get_supported_openai_params.py
@@ -11,6 +11,7 @@
request_type: Literal[
"chat_completion", "embeddings", "transcription"
] = "chat_completion",
+ base_model: Optional[str] = None,
) -> Optional[list]:
"""
Returns the supported openai params for a given model + provider
@@ -20,6 +21,11 @@
get_supported_openai_params(model="anthropic.claude-3", custom_llm_provider="bedrock")
```
+ Args:
+ base_model: For Azure, the true underlying model (e.g. ``"azure/gpt-5.2"``)
+ when the deployment name differs. Used for model-type detection so that
+ non-standard deployment names route to the correct config.
+
Returns:
- List if custom_llm_provider is mapped
- None if unmapped
@@ -32,17 +38,24 @@
if custom_llm_provider in LlmProvidersSet:
provider_config = litellm.ProviderConfigManager.get_provider_chat_config(
- model=model, provider=LlmProviders(custom_llm_provider)
+ model=model,
+ provider=LlmProviders(custom_llm_provider),
+ base_model=base_model,
)
elif custom_llm_provider.split("/")[0] in LlmProvidersSet:
provider_config = litellm.ProviderConfigManager.get_provider_chat_config(
- model=model, provider=LlmProviders(custom_llm_provider.split("/")[0])
+ model=model,
+ provider=LlmProviders(custom_llm_provider.split("/")[0]),
+ base_model=base_model,
)
else:
provider_config = None
if provider_config and request_type == "chat_completion":
- return provider_config.get_supported_openai_params(model=model)
+ effective_model = (
+ base_model if custom_llm_provider == "azure" and base_model else model
+ )
+ return provider_config.get_supported_openai_params(model=effective_model)
if custom_llm_provider == "bedrock":
return litellm.AmazonConverseConfig().get_supported_openai_params(model=model)
@@ -130,13 +143,18 @@
model=model
)
elif custom_llm_provider == "azure":
- if litellm.AzureOpenAIO1Config().is_o_series_model(model=model):
+ _azure_detection_model = base_model or model
+ if litellm.AzureOpenAIO1Config().is_o_series_model(
+ model=_azure_detection_model
+ ):
return litellm.AzureOpenAIO1Config().get_supported_openai_params(
- model=model
+ model=_azure_detection_model
)
- elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=model):
+ elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(
+ model=_azure_detection_model
+ ):
return litellm.AzureOpenAIGPT5Config().get_supported_openai_params(
- model=model
+ model=_azure_detection_model
)
else:
return litellm.AzureOpenAIConfig().get_supported_openai_params(model=model)
diff --git a/litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py b/litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
--- a/litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
+++ b/litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
@@ -1476,7 +1476,7 @@
for choice in choices:
if choice.delta.content is not None and len(choice.delta.content) > 0:
text += choice.delta.content
- if choice.delta.tool_calls is not None:
+ if choice.delta.tool_calls:
partial_json = ""
for tool in choice.delta.tool_calls:
if (
diff --git a/litellm/llms/azure/azure.py b/litellm/llms/azure/azure.py
--- a/litellm/llms/azure/azure.py
+++ b/litellm/llms/azure/azure.py
@@ -239,7 +239,9 @@
)
data = {"model": None, "messages": messages, **optional_params}
- elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=model):
+ elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(
+ model=litellm_params.get("base_model") or model
+ ):
data = litellm.AzureOpenAIGPT5Config().transform_request(
model=model,
messages=messages,
diff --git a/litellm/llms/openai/responses/transformation.py b/litellm/llms/openai/responses/transformation.py
--- a/litellm/llms/openai/responses/transformation.py
+++ b/litellm/llms/openai/responses/transformation.py
@@ -126,9 +126,21 @@
litellm_params: GenericLiteLLMParams,
headers: dict,
) -> Dict:
- """No transform applied since inputs are in OpenAI spec already"""
+ """Strip Anthropic-only `cache_control` markers before sending to OpenAI.
+ OpenAI's Responses API rejects unknown fields on input content blocks
+ with HTTP 400 ("Unknown parameter: 'input[0].content[0].cache_control'").
+ Chat Completions strips these in
+ `remove_cache_control_flag_from_messages_and_tools`; mirror that here.
+ """
+
input = self._validate_input_param(input)
+ tools = response_api_optional_request_params.get("tools")
+ input, tools = self.remove_cache_control_flag_from_input_and_tools(
+ model=model, input=input, tools=tools
+ )
+ if tools is not None:
+ response_api_optional_request_params["tools"] = tools
final_request_params = dict(
ResponsesAPIRequestParams(
model=model, input=input, **response_api_optional_request_params
@@ -137,6 +149,38 @@
return final_request_params
+ def remove_cache_control_flag_from_input_and_tools(
+ self,
+ model: str, # allows overrides to selectively run this
+ input: Union[str, ResponseInputParam],
+ tools: Optional[List[ALL_RESPONSES_API_TOOL_PARAMS]] = None,
+ ) -> Tuple[
+ Union[str, ResponseInputParam],
+ Optional[List[ALL_RESPONSES_API_TOOL_PARAMS]],
+ ]:
+ """Sibling of `remove_cache_control_flag_from_messages_and_tools` on
+ the chat path. Strips Anthropic-only `cache_control` markers from
+ Responses API input content blocks and tools.
+
+ `filter_value_from_dict` mutates each dict in place, so the same
+ objects are returned.
+ """
+ from litellm.litellm_core_utils.prompt_templates.common_utils import (
+ filter_value_from_dict,
+ )
+
+ if isinstance(input, list):
+ for item in input:
+ if isinstance(item, dict):
+ filter_value_from_dict(cast(dict, item), "cache_control")
+
+ if tools is not None:
+ for tool in tools:
+ if isinstance(tool, dict):
+ filter_value_from_dict(cast(dict, tool), "cache_control")
+
+ return input, tools
+
def _validate_input_param(
self, input: Union[str, ResponseInputParam]
) -> Union[str, ResponseInputParam]:
diff --git a/litellm/main.py b/litellm/main.py
--- a/litellm/main.py
+++ b/litellm/main.py
@@ -1491,7 +1491,9 @@
provider.value for provider in LlmProviders
]:
provider_config = ProviderConfigManager.get_provider_chat_config(
- model=model, provider=LlmProviders(custom_llm_provider)
+ model=model,
+ provider=LlmProviders(custom_llm_provider),
+ base_model=base_model,
)
if provider_config is not None:
@@ -1550,6 +1552,7 @@
"safety_identifier": safety_identifier,
"service_tier": service_tier,
"allowed_openai_params": kwargs.get("allowed_openai_params"),
+ "base_model": base_model,
}
optional_params = get_optional_params(
**optional_param_args, **non_default_params
@@ -1766,7 +1769,12 @@
if max_retries is not None:
optional_params["max_retries"] = max_retries
- if litellm.AzureOpenAIO1Config().is_o_series_model(model=model):
+ # Use base_model (the true underlying model) for model-type
+ # detection when the deployment name differs from the model name.
+ _azure_detection_model = base_model or model
+ if litellm.AzureOpenAIO1Config().is_o_series_model(
+ model=_azure_detection_model
+ ):
## LOAD CONFIG - if set
config = litellm.AzureOpenAIO1Config.get_config()
for k, v in config.items():
diff --git a/litellm/utils.py b/litellm/utils.py
--- a/litellm/utils.py
+++ b/litellm/utils.py
@@ -2933,6 +2933,13 @@
except Exception:
existing_model = {}
model_cost_key = key
+ # ``get_model_info`` returns ``litellm_provider: None`` when the
+ # provider is unknown (e.g. custom deployments registered via
+ # ``Router.add_deployment``). Persisting that None into
+ # ``litellm.model_cost`` causes ``_check_provider_match`` to drop
+ # custom pricing on subsequent cost lookups.
+ if existing_model.get("litellm_provider") is None:
+ existing_model.pop("litellm_provider", None)
## override / add new keys to the existing model cost dictionary
updated_dictionary = _update_dictionary(existing_model, value)
litellm.model_cost.setdefault(model_cost_key, {}).update(updated_dictionary)
@@ -4019,16 +4026,23 @@
thinking: Optional[AnthropicThinkingParam] = None,
web_search_options: Optional[OpenAIWebSearchOptions] = None,
safety_identifier: Optional[str] = None,
+ base_model: Optional[str] = None,
**kwargs,
):
passed_params = locals().copy()
special_params = passed_params.pop("kwargs")
+ # Remove base_model from passed_params so it doesn't interfere with
+ # non_default_params / _check_valid_arg — it's a routing hint, not an
+ # OpenAI param.
+ passed_params.pop("base_model", None)
provider_config: Optional[BaseConfig] = None
if custom_llm_provider is not None and custom_llm_provider in [
provider.value for provider in LlmProviders
]:
provider_config = ProviderConfigManager.get_provider_chat_config(
- model=model, provider=LlmProviders(custom_llm_provider)
+ model=model,
+ provider=LlmProviders(custom_llm_provider),
+ base_model=base_model,
)
non_default_params = pre_process_non_default_params(
passed_params=passed_params,
@@ -4091,7 +4105,7 @@
sys.modules[__name__], "get_supported_openai_params"
)
supported_params = get_supported_openai_params(
- model=model, custom_llm_provider=custom_llm_provider
+ model=model, custom_llm_provider=custom_llm_provider, base_model=base_model
)
if supported_params is None:
supported_params = get_supported_openai_params(
@@ -4702,22 +4716,27 @@
),
)
elif custom_llm_provider == "azure":
- if litellm.AzureOpenAIO1Config().is_o_series_model(model=model):
+ _azure_detection_model = base_model or model
+ if litellm.AzureOpenAIO1Config().is_o_series_model(
+ model=_azure_detection_model
+ ):
optional_params = litellm.AzureOpenAIO1Config().map_openai_params(
non_default_params=non_default_params,
optional_params=optional_params,
- model=model,
+ model=_azure_detection_model,
drop_params=(
drop_params
if drop_params is not None and isinstance(drop_params, bool)
else False
),
)
- elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=model):
+ elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(
+ model=_azure_detection_model
+ ):
optional_params = litellm.AzureOpenAIGPT5Config().map_openai_params(
non_default_params=non_default_params,
optional_params=optional_params,
- model=model,
+ model=_azure_detection_model,
drop_params=(
drop_params
if drop_params is not None and isinstance(drop_params, bool)
@@ -4739,7 +4758,7 @@
optional_params = litellm.AzureOpenAIConfig().map_openai_params(
non_default_params=non_default_params,
optional_params=optional_params,
- model=model,
+ model=_azure_detection_model,
api_version=api_version, # type: ignore
drop_params=(
drop_params
@@ -5510,9 +5529,15 @@
def _check_provider_match(model_info: dict, custom_llm_provider: Optional[str]) -> bool:
"""
Check if the model info provider matches the custom provider.
+
+ A missing ``litellm_provider`` key and a ``litellm_provider`` set to
+ ``None`` both mean "no specific provider constraint" and are treated
+ as a wildcard match. ``register_model`` may persist ``None`` here via
+ ``get_model_info`` when a deployment is registered without a provider,
+ so normalising the two cases keeps custom pricing applied consistently.
"""
if custom_llm_provider and (
- "litellm_provider" in model_info
+ model_info.get("litellm_provider") is not None
and model_info["litellm_provider"] != custom_llm_provider
):
if custom_llm_provider == "vertex_ai" and model_info[
@@ -8124,10 +8149,8 @@
# Format: (factory_function, needs_model_parameter: bool)
LlmProviders.OPENAI: (lambda: litellm.OpenAIGPTConfig(), False),
LlmProviders.ANTHROPIC: (lambda: litellm.AnthropicConfig(), False),
- LlmProviders.AZURE: (
- lambda model: ProviderConfigManager._get_azure_config(model),
- True,
- ),
+ # AZURE is handled as a special case in get_provider_chat_config()
+ # so that base_model can be threaded through for model-type detection.
LlmProviders.AZURE_AI: (
lambda model: ProviderConfigManager._get_azure_ai_config(model),
True,
@@ -8267,11 +8290,19 @@
}
@staticmethod
- def _get_azure_config(model: str) -> BaseConfig:
- """Get Azure config based on model type."""
- if litellm.AzureOpenAIO1Config().is_o_series_model(model=model):
+ def _get_azure_config(model: str, base_model: Optional[str] = None) -> BaseConfig:
+ """Get Azure config based on model type.
+
+ When *base_model* is provided (e.g. ``"azure/gpt-5.2"``), it is used
+ for model-type detection instead of *model* (the deployment name).
+ This allows non-standard deployment names like ``"azure/foo"`` to be
+ routed through the correct config when the user specifies the true
+ underlying model via ``base_model``.
+ """
+ detection_model = base_model or model
+ if litellm.AzureOpenAIO1Config().is_o_series_model(model=detection_model):
return litellm.AzureOpenAIO1Config()
- if litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=model):
+ if litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=detection_model):
return litellm.AzureOpenAIGPT5Config()
return litellm.AzureOpenAIConfig()
@@ -8329,13 +8360,18 @@
@staticmethod
def get_provider_chat_config( # noqa: PLR0915
- model: str, provider: LlmProviders
+ model: str,
+ provider: LlmProviders,
+ base_model: Optional[str] = None,
) -> Optional[BaseConfig]:
"""
Returns the provider config for a given provider.
Uses O(1) dictionary lookup for fast provider resolution.
Python classes take priority over JSON (they have custom overrides).
+
+ For Azure, *base_model* (when set) drives model-type detection so that
+ non-standard deployment names still route to the correct config.
"""
# Handle OpenAI special cases (O-series and GPT-5 models)
if provider == LlmProviders.OPENAI:
@@ -8344,6 +8380,12 @@
if litellm.OpenAIGPT5Config.is_model_gpt_5_model(model=model):
return litellm.OpenAIGPT5Config()
+ # Handle Azure before the generic map so base_model can be threaded through
+ if provider == LlmProviders.AZURE:
+ return ProviderConfigManager._get_azure_config(
+ model=model, base_model=base_model
+ )
+
# Initialize provider config map lazily (avoids circular imports)
if ProviderConfigManager._PROVIDER_CONFIG_MAP is None:
ProviderConfigManager._PROVIDER_CONFIG_MAP = (
diff --git a/model_prices_and_context_window.json b/model_prices_and_context_window.json
--- a/model_prices_and_context_window.json
+++ b/model_prices_and_context_window.json
@@ -15006,10 +15006,16 @@
"supports_service_tier": true
},
"gemini-3.1-flash-lite": {
- "cache_read_input_token_cost": 4.5e-08,
- "cache_read_input_token_cost_per_audio_token": 9e-08,
- "input_cost_per_audio_token": 9e-07,
- "input_cost_per_token": 4.5e-07,
+ "cache_read_input_token_cost": 2.5e-08,
+ "cache_read_input_token_cost_batches": 1.25e-08,
+ "cache_read_input_token_cost_flex": 1.25e-08,
+ "cache_read_input_token_cost_per_audio_token": 5e-08,
+ "cache_read_input_token_cost_priority": 4.5e-08,
+ "input_cost_per_audio_token": 5e-07,
+ "input_cost_per_token": 2.5e-07,
+ "input_cost_per_token_batches": 1.25e-07,
+ "input_cost_per_token_flex": 1.25e-07,
+ "input_cost_per_token_priority": 4.5e-07,
"litellm_provider": "vertex_ai-language-models",
"max_audio_length_hours": 8.4,
"max_audio_per_prompt": 1,
@@ -15021,9 +15027,12 @@
"max_video_length": 1,
"max_videos_per_prompt": 10,
"mode": "chat",
- "output_cost_per_reasoning_token": 2.7e-06,
- "output_cost_per_token": 2.7e-06,
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#gemini-models",
+ "output_cost_per_reasoning_token": 1.5e-06,
+ "output_cost_per_token": 1.5e-06,
+ "output_cost_per_token_batches": 7.5e-07,
+ "output_cost_per_token_flex": 7.5e-07,
+ "output_cost_per_token_priority": 2.7e-06,
+ "source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-3.1-flash-lite",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
@@ -17128,10 +17137,16 @@
"supports_service_tier": true
},
"gemini/gemini-3.1-flash-lite": {
- "cache_read_input_token_cost": 4.5e-08,
- "cache_read_input_token_cost_per_audio_token": 9e-08,
- "input_cost_per_audio_token": 9e-07,
- "input_cost_per_token": 4.5e-07,
+ "cache_read_input_token_cost": 2.5e-08,
+ "cache_read_input_token_cost_batches": 1.25e-08,
+ "cache_read_input_token_cost_flex": 1.25e-08,
+ "cache_read_input_token_cost_per_audio_token": 5e-08,
+ "cache_read_input_token_cost_priority": 4.5e-08,
+ "input_cost_per_audio_token": 5e-07,
+ "input_cost_per_token": 2.5e-07,
+ "input_cost_per_token_batches": 1.25e-07,
+ "input_cost_per_token_flex": 1.25e-07,
+ "input_cost_per_token_priority": 4.5e-07,
"litellm_provider": "gemini",
"max_audio_length_hours": 8.4,
"max_audio_per_prompt": 1,
@@ -17143,10 +17158,13 @@
"max_video_length": 1,
"max_videos_per_prompt": 10,
"mode": "chat",
- "output_cost_per_reasoning_token": 2.7e-06,
- "output_cost_per_token": 2.7e-06,
+ "output_cost_per_reasoning_token": 1.5e-06,
+ "output_cost_per_token": 1.5e-06,
+ "output_cost_per_token_batches": 7.5e-07,
+ "output_cost_per_token_flex": 7.5e-07,
+ "output_cost_per_token_priority": 2.7e-06,
"rpm": 15,
- "source": "https://ai.google.dev/gemini-api/docs/pricing",
+ "source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-3.1-flash-lite",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
@@ -33932,10 +33950,16 @@
"web_search_billing_unit": "per_query"
},
"vertex_ai/gemini-3.1-flash-lite": {
- "cache_read_input_token_cost": 4.5e-08,
- "cache_read_input_token_cost_per_audio_token": 9e-08,
- "input_cost_per_audio_token": 9e-07,
- "input_cost_per_token": 4.5e-07,
+ "cache_read_input_token_cost": 2.5e-08,
+ "cache_read_input_token_cost_batches": 1.25e-08,
+ "cache_read_input_token_cost_flex": 1.25e-08,
+ "cache_read_input_token_cost_per_audio_token": 5e-08,
+ "cache_read_input_token_cost_priority": 4.5e-08,
+ "input_cost_per_audio_token": 5e-07,
+ "input_cost_per_token": 2.5e-07,
+ "input_cost_per_token_batches": 1.25e-07,
+ "input_cost_per_token_flex": 1.25e-07,
+ "input_cost_per_token_priority": 4.5e-07,
"litellm_provider": "vertex_ai-language-models",
"max_audio_length_hours": 8.4,
"max_audio_per_prompt": 1,
@@ -33947,8 +33971,11 @@
"max_video_length": 1,
"max_videos_per_prompt": 10,
"mode": "chat",
- "output_cost_per_reasoning_token": 2.7e-06,
- "output_cost_per_token": 2.7e-06,
+ "output_cost_per_reasoning_token": 1.5e-06,
+ "output_cost_per_token": 1.5e-06,
+ "output_cost_per_token_batches": 7.5e-07,
+ "output_cost_per_token_flex": 7.5e-07,
+ "output_cost_per_token_priority": 2.7e-06,
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#gemini-models",
"supported_endpoints": [
"/v1/chat/completions",
diff --git a/tests/test_litellm/llms/anthropic/experimental_pass_through/adapters/test_anthropic_experimental_pass_through_adapters_transformation.py b/tests/test_litellm/llms/anthropic/experimental_pass_through/adapters/test_anthropic_experimental_pass_through_adapters_transformation.py
--- a/tests/test_litellm/llms/anthropic/experimental_pass_through/adapters/test_anthropic_experimental_pass_through_adapters_transformation.py
+++ b/tests/test_litellm/llms/anthropic/experimental_pass_through/adapters/test_anthropic_experimental_pass_through_adapters_transformation.py
@@ -1203,6 +1203,51 @@
assert content_block_start["id"] == "toolu_bdrk_013xRVejhv3ybmLEGCoZib2b"
+def test_streaming_chunk_with_text_and_empty_tool_calls_returns_text_delta():
+ """
+ Some OpenAI-compatible providers emit `tool_calls: []` on regular text chunks.
+
+ Empty tool_calls should be treated as no tool call so the Anthropic adapter
+ does not shadow text with an empty input_json_delta.
+ """
+ choices = [
+ StreamingChoices(
+ finish_reason=None,
+ index=0,
+ delta=Delta(
+ provider_specific_fields=None,
+ content="Hello from vLLM",
+ role="assistant",
+ function_call=None,
+ tool_calls=[],
+ audio=None,
+ ),
+ logprobs=None,
+ )
+ ]
+
+ adapter = LiteLLMAnthropicMessagesAdapter()
+
+ (
+ type_of_content,
+ content_block_delta,
+ ) = adapter._translate_streaming_openai_chunk_to_anthropic(choices=choices)
+
+ assert type_of_content == "text_delta"
+ assert content_block_delta["type"] == "text_delta"
+ assert content_block_delta["text"] == "Hello from vLLM"
+
+ (
+ block_type,
+ content_block_start,
+ ) = adapter._translate_streaming_openai_chunk_to_anthropic_content_block(
+ choices=choices
+ )
+
+ assert block_type == "text"
+ assert content_block_start == {"type": "text", "text": ""}
+
+
# ============================================================================
# Cache Control Transformation Tests
# ============================================================================
diff --git a/tests/test_litellm/llms/azure/chat/test_azure_base_model_routing.py b/tests/test_litellm/llms/azure/chat/test_azure_base_model_routing.py
new file mode 100644
--- /dev/null
+++ b/tests/test_litellm/llms/azure/chat/test_azure_base_model_routing.py
@@ -1,0 +1,274 @@
+"""Tests for decoupling Azure deployment IDs from underlying model names.
+
+When users name their Azure deployment something non-standard (e.g. "my-deployment-id"),
+setting ``base_model`` should drive model-type detection (o-series, gpt-5,
+etc.) so the correct config, supported params, and param mapping are used.
+"""
+
+import pytest
+
+import litellm
+from litellm.llms.azure.chat.gpt_5_transformation import AzureOpenAIGPT5Config
+from litellm.llms.azure.chat.o_series_transformation import AzureOpenAIO1Config
+from litellm.utils import ProviderConfigManager, get_optional_params
+
+
+# ---------------------------------------------------------------------------
+# _get_azure_config — routes to the correct config based on base_model
+# ---------------------------------------------------------------------------
+class TestGetAzureConfigWithBaseModel:
+ """ProviderConfigManager._get_azure_config should use base_model for detection."""
+
+ def test_should_return_gpt5_config_when_base_model_is_gpt5(self):
+ config = ProviderConfigManager._get_azure_config(
+ model="my-deployment-id", base_model="azure/gpt-5.2"
+ )
+ assert isinstance(config, AzureOpenAIGPT5Config)
+
+ def test_should_return_o_series_config_when_base_model_is_o_series(self):
+ config = ProviderConfigManager._get_azure_config(
+ model="my-deployment-id", base_model="azure/o4-mini"
+ )
+ assert isinstance(config, AzureOpenAIO1Config)
+
+ def test_should_return_default_config_when_base_model_is_regular(self):
+ config = ProviderConfigManager._get_azure_config(
+ model="my-deployment-id", base_model="azure/gpt-4o"
+ )
+ assert type(config).__name__ == "AzureOpenAIConfig"
+
+ def test_should_fallback_to_model_when_base_model_is_none(self):
+ config = ProviderConfigManager._get_azure_config(
+ model="gpt-5.2", base_model=None
+ )
+ assert isinstance(config, AzureOpenAIGPT5Config)
+
+ def test_should_return_default_config_when_both_are_non_standard(self):
+ config = ProviderConfigManager._get_azure_config(
+ model="my-deployment-id", base_model=None
+ )
+ assert type(config).__name__ == "AzureOpenAIConfig"
+
+
+# ---------------------------------------------------------------------------
+# get_provider_chat_config — threads base_model through for Azure
+# ---------------------------------------------------------------------------
+class TestGetProviderChatConfigWithBaseModel:
+ """get_provider_chat_config should pass base_model to Azure config selection."""
+
+ def test_should_return_gpt5_config_for_custom_deployment_with_base_model(self):
+ from litellm.types.utils import LlmProviders
+
+ config = ProviderConfigManager.get_provider_chat_config(
+ model="my-deployment-id",
+ provider=LlmProviders.AZURE,
+ base_model="azure/gpt-5",
+ )
+ assert isinstance(config, AzureOpenAIGPT5Config)
+
+ def test_should_return_o_series_config_for_custom_deployment_with_base_model(self):
+ from litellm.types.utils import LlmProviders
+
+ config = ProviderConfigManager.get_provider_chat_config(
+ model="my-other-deployment",
+ provider=LlmProviders.AZURE,
+ base_model="azure/o3-mini",
+ )
+ assert isinstance(config, AzureOpenAIO1Config)
+
+
+# ---------------------------------------------------------------------------
+# get_supported_openai_params — base_model drives Azure param detection
+# ---------------------------------------------------------------------------
+class TestGetSupportedOpenAIParamsWithBaseModel:
+ """get_supported_openai_params should use base_model for Azure detection."""
+
+ def test_should_return_gpt5_params_for_custom_deployment_with_gpt5_base_model(
+ self,
+ ):
+ params = litellm.get_supported_openai_params(
+ model="my-deployment-id",
+ custom_llm_provider="azure",
+ base_model="azure/gpt-5",
+ )
+ assert params is not None
+ assert "reasoning_effort" in params
+ # gpt-5 maps max_tokens -> max_completion_tokens, verifying we got GPT-5 config
+ assert "max_completion_tokens" in params
+
+ def test_should_return_o_series_params_for_custom_deployment_with_o_series_base_model(
+ self,
+ ):
+ params = litellm.get_supported_openai_params(
+ model="my-other-deployment",
+ custom_llm_provider="azure",
+ base_model="azure/o4-mini",
+ )
+ assert params is not None
+ assert "reasoning_effort" in params
+
+ def test_should_return_regular_params_when_no_base_model(self):
+ """When base_model is not set and model is non-standard, default Azure config."""
+ params = litellm.get_supported_openai_params(
+ model="my-deployment-id",
+ custom_llm_provider="azure",
+ )
+ assert params is not None
+ # Default Azure config supports temperature
+ assert "temperature" in params
+
+
+# ---------------------------------------------------------------------------
+# get_optional_params — base_model drives Azure param mapping
+# ---------------------------------------------------------------------------
+class TestGetOptionalParamsWithBaseModel:
+ """get_optional_params should use base_model for Azure model-type detection."""
+
+ def test_should_map_max_tokens_for_custom_deployment_with_gpt5_base_model(self):
+ """A non-standard deployment name + gpt-5 base_model should map max_tokens -> max_completion_tokens."""
+ params = get_optional_params(
+ model="my-deployment-id",
+ custom_llm_provider="azure",
+ max_tokens=100,
+ base_model="azure/gpt-5",
+ )
+ assert params.get("max_completion_tokens") == 100
+ assert "max_tokens" not in params
+
+ def test_should_keep_max_tokens_for_custom_deployment_without_base_model(self):
+ """A non-standard deployment name without base_model should use default Azure config."""
+ params = get_optional_params(
+ model="my-deployment-id",
+ custom_llm_provider="azure",
+ max_tokens=100,
+ api_version="2024-05-01-preview",
+ )
+ # Default AzureOpenAIConfig keeps max_tokens as-is (or maps based on api_version)
+ assert "max_tokens" in params or "max_completion_tokens" in params
+
+ def test_should_support_reasoning_effort_for_custom_deployment_with_o_series_base_model(
+ self,
+ ):
+ """A non-standard deployment name + o-series base_model should accept reasoning_effort."""
+ params = get_optional_params(
+ model="my-other-deployment",
+ custom_llm_provider="azure",
+ reasoning_effort="low",
+ base_model="azure/o4-mini",
+ )
+ assert params.get("reasoning_effort") == "low"
+
+ def test_should_reject_temperature_for_custom_deployment_with_gpt5_base_model(
+ self,
+ ):
+ """A non-standard deployment + gpt-5 base_model should reject temperature."""
+ with pytest.raises(litellm.UnsupportedParamsError):
+ get_optional_params(
+ model="my-deployment-id",
+ custom_llm_provider="azure",
+ temperature=0.5,
+ base_model="azure/gpt-5",
+ )
+
+
+# ---------------------------------------------------------------------------
+# Backward compatibility — existing patterns still work
+# ---------------------------------------------------------------------------
+class TestBackwardCompatibility:
+ """Existing model-name-based and prefix-based patterns must keep working."""
+
+ def test_should_detect_gpt5_from_model_name(self):
+ config = ProviderConfigManager._get_azure_config(model="gpt-5.2")
+ assert isinstance(config, AzureOpenAIGPT5Config)
+
+ def test_should_detect_gpt5_from_gpt5_series_prefix(self):
+ config = ProviderConfigManager._get_azure_config(
+ model="gpt5_series/my-deployment"
+ )
+ assert isinstance(config, AzureOpenAIGPT5Config)
+
+ def test_should_detect_o_series_from_model_name(self):
+ config = ProviderConfigManager._get_azure_config(model="o4-mini")
+ assert isinstance(config, AzureOpenAIO1Config)
+
+ def test_should_detect_o_series_from_o_series_prefix(self):
+ config = ProviderConfigManager._get_azure_config(model="o_series/my-deployment")
+ assert isinstance(config, AzureOpenAIO1Config)
+
+ def test_should_handle_gpt5_chat_model_correctly(self):
+ """gpt-5-chat models should NOT be routed to GPT-5 config."""
+ config = ProviderConfigManager._get_azure_config(model="gpt-5-chat")
+ assert type(config).__name__ == "AzureOpenAIConfig"
+
+ def test_base_model_overrides_model_detection(self):
+ """base_model should take priority over model for type detection."""
+ # model looks like o-series, but base_model says gpt-5
+ config = ProviderConfigManager._get_azure_config(
+ model="o3-mini", base_model="azure/gpt-5.2"
+ )
+ assert isinstance(config, AzureOpenAIGPT5Config)
+
+
+# ---------------------------------------------------------------------------
+# Deep config method awareness — base_model flows into config internals
+# ---------------------------------------------------------------------------
+class TestBaseModelFlowsIntoConfigInternals:
+ """base_model should be used by config internal methods (e.g. is_model_gpt_5_2_model)."""
+
+ def test_should_support_logprobs_for_prefixed_deployment_with_gpt52_base_model(
+ self,
+ ):
+ """Deployment 'my-gpt-5.2' with base_model='azure/gpt-5.2' should support logprobs."""
+ params = litellm.get_supported_openai_params(
+ model="gpt5_series/my-gpt-5.2",
+ custom_llm_provider="azure",
+ base_model="azure/gpt-5.2",
+ )
+ assert params is not None
+ assert "logprobs" in params
+ assert "top_logprobs" in params
+
+ def test_should_support_logprobs_for_plain_deployment_with_gpt52_base_model(self):
+ """Deployment 'my-deployment-id' with base_model='azure/gpt-5.2' should support logprobs."""
+ params = litellm.get_supported_openai_params(
+ model="my-deployment-id",
+ custom_llm_provider="azure",
+ base_model="azure/gpt-5.2",
+ )
+ assert params is not None
+ assert "logprobs" in params
... diff truncated: showing 800 of 1138 linesYou can send follow-ups to the cloud agent here.
399de63 to
89e07ee
Compare
…ted_openai_params Co-authored-by: Yassin Kortam <[email protected]>
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using high effort and found 1 potential issue.
Bugbot Autofix prepared a fix for the issue found in the latest run.
- ✅ Fixed: Cache control stripping missing from compact response path
- Added the same
remove_cache_control_flag_from_input_and_toolscall intransform_compact_response_api_requestso cache_control markers are stripped from input and tools before hitting the/compactendpoint.
- Added the same
Preview (c044ec3f6d)
diff --git a/litellm/litellm_core_utils/get_supported_openai_params.py b/litellm/litellm_core_utils/get_supported_openai_params.py
--- a/litellm/litellm_core_utils/get_supported_openai_params.py
+++ b/litellm/litellm_core_utils/get_supported_openai_params.py
@@ -11,6 +11,7 @@
request_type: Literal[
"chat_completion", "embeddings", "transcription"
] = "chat_completion",
+ base_model: Optional[str] = None,
) -> Optional[list]:
"""
Returns the supported openai params for a given model + provider
@@ -20,6 +21,11 @@
get_supported_openai_params(model="anthropic.claude-3", custom_llm_provider="bedrock")
```
+ Args:
+ base_model: For Azure, the true underlying model (e.g. ``"azure/gpt-5.2"``)
+ when the deployment name differs. Used for model-type detection so that
+ non-standard deployment names route to the correct config.
+
Returns:
- List if custom_llm_provider is mapped
- None if unmapped
@@ -32,17 +38,21 @@
if custom_llm_provider in LlmProvidersSet:
provider_config = litellm.ProviderConfigManager.get_provider_chat_config(
- model=model, provider=LlmProviders(custom_llm_provider)
+ model=model,
+ provider=LlmProviders(custom_llm_provider),
+ base_model=base_model,
)
elif custom_llm_provider.split("/")[0] in LlmProvidersSet:
provider_config = litellm.ProviderConfigManager.get_provider_chat_config(
- model=model, provider=LlmProviders(custom_llm_provider.split("/")[0])
+ model=model,
+ provider=LlmProviders(custom_llm_provider.split("/")[0]),
+ base_model=base_model,
)
else:
provider_config = None
if provider_config and request_type == "chat_completion":
- return provider_config.get_supported_openai_params(model=model)
+ return provider_config.get_supported_openai_params(model=base_model or model)
if custom_llm_provider == "bedrock":
return litellm.AmazonConverseConfig().get_supported_openai_params(model=model)
@@ -130,16 +140,23 @@
model=model
)
elif custom_llm_provider == "azure":
- if litellm.AzureOpenAIO1Config().is_o_series_model(model=model):
+ _azure_detection_model = base_model or model
+ if litellm.AzureOpenAIO1Config().is_o_series_model(
+ model=_azure_detection_model
+ ):
return litellm.AzureOpenAIO1Config().get_supported_openai_params(
- model=model
+ model=_azure_detection_model
)
- elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=model):
+ elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(
+ model=_azure_detection_model
+ ):
return litellm.AzureOpenAIGPT5Config().get_supported_openai_params(
- model=model
+ model=_azure_detection_model
)
else:
- return litellm.AzureOpenAIConfig().get_supported_openai_params(model=model)
+ return litellm.AzureOpenAIConfig().get_supported_openai_params(
+ model=_azure_detection_model
+ )
elif custom_llm_provider == "openrouter":
return litellm.OpenrouterConfig().get_supported_openai_params(model=model)
elif custom_llm_provider == "vercel_ai_gateway":
diff --git a/litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py b/litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
--- a/litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
+++ b/litellm/llms/anthropic/experimental_pass_through/adapters/transformation.py
@@ -1476,7 +1476,7 @@
for choice in choices:
if choice.delta.content is not None and len(choice.delta.content) > 0:
text += choice.delta.content
- if choice.delta.tool_calls is not None:
+ if choice.delta.tool_calls:
partial_json = ""
for tool in choice.delta.tool_calls:
if (
diff --git a/litellm/llms/azure/azure.py b/litellm/llms/azure/azure.py
--- a/litellm/llms/azure/azure.py
+++ b/litellm/llms/azure/azure.py
@@ -239,7 +239,9 @@
)
data = {"model": None, "messages": messages, **optional_params}
- elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=model):
+ elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(
+ model=litellm_params.get("base_model") or model
+ ):
data = litellm.AzureOpenAIGPT5Config().transform_request(
model=model,
messages=messages,
diff --git a/litellm/llms/openai/responses/transformation.py b/litellm/llms/openai/responses/transformation.py
--- a/litellm/llms/openai/responses/transformation.py
+++ b/litellm/llms/openai/responses/transformation.py
@@ -126,9 +126,21 @@
litellm_params: GenericLiteLLMParams,
headers: dict,
) -> Dict:
- """No transform applied since inputs are in OpenAI spec already"""
+ """Strip Anthropic-only `cache_control` markers before sending to OpenAI.
+ OpenAI's Responses API rejects unknown fields on input content blocks
+ with HTTP 400 ("Unknown parameter: 'input[0].content[0].cache_control'").
+ Chat Completions strips these in
+ `remove_cache_control_flag_from_messages_and_tools`; mirror that here.
+ """
+
input = self._validate_input_param(input)
+ tools = response_api_optional_request_params.get("tools")
+ input, tools = self.remove_cache_control_flag_from_input_and_tools(
+ model=model, input=input, tools=tools
+ )
+ if tools is not None:
+ response_api_optional_request_params["tools"] = tools
final_request_params = dict(
ResponsesAPIRequestParams(
model=model, input=input, **response_api_optional_request_params
@@ -137,6 +149,38 @@
return final_request_params
+ def remove_cache_control_flag_from_input_and_tools(
+ self,
+ model: str, # allows overrides to selectively run this
+ input: Union[str, ResponseInputParam],
+ tools: Optional[List[ALL_RESPONSES_API_TOOL_PARAMS]] = None,
+ ) -> Tuple[
+ Union[str, ResponseInputParam],
+ Optional[List[ALL_RESPONSES_API_TOOL_PARAMS]],
+ ]:
+ """Sibling of `remove_cache_control_flag_from_messages_and_tools` on
+ the chat path. Strips Anthropic-only `cache_control` markers from
+ Responses API input content blocks and tools.
+
+ `filter_value_from_dict` mutates each dict in place, so the same
+ objects are returned.
+ """
+ from litellm.litellm_core_utils.prompt_templates.common_utils import (
+ filter_value_from_dict,
+ )
+
+ if isinstance(input, list):
+ for item in input:
+ if isinstance(item, dict):
+ filter_value_from_dict(cast(dict, item), "cache_control")
+
+ if tools is not None:
+ for tool in tools:
+ if isinstance(tool, dict):
+ filter_value_from_dict(cast(dict, tool), "cache_control")
+
+ return input, tools
+
def _validate_input_param(
self, input: Union[str, ResponseInputParam]
) -> Union[str, ResponseInputParam]:
@@ -604,6 +648,12 @@
url = str(parsed_url.copy_with(path=compact_path))
input = self._validate_input_param(input)
+ tools = response_api_optional_request_params.get("tools")
+ input, tools = self.remove_cache_control_flag_from_input_and_tools(
+ model=model, input=input, tools=tools
+ )
+ if tools is not None:
+ response_api_optional_request_params["tools"] = tools
data = dict(
ResponsesAPIRequestParams(
model=model, input=input, **response_api_optional_request_params
diff --git a/litellm/main.py b/litellm/main.py
--- a/litellm/main.py
+++ b/litellm/main.py
@@ -1491,7 +1491,9 @@
provider.value for provider in LlmProviders
]:
provider_config = ProviderConfigManager.get_provider_chat_config(
- model=model, provider=LlmProviders(custom_llm_provider)
+ model=model,
+ provider=LlmProviders(custom_llm_provider),
+ base_model=base_model,
)
if provider_config is not None:
@@ -1550,6 +1552,7 @@
"safety_identifier": safety_identifier,
"service_tier": service_tier,
"allowed_openai_params": kwargs.get("allowed_openai_params"),
+ "base_model": base_model,
}
optional_params = get_optional_params(
**optional_param_args, **non_default_params
@@ -1670,6 +1673,10 @@
reasoning_summary=_reasoning_summary_for_bridge,
)
+ # Use base_model (the true underlying model) for Azure model-type
+ # detection when the deployment name differs from the model name.
+ _azure_detection_model = base_model or model
+
if responses_api_model_info.get("mode") == "responses":
from litellm.completion_extras import responses_api_bridge
@@ -1713,7 +1720,9 @@
and OpenAIGPT5Config.is_model_gpt_5_model(model)
) or (
custom_llm_provider == "azure"
- and litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model)
+ and litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(
+ _azure_detection_model
+ )
):
optional_params, _ = strip_reasoning_summary_aliases_from_optional_params(
optional_params
@@ -1766,7 +1775,9 @@
if max_retries is not None:
optional_params["max_retries"] = max_retries
- if litellm.AzureOpenAIO1Config().is_o_series_model(model=model):
+ if litellm.AzureOpenAIO1Config().is_o_series_model(
+ model=_azure_detection_model
+ ):
## LOAD CONFIG - if set
config = litellm.AzureOpenAIO1Config.get_config()
for k, v in config.items():
diff --git a/litellm/utils.py b/litellm/utils.py
--- a/litellm/utils.py
+++ b/litellm/utils.py
@@ -2933,6 +2933,13 @@
except Exception:
existing_model = {}
model_cost_key = key
+ # ``get_model_info`` returns ``litellm_provider: None`` when the
+ # provider is unknown (e.g. custom deployments registered via
+ # ``Router.add_deployment``). Persisting that None into
+ # ``litellm.model_cost`` causes ``_check_provider_match`` to drop
+ # custom pricing on subsequent cost lookups.
+ if existing_model.get("litellm_provider") is None:
+ existing_model.pop("litellm_provider", None)
## override / add new keys to the existing model cost dictionary
updated_dictionary = _update_dictionary(existing_model, value)
litellm.model_cost.setdefault(model_cost_key, {}).update(updated_dictionary)
@@ -4019,16 +4026,23 @@
thinking: Optional[AnthropicThinkingParam] = None,
web_search_options: Optional[OpenAIWebSearchOptions] = None,
safety_identifier: Optional[str] = None,
+ base_model: Optional[str] = None,
**kwargs,
):
passed_params = locals().copy()
special_params = passed_params.pop("kwargs")
+ # Remove base_model from passed_params so it doesn't interfere with
+ # non_default_params / _check_valid_arg — it's a routing hint, not an
+ # OpenAI param.
+ passed_params.pop("base_model", None)
provider_config: Optional[BaseConfig] = None
if custom_llm_provider is not None and custom_llm_provider in [
provider.value for provider in LlmProviders
]:
provider_config = ProviderConfigManager.get_provider_chat_config(
- model=model, provider=LlmProviders(custom_llm_provider)
+ model=model,
+ provider=LlmProviders(custom_llm_provider),
+ base_model=base_model,
)
non_default_params = pre_process_non_default_params(
passed_params=passed_params,
@@ -4091,7 +4105,7 @@
sys.modules[__name__], "get_supported_openai_params"
)
supported_params = get_supported_openai_params(
- model=model, custom_llm_provider=custom_llm_provider
+ model=model, custom_llm_provider=custom_llm_provider, base_model=base_model
)
if supported_params is None:
supported_params = get_supported_openai_params(
@@ -4702,22 +4716,27 @@
),
)
elif custom_llm_provider == "azure":
- if litellm.AzureOpenAIO1Config().is_o_series_model(model=model):
+ _azure_detection_model = base_model or model
+ if litellm.AzureOpenAIO1Config().is_o_series_model(
+ model=_azure_detection_model
+ ):
optional_params = litellm.AzureOpenAIO1Config().map_openai_params(
non_default_params=non_default_params,
optional_params=optional_params,
- model=model,
+ model=_azure_detection_model,
drop_params=(
drop_params
if drop_params is not None and isinstance(drop_params, bool)
else False
),
)
- elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=model):
+ elif litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(
+ model=_azure_detection_model
+ ):
optional_params = litellm.AzureOpenAIGPT5Config().map_openai_params(
non_default_params=non_default_params,
optional_params=optional_params,
- model=model,
+ model=_azure_detection_model,
drop_params=(
drop_params
if drop_params is not None and isinstance(drop_params, bool)
@@ -4739,7 +4758,7 @@
optional_params = litellm.AzureOpenAIConfig().map_openai_params(
non_default_params=non_default_params,
optional_params=optional_params,
- model=model,
+ model=_azure_detection_model,
api_version=api_version, # type: ignore
drop_params=(
drop_params
@@ -5510,9 +5529,15 @@
def _check_provider_match(model_info: dict, custom_llm_provider: Optional[str]) -> bool:
"""
Check if the model info provider matches the custom provider.
+
+ A missing ``litellm_provider`` key and a ``litellm_provider`` set to
+ ``None`` both mean "no specific provider constraint" and are treated
+ as a wildcard match. ``register_model`` may persist ``None`` here via
+ ``get_model_info`` when a deployment is registered without a provider,
+ so normalising the two cases keeps custom pricing applied consistently.
"""
if custom_llm_provider and (
- "litellm_provider" in model_info
+ model_info.get("litellm_provider") is not None
and model_info["litellm_provider"] != custom_llm_provider
):
if custom_llm_provider == "vertex_ai" and model_info[
@@ -8124,10 +8149,8 @@
# Format: (factory_function, needs_model_parameter: bool)
LlmProviders.OPENAI: (lambda: litellm.OpenAIGPTConfig(), False),
LlmProviders.ANTHROPIC: (lambda: litellm.AnthropicConfig(), False),
- LlmProviders.AZURE: (
- lambda model: ProviderConfigManager._get_azure_config(model),
- True,
- ),
+ # AZURE is handled as a special case in get_provider_chat_config()
+ # so that base_model can be threaded through for model-type detection.
LlmProviders.AZURE_AI: (
lambda model: ProviderConfigManager._get_azure_ai_config(model),
True,
@@ -8267,11 +8290,19 @@
}
@staticmethod
- def _get_azure_config(model: str) -> BaseConfig:
- """Get Azure config based on model type."""
- if litellm.AzureOpenAIO1Config().is_o_series_model(model=model):
+ def _get_azure_config(model: str, base_model: Optional[str] = None) -> BaseConfig:
+ """Get Azure config based on model type.
+
+ When *base_model* is provided (e.g. ``"azure/gpt-5.2"``), it is used
+ for model-type detection instead of *model* (the deployment name).
+ This allows non-standard deployment names like ``"azure/foo"`` to be
+ routed through the correct config when the user specifies the true
+ underlying model via ``base_model``.
+ """
+ detection_model = base_model or model
+ if litellm.AzureOpenAIO1Config().is_o_series_model(model=detection_model):
return litellm.AzureOpenAIO1Config()
- if litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=model):
+ if litellm.AzureOpenAIGPT5Config.is_model_gpt_5_model(model=detection_model):
return litellm.AzureOpenAIGPT5Config()
return litellm.AzureOpenAIConfig()
@@ -8329,13 +8360,18 @@
@staticmethod
def get_provider_chat_config( # noqa: PLR0915
- model: str, provider: LlmProviders
+ model: str,
+ provider: LlmProviders,
+ base_model: Optional[str] = None,
) -> Optional[BaseConfig]:
"""
Returns the provider config for a given provider.
Uses O(1) dictionary lookup for fast provider resolution.
Python classes take priority over JSON (they have custom overrides).
+
+ For Azure, *base_model* (when set) drives model-type detection so that
+ non-standard deployment names still route to the correct config.
"""
# Handle OpenAI special cases (O-series and GPT-5 models)
if provider == LlmProviders.OPENAI:
@@ -8344,6 +8380,12 @@
if litellm.OpenAIGPT5Config.is_model_gpt_5_model(model=model):
return litellm.OpenAIGPT5Config()
+ # Handle Azure before the generic map so base_model can be threaded through
+ if provider == LlmProviders.AZURE:
+ return ProviderConfigManager._get_azure_config(
+ model=model, base_model=base_model
+ )
+
# Initialize provider config map lazily (avoids circular imports)
if ProviderConfigManager._PROVIDER_CONFIG_MAP is None:
ProviderConfigManager._PROVIDER_CONFIG_MAP = (
diff --git a/model_prices_and_context_window.json b/model_prices_and_context_window.json
--- a/model_prices_and_context_window.json
+++ b/model_prices_and_context_window.json
@@ -15006,10 +15006,16 @@
"supports_service_tier": true
},
"gemini-3.1-flash-lite": {
- "cache_read_input_token_cost": 4.5e-08,
- "cache_read_input_token_cost_per_audio_token": 9e-08,
- "input_cost_per_audio_token": 9e-07,
- "input_cost_per_token": 4.5e-07,
+ "cache_read_input_token_cost": 2.5e-08,
+ "cache_read_input_token_cost_batches": 1.25e-08,
+ "cache_read_input_token_cost_flex": 1.25e-08,
+ "cache_read_input_token_cost_per_audio_token": 5e-08,
+ "cache_read_input_token_cost_priority": 4.5e-08,
+ "input_cost_per_audio_token": 5e-07,
+ "input_cost_per_token": 2.5e-07,
+ "input_cost_per_token_batches": 1.25e-07,
+ "input_cost_per_token_flex": 1.25e-07,
+ "input_cost_per_token_priority": 4.5e-07,
"litellm_provider": "vertex_ai-language-models",
"max_audio_length_hours": 8.4,
"max_audio_per_prompt": 1,
@@ -15021,9 +15027,12 @@
"max_video_length": 1,
"max_videos_per_prompt": 10,
"mode": "chat",
- "output_cost_per_reasoning_token": 2.7e-06,
- "output_cost_per_token": 2.7e-06,
- "source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#gemini-models",
+ "output_cost_per_reasoning_token": 1.5e-06,
+ "output_cost_per_token": 1.5e-06,
+ "output_cost_per_token_batches": 7.5e-07,
+ "output_cost_per_token_flex": 7.5e-07,
+ "output_cost_per_token_priority": 2.7e-06,
+ "source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-3.1-flash-lite",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
@@ -17128,10 +17137,16 @@
"supports_service_tier": true
},
"gemini/gemini-3.1-flash-lite": {
- "cache_read_input_token_cost": 4.5e-08,
- "cache_read_input_token_cost_per_audio_token": 9e-08,
- "input_cost_per_audio_token": 9e-07,
- "input_cost_per_token": 4.5e-07,
+ "cache_read_input_token_cost": 2.5e-08,
+ "cache_read_input_token_cost_batches": 1.25e-08,
+ "cache_read_input_token_cost_flex": 1.25e-08,
+ "cache_read_input_token_cost_per_audio_token": 5e-08,
+ "cache_read_input_token_cost_priority": 4.5e-08,
+ "input_cost_per_audio_token": 5e-07,
+ "input_cost_per_token": 2.5e-07,
+ "input_cost_per_token_batches": 1.25e-07,
+ "input_cost_per_token_flex": 1.25e-07,
+ "input_cost_per_token_priority": 4.5e-07,
"litellm_provider": "gemini",
"max_audio_length_hours": 8.4,
"max_audio_per_prompt": 1,
@@ -17143,10 +17158,13 @@
"max_video_length": 1,
"max_videos_per_prompt": 10,
"mode": "chat",
- "output_cost_per_reasoning_token": 2.7e-06,
- "output_cost_per_token": 2.7e-06,
+ "output_cost_per_reasoning_token": 1.5e-06,
+ "output_cost_per_token": 1.5e-06,
+ "output_cost_per_token_batches": 7.5e-07,
+ "output_cost_per_token_flex": 7.5e-07,
+ "output_cost_per_token_priority": 2.7e-06,
"rpm": 15,
- "source": "https://ai.google.dev/gemini-api/docs/pricing",
+ "source": "https://ai.google.dev/gemini-api/docs/pricing#gemini-3.1-flash-lite",
"supported_endpoints": [
"/v1/chat/completions",
"/v1/completions",
@@ -33932,10 +33950,16 @@
"web_search_billing_unit": "per_query"
},
"vertex_ai/gemini-3.1-flash-lite": {
- "cache_read_input_token_cost": 4.5e-08,
- "cache_read_input_token_cost_per_audio_token": 9e-08,
- "input_cost_per_audio_token": 9e-07,
- "input_cost_per_token": 4.5e-07,
+ "cache_read_input_token_cost": 2.5e-08,
+ "cache_read_input_token_cost_batches": 1.25e-08,
+ "cache_read_input_token_cost_flex": 1.25e-08,
+ "cache_read_input_token_cost_per_audio_token": 5e-08,
+ "cache_read_input_token_cost_priority": 4.5e-08,
+ "input_cost_per_audio_token": 5e-07,
+ "input_cost_per_token": 2.5e-07,
+ "input_cost_per_token_batches": 1.25e-07,
+ "input_cost_per_token_flex": 1.25e-07,
+ "input_cost_per_token_priority": 4.5e-07,
"litellm_provider": "vertex_ai-language-models",
"max_audio_length_hours": 8.4,
"max_audio_per_prompt": 1,
@@ -33947,8 +33971,11 @@
"max_video_length": 1,
"max_videos_per_prompt": 10,
"mode": "chat",
- "output_cost_per_reasoning_token": 2.7e-06,
- "output_cost_per_token": 2.7e-06,
+ "output_cost_per_reasoning_token": 1.5e-06,
+ "output_cost_per_token": 1.5e-06,
+ "output_cost_per_token_batches": 7.5e-07,
+ "output_cost_per_token_flex": 7.5e-07,
+ "output_cost_per_token_priority": 2.7e-06,
"source": "https://cloud.google.com/vertex-ai/generative-ai/pricing#gemini-models",
"supported_endpoints": [
"/v1/chat/completions",
diff --git a/tests/test_litellm/llms/anthropic/experimental_pass_through/adapters/test_anthropic_experimental_pass_through_adapters_transformation.py b/tests/test_litellm/llms/anthropic/experimental_pass_through/adapters/test_anthropic_experimental_pass_through_adapters_transformation.py
--- a/tests/test_litellm/llms/anthropic/experimental_pass_through/adapters/test_anthropic_experimental_pass_through_adapters_transformation.py
+++ b/tests/test_litellm/llms/anthropic/experimental_pass_through/adapters/test_anthropic_experimental_pass_through_adapters_transformation.py
@@ -1203,6 +1203,51 @@
assert content_block_start["id"] == "toolu_bdrk_013xRVejhv3ybmLEGCoZib2b"
+def test_streaming_chunk_with_text_and_empty_tool_calls_returns_text_delta():
+ """
+ Some OpenAI-compatible providers emit `tool_calls: []` on regular text chunks.
+
+ Empty tool_calls should be treated as no tool call so the Anthropic adapter
+ does not shadow text with an empty input_json_delta.
+ """
+ choices = [
+ StreamingChoices(
+ finish_reason=None,
+ index=0,
+ delta=Delta(
+ provider_specific_fields=None,
+ content="Hello from vLLM",
+ role="assistant",
+ function_call=None,
+ tool_calls=[],
+ audio=None,
+ ),
+ logprobs=None,
+ )
+ ]
+
+ adapter = LiteLLMAnthropicMessagesAdapter()
+
+ (
+ type_of_content,
+ content_block_delta,
+ ) = adapter._translate_streaming_openai_chunk_to_anthropic(choices=choices)
+
+ assert type_of_content == "text_delta"
+ assert content_block_delta["type"] == "text_delta"
+ assert content_block_delta["text"] == "Hello from vLLM"
+
+ (
+ block_type,
+ content_block_start,
+ ) = adapter._translate_streaming_openai_chunk_to_anthropic_content_block(
+ choices=choices
+ )
+
+ assert block_type == "text"
+ assert content_block_start == {"type": "text", "text": ""}
+
+
# ============================================================================
# Cache Control Transformation Tests
# ============================================================================
diff --git a/tests/test_litellm/llms/azure/chat/test_azure_base_model_routing.py b/tests/test_litellm/llms/azure/chat/test_azure_base_model_routing.py
new file mode 100644
--- /dev/null
+++ b/tests/test_litellm/llms/azure/chat/test_azure_base_model_routing.py
@@ -1,0 +1,274 @@
+"""Tests for decoupling Azure deployment IDs from underlying model names.
+
+When users name their Azure deployment something non-standard (e.g. "my-deployment-id"),
+setting ``base_model`` should drive model-type detection (o-series, gpt-5,
+etc.) so the correct config, supported params, and param mapping are used.
+"""
+
+import pytest
+
+import litellm
+from litellm.llms.azure.chat.gpt_5_transformation import AzureOpenAIGPT5Config
+from litellm.llms.azure.chat.o_series_transformation import AzureOpenAIO1Config
+from litellm.utils import ProviderConfigManager, get_optional_params
+
+
+# ---------------------------------------------------------------------------
+# _get_azure_config — routes to the correct config based on base_model
+# ---------------------------------------------------------------------------
+class TestGetAzureConfigWithBaseModel:
+ """ProviderConfigManager._get_azure_config should use base_model for detection."""
+
+ def test_should_return_gpt5_config_when_base_model_is_gpt5(self):
+ config = ProviderConfigManager._get_azure_config(
+ model="my-deployment-id", base_model="azure/gpt-5.2"
+ )
+ assert isinstance(config, AzureOpenAIGPT5Config)
+
+ def test_should_return_o_series_config_when_base_model_is_o_series(self):
+ config = ProviderConfigManager._get_azure_config(
+ model="my-deployment-id", base_model="azure/o4-mini"
+ )
+ assert isinstance(config, AzureOpenAIO1Config)
+
+ def test_should_return_default_config_when_base_model_is_regular(self):
+ config = ProviderConfigManager._get_azure_config(
+ model="my-deployment-id", base_model="azure/gpt-4o"
+ )
+ assert type(config).__name__ == "AzureOpenAIConfig"
+
+ def test_should_fallback_to_model_when_base_model_is_none(self):
+ config = ProviderConfigManager._get_azure_config(
+ model="gpt-5.2", base_model=None
+ )
+ assert isinstance(config, AzureOpenAIGPT5Config)
+
+ def test_should_return_default_config_when_both_are_non_standard(self):
+ config = ProviderConfigManager._get_azure_config(
+ model="my-deployment-id", base_model=None
+ )
+ assert type(config).__name__ == "AzureOpenAIConfig"
+
+
+# ---------------------------------------------------------------------------
+# get_provider_chat_config — threads base_model through for Azure
+# ---------------------------------------------------------------------------
+class TestGetProviderChatConfigWithBaseModel:
+ """get_provider_chat_config should pass base_model to Azure config selection."""
+
+ def test_should_return_gpt5_config_for_custom_deployment_with_base_model(self):
+ from litellm.types.utils import LlmProviders
+
+ config = ProviderConfigManager.get_provider_chat_config(
+ model="my-deployment-id",
+ provider=LlmProviders.AZURE,
+ base_model="azure/gpt-5",
+ )
+ assert isinstance(config, AzureOpenAIGPT5Config)
+
+ def test_should_return_o_series_config_for_custom_deployment_with_base_model(self):
+ from litellm.types.utils import LlmProviders
+
+ config = ProviderConfigManager.get_provider_chat_config(
+ model="my-other-deployment",
+ provider=LlmProviders.AZURE,
+ base_model="azure/o3-mini",
+ )
+ assert isinstance(config, AzureOpenAIO1Config)
+
+
+# ---------------------------------------------------------------------------
+# get_supported_openai_params — base_model drives Azure param detection
+# ---------------------------------------------------------------------------
+class TestGetSupportedOpenAIParamsWithBaseModel:
+ """get_supported_openai_params should use base_model for Azure detection."""
+
+ def test_should_return_gpt5_params_for_custom_deployment_with_gpt5_base_model(
+ self,
+ ):
+ params = litellm.get_supported_openai_params(
+ model="my-deployment-id",
+ custom_llm_provider="azure",
+ base_model="azure/gpt-5",
+ )
+ assert params is not None
+ assert "reasoning_effort" in params
+ # gpt-5 maps max_tokens -> max_completion_tokens, verifying we got GPT-5 config
+ assert "max_completion_tokens" in params
+
+ def test_should_return_o_series_params_for_custom_deployment_with_o_series_base_model(
+ self,
+ ):
+ params = litellm.get_supported_openai_params(
+ model="my-other-deployment",
+ custom_llm_provider="azure",
+ base_model="azure/o4-mini",
+ )
+ assert params is not None
+ assert "reasoning_effort" in params
+
+ def test_should_return_regular_params_when_no_base_model(self):
+ """When base_model is not set and model is non-standard, default Azure config."""
+ params = litellm.get_supported_openai_params(
+ model="my-deployment-id",
+ custom_llm_provider="azure",
+ )
+ assert params is not None
+ # Default Azure config supports temperature
+ assert "temperature" in params
+
+
+# ---------------------------------------------------------------------------
+# get_optional_params — base_model drives Azure param mapping
+# ---------------------------------------------------------------------------
+class TestGetOptionalParamsWithBaseModel:
+ """get_optional_params should use base_model for Azure model-type detection."""
+
+ def test_should_map_max_tokens_for_custom_deployment_with_gpt5_base_model(self):
+ """A non-standard deployment name + gpt-5 base_model should map max_tokens -> max_completion_tokens."""
+ params = get_optional_params(
+ model="my-deployment-id",
+ custom_llm_provider="azure",
+ max_tokens=100,
+ base_model="azure/gpt-5",
+ )
+ assert params.get("max_completion_tokens") == 100
+ assert "max_tokens" not in params
+
+ def test_should_keep_max_tokens_for_custom_deployment_without_base_model(self):
+ """A non-standard deployment name without base_model should use default Azure config."""
+ params = get_optional_params(
+ model="my-deployment-id",
+ custom_llm_provider="azure",
+ max_tokens=100,
+ api_version="2024-05-01-preview",
+ )
+ # Default AzureOpenAIConfig keeps max_tokens as-is (or maps based on api_version)
+ assert "max_tokens" in params or "max_completion_tokens" in params
+
+ def test_should_support_reasoning_effort_for_custom_deployment_with_o_series_base_model(
+ self,
+ ):
+ """A non-standard deployment name + o-series base_model should accept reasoning_effort."""
+ params = get_optional_params(
+ model="my-other-deployment",
+ custom_llm_provider="azure",
+ reasoning_effort="low",
+ base_model="azure/o4-mini",
+ )
+ assert params.get("reasoning_effort") == "low"
+
+ def test_should_reject_temperature_for_custom_deployment_with_gpt5_base_model(
+ self,
+ ):
+ """A non-standard deployment + gpt-5 base_model should reject temperature."""
+ with pytest.raises(litellm.UnsupportedParamsError):
+ get_optional_params(
+ model="my-deployment-id",
+ custom_llm_provider="azure",
+ temperature=0.5,
+ base_model="azure/gpt-5",
+ )
+
+
+# ---------------------------------------------------------------------------
+# Backward compatibility — existing patterns still work
+# ---------------------------------------------------------------------------
+class TestBackwardCompatibility:
+ """Existing model-name-based and prefix-based patterns must keep working."""
+
+ def test_should_detect_gpt5_from_model_name(self):
+ config = ProviderConfigManager._get_azure_config(model="gpt-5.2")
+ assert isinstance(config, AzureOpenAIGPT5Config)
+
+ def test_should_detect_gpt5_from_gpt5_series_prefix(self):
+ config = ProviderConfigManager._get_azure_config(
+ model="gpt5_series/my-deployment"
+ )
+ assert isinstance(config, AzureOpenAIGPT5Config)
+
+ def test_should_detect_o_series_from_model_name(self):
+ config = ProviderConfigManager._get_azure_config(model="o4-mini")
+ assert isinstance(config, AzureOpenAIO1Config)
+
+ def test_should_detect_o_series_from_o_series_prefix(self):
+ config = ProviderConfigManager._get_azure_config(model="o_series/my-deployment")
+ assert isinstance(config, AzureOpenAIO1Config)
+
+ def test_should_handle_gpt5_chat_model_correctly(self):
+ """gpt-5-chat models should NOT be routed to GPT-5 config."""
+ config = ProviderConfigManager._get_azure_config(model="gpt-5-chat")
+ assert type(config).__name__ == "AzureOpenAIConfig"
+
+ def test_base_model_overrides_model_detection(self):
+ """base_model should take priority over model for type detection."""
... diff truncated: showing 800 of 1173 linesYou can send follow-ups to the cloud agent here.
Reviewed by Cursor Bugbot for commit acba553. Configure here.
Co-authored-by: Yassin Kortam <[email protected]>
* fix(anthropic): handle empty streaming tool calls (BerriAI#28549) Co-authored-by: shin-berri <[email protected]> Co-authored-by: yuneng-jiang <[email protected]> * [Feature][Bug Fix] Decouple Azure OpenAI Deployment ID from model name via base_model to fix gpt5 model routing (BerriAI#28490) * feat(azure): decouple deployment ID from model name via base_model Azure OpenAI deployments have arbitrary names (deployment IDs) that may not match the underlying model. Previously, model-type detection (o-series, gpt-5, etc.) relied on substring matching against the deployment name, causing misrouted configs and rejected params when deployment names were non-standard (e.g. 'my-deployment-id' for gpt-5.2). This change extends the existing base_model field to drive model-type detection, config selection, supported param resolution, and param mapping throughout the Azure call path: - _get_azure_config() uses base_model for is_o_series/is_gpt_5 checks - get_provider_chat_config() threads base_model for Azure - get_supported_openai_params() accepts and uses base_model - get_optional_params() accepts base_model and passes it to all Azure config method calls (get_supported_openai_params, map_openai_params) - azure.py completion handler uses base_model for GPT-5 detection - Config internal methods (e.g. is_model_gpt_5_2_model) now receive base_model so features like logprobs are correctly enabled Fully backward compatible - when base_model is unset, behavior is identical. Existing o_series/ and gpt5_series/ prefix workarounds continue to work. Usage in proxy config: model_list: - model_name: my-gpt5 litellm_params: model: azure/my-deployment-id model_info: base_model: azure/gpt-5.2 Fixes: non-standard deployment names like 'prefix-gpt-5.2' rejecting logprobs/top_logprobs despite the underlying model supporting them. * Addressing Greptile comments. * gemini-3.1-flash-lite pricing (BerriAI#27933) * feat(model_prices): add gemini-3.1-flash-lite pricing with standard/batch/flex/priority tiers * fix pricing * add service tier --------- Co-authored-by: shin-berri <[email protected]> * fix(openai-responses): strip Anthropic cache_control from Responses API requests (BerriAI#28431) Squash-merged by litellm-agent from cwang-otto's PR. * Treat None litellm_provider as wildcard in _check_provider_match (BerriAI#28523) Squash-merged by litellm-agent from adityasingh2400's PR. * fix greptile * fix: use _azure_detection_model in default Azure branch of get_supported_openai_params Co-authored-by: Yassin Kortam <[email protected]> * fix(openai-responses): strip cache_control on compact endpoint as well Co-authored-by: Yassin Kortam <[email protected]> --------- Co-authored-by: Felipe Garé <[email protected]> Co-authored-by: shin-berri <[email protected]> Co-authored-by: yuneng-jiang <[email protected]> Co-authored-by: withomasmicrosoft <[email protected]> Co-authored-by: mubashir1osmani <[email protected]> Co-authored-by: cwang-otto <[email protected]> Co-authored-by: Aditya Singh <[email protected]> Co-authored-by: Cursor Agent <[email protected]> Co-authored-by: Yassin Kortam <[email protected]>

Relevant issues
Linear ticket
Pre-Submission checklist
Please complete all items before asking a LiteLLM maintainer to review your PR
tests/test_litellm/directory, Adding at least 1 test is a hard requirement - see detailsmake test-unit@greptileaiand received a Confidence Score of at least 4/5 before requesting a maintainer reviewDelays in PR merge?
If you're seeing a delay in your PR being merged, ping the LiteLLM Team on Slack (#pr-review).
CI (LiteLLM team)
Branch creation CI run
Link:
CI run for the last commit
Link:
Merge / cherry-pick CI run
Links:
Screenshots / Proof of Fix
Type
🆕 New Feature
🐛 Bug Fix
🧹 Refactoring
📖 Documentation
🚄 Infrastructure
✅ Test
Changes
Note
Medium Risk
Medium risk: changes model-type detection and parameter mapping for Azure plus request payload sanitization for OpenAI Responses, which could affect routing/validation behavior across providers; adds targeted regression tests to reduce risk.
Overview
Adds a new
base_modelhint that is threaded throughcompletion/get_optional_params/get_supported_openai_paramsandProviderConfigManagerso Azure deployments with non-standard names still route to the correct config (O-series vs GPT-5) for supported-params detection and OpenAI-param mapping.Fixes a streaming edge case in the Anthropic experimental pass-through adapter by treating
tool_calls: []as absent (avoiding empty tool deltas shadowing text), and updates Azure request transformation to usebase_modelfor GPT-5 detection.Hardens OpenAI Responses API request building by stripping Anthropic-only
cache_controlmarkers frominputcontent blocks andtoolsbefore sending, and fixes custom pricing persistence by not storinglitellm_provider: None(and treatingNonelike a wildcard match) soregister_model/Router.add_deploymentpricing isn’t dropped. Updates Gemini 3.1 Flash Lite pricing entries accordingly and adds regression tests for all of the above.Reviewed by Cursor Bugbot for commit c044ec3. Bugbot is set up for automated code reviews on this repo. Configure here.