[Bug]: Responses API streaming falls back to fake stream for custom models, silently dropping function_call events #21090

Check for existing issues

• I have searched the existing issues and checked that my issue is not a duplicate.

What happened?

When using the Responses API (/v1/responses) with stream: true for custom models not registered in model_prices_and_context_window.json (e.g., models served by vLLM, TGI, or other OpenAI-compatible backends), LiteLLM falls back to fake streaming (MockResponsesAPIStreamingIterator) instead of real streaming.

This causes all tool call streaming events to be silently dropped — the client receives only response.completed and [DONE], with no intermediate events like response.function_call_arguments.delta.

Note: The same custom model (gpt-oss-20b served by vLLM) works perfectly with the Chat Completions API (/v1/chat/completions) — streaming, tool calling, and all features function correctly. The issue is specific to the Responses API streaming path.

The call chain is:

1. supports_native_streaming() (litellm/utils.py) — raises exception for unknown models, returns False in the except block
2. OpenAIResponsesAPIConfig.should_fake_stream() — sees False, returns True
3. llm_http_handler.py — sends stream=false to the backend, wraps response in MockResponsesAPIStreamingIterator
4. MockResponsesAPIStreamingIterator — only synthesizes OutputTextDeltaEvent (5-char text chunks) + ResponseCompletedEvent, cannot produce function_call_arguments.delta or other intermediate events

The root issue is the default return False in the except block of supports_native_streaming():

def supports_native_streaming(model, custom_llm_provider):
    try:
        model_info = _get_model_info_helper(model=model, custom_llm_provider=custom_llm_provider)
        return model_info.get("supports_native_streaming", True)
    except Exception as e:
        return False  # <-- All unknown models default to False → fake streaming

Only 4 models in the DB have supports_native_streaming: false (gpt-5-pro, gpt-5-pro-2025-10-06, o1-pro, o1-pro-2025-03-19). They are properly registered and never hit the except block. All models that reach the except block are custom/unknown models that almost certainly support native streaming.

Suggested fix: Change return False to return True in the except block. This way:

• DB-registered models with supports_native_streaming: false are unaffected (they don't hit except)
• Custom models use real streaming, receiving all events correctly
• If a custom model genuinely doesn't support streaming, the backend returns an explicit error — better than silent data loss

Related issues: #20711 #13393 #20356

Steps to Reproduce

1. Set up vLLM serving a custom model (tested with gpt-oss-20b)

2. Configure LiteLLM proxy:

model_list:
  - model_name: gpt-oss-20b
    litellm_params:
      model: openai/gpt-oss-20b
      api_base: http://vllm-server:8000/v1

3. Send a streaming Responses API request with tools:

curl -X POST http://localhost:4000/v1/responses \
  -H "Authorization: Bearer sk-1234" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gpt-oss-20b",
    "input": [{"role": "user", "content": "What is the weather in Seoul?"}],
    "stream": true,
    "tools": [{
      "type": "function",
      "name": "get_weather",
      "description": "Get weather",
      "parameters": {
        "type": "object",
        "properties": {"location": {"type": "string"}},
        "required": ["location"]
      }
    }]
  }'

4. Expected: ~30+ SSE events including response.output_item.added, response.function_call_arguments.delta, etc.

5. Actual: Only response.completed + [DONE] (2 events). All intermediate events are missing.

6. Same request sent directly to vLLM (http://vllm-server:8000/v1/responses) returns all ~59 events correctly. Chat Completions API (/v1/chat/completions) with the same model also works fine through the proxy — this issue is specific to the Responses API path.

Relevant log output

Model not found or error in checking supports_native_streaming support. You passed model=openai/gpt-oss-20b, custom_llm_provider=openai. Error: ...

What part of LiteLLM is this about?

Proxy

What LiteLLM version are you on ?

v1.81.6 (commit d9c69ae)

Twitter / LinkedIn details

No response