Skip to content

[Feature]: Support enabling fake streaming generally for all providers, or at least for OpenAI #30208

Open

Description

@segevfiner

Check for existing issues

  • I have searched the existing issues and checked that my issue is not a duplicate.

The Feature

Some OpenAI compatible providers are poorly implemented and break when you try streaming. LiteLLM has internal code to handle faking streaming internally, but I couldn't find a way to enable it with just the plain OpenAI provider and an arbitrary model.

Motivation, pitch

Being able to use the LiteLLM proxy with bad AI gateways that don't implement streaming properly

What part of LiteLLM is this about?

Proxy

LiteLLM is hiring a founding backend engineer, are you interested in joining us and shipping to all our users?

No

Twitter / LinkedIn details

No response

Activity

  1. segevfiner commented on Jun 11, 2026

    @segevfiner
    Author

    Here is what Claude created in the end of plenty of back and forth to get it working and convincing it to dive deep enough into LiteLLM code. If we just add that should_fake_stream check to the base config class or to those specific config classes, this feature will be builtin and I won't need this monkeypatch.

    """
    fake_stream_monkeypatch.py
    
    Per-model "fake streaming" for upstream providers whose streaming is broken
    or unsupported: when a client requests stream=true, LiteLLM makes a regular
    non-streaming request upstream and replays the complete response to the
    client as a single SSE chunk (correct usage/cost tracking included).
    
    Verified against litellm v1.88.1. Covers the `openai` provider (use it for
    Azure too, via the v1 API: api_base https://<resource>.openai.azure.com/openai/v1).
    
    Usage
    -----
    1. Place this file in the same directory as config.yaml (inside the
       container, next to wherever --config points; the proxy resolves the
       module as <config_dir>/fake_stream_monkeypatch.py).
    
    2. Load it via callbacks and flag models in model_info:
    
        model_list:
          - model_name: my-broken-model
            litellm_params:
              model: openai/gpt-5.2
              api_base: https://gateway.internal/v1
              api_key: os.environ/GATEWAY_KEY
            model_info:
              supports_native_streaming: false  # <- the per-model switch
    
        litellm_settings:
          callbacks: fake_stream_monkeypatch.callback
    
    How it works
    ------------
    The OpenAI SDK handler (llms/openai/openai.py) consults
    provider_config.should_fake_stream() on every request and, when true,
    forces a non-streaming upstream call and replays it as a stream. The
    provider config class is resolved per-request through late-bound module
    attributes, which is what makes these patches effective. We override
    should_fake_stream() to key off model_info.supports_native_streaming,
    which the Router registers from config.yaml into litellm's model-info
    lookup.
    
    get_provider_chat_config short-circuits on model NAME before reaching its
    generic provider map, so all three resolution targets are patched:
      - litellm.OpenAIGPTConfig      default openai-provider models
      - litellm.OpenAIGPT5Config     gpt-5* / gpt-5.<x>* names
      - litellm.openaiOSeriesConfig  o1*/o3*/... names (pre-built instance!)
    
    Caveats
    -------
    - model_info must be on the deployment actually being served. If
      store_model_in_db is enabled and the model was added via the Admin UI,
      set supports_native_streaming there; the YAML copy is not used.
      Verify with GET /model/info.
    - This relies on internal (if stable) litellm behavior. Re-verify on
      version bumps: send stream=true to a flagged model and assert the
      upstream request body contains no "stream" key.
    """
    
    from typing import Optional
    
    import litellm
    from litellm.integrations.custom_logger import CustomLogger
    from litellm.llms.openai.chat.gpt_5_transformation import OpenAIGPT5Config
    from litellm.llms.openai.chat.gpt_transformation import OpenAIGPTConfig
    from litellm.llms.openai.chat.o_series_transformation import OpenAIOSeriesConfig
    
    
    def _flagged(model: Optional[str], provider: str) -> bool:
        """True if this model is marked supports_native_streaming: false."""
        if not model:
            return False
        try:
            info = litellm.get_model_info(model=model, custom_llm_provider=provider)
            return info.get("supports_native_streaming") is False
        except Exception:
            # unknown model -> default to real streaming, never fake-stream by accident
            return False
    
    
    class _FakeStreamMixin:
        def should_fake_stream(self, model, stream, custom_llm_provider=None):
            if _flagged(model, custom_llm_provider or "openai"):
                return stream is True
            return super().should_fake_stream(model, stream, custom_llm_provider)
    
    
    class GatewayOpenAIConfig(_FakeStreamMixin, OpenAIGPTConfig):
        pass
    
    
    class GatewayGPT5Config(_FakeStreamMixin, OpenAIGPT5Config):
        pass
    
    
    class GatewayOSeriesConfig(_FakeStreamMixin, OpenAIOSeriesConfig):
        pass
    
    
    litellm.OpenAIGPTConfig = GatewayOpenAIConfig
    litellm.OpenAIGPT5Config = GatewayGPT5Config
    litellm.openaiOSeriesConfig = GatewayOSeriesConfig()  # instance, not class
    
    
    class DummyCallback(CustomLogger):
        """No-op. Exists only so `litellm_settings.callbacks` can reference this
        module, which triggers the monkeypatches at proxy startup."""
    
    
    callback = DummyCallback()
  2. leedsjb commented on Jun 12, 2026

    @leedsjb

    Same issue here - I am using Gemma-4B-it served by litert-lm w/ OpenAI API compat. It cannot handle streaming but I cannot disable it due to this issue.

  3. github-actions commented on Sep 11, 2026

    @github-actions
    Contributor

    This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

  4. segevfiner commented on Sep 11, 2026

    @segevfiner
    Author

    Bump

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions