You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Feature]: Support enabling fake streaming generally for all providers, or at least for OpenAI #30208
I have searched the existing issues and checked that my issue is not a duplicate.
The Feature
Some OpenAI compatible providers are poorly implemented and break when you try streaming. LiteLLM has internal code to handle faking streaming internally, but I couldn't find a way to enable it with just the plain OpenAI provider and an arbitrary model.
Motivation, pitch
Being able to use the LiteLLM proxy with bad AI gateways that don't implement streaming properly
What part of LiteLLM is this about?
Proxy
LiteLLM is hiring a founding backend engineer, are you interested in joining us and shipping to all our users?
Here is what Claude created in the end of plenty of back and forth to get it working and convincing it to dive deep enough into LiteLLM code. If we just add that should_fake_stream check to the base config class or to those specific config classes, this feature will be builtin and I won't need this monkeypatch.
"""fake_stream_monkeypatch.pyPer-model "fake streaming" for upstream providers whose streaming is brokenor unsupported: when a client requests stream=true, LiteLLM makes a regularnon-streaming request upstream and replays the complete response to theclient as a single SSE chunk (correct usage/cost tracking included).Verified against litellm v1.88.1. Covers the `openai` provider (use it forAzure too, via the v1 API: api_base https://<resource>.openai.azure.com/openai/v1).Usage-----1. Place this file in the same directory as config.yaml (inside the container, next to wherever --config points; the proxy resolves the module as <config_dir>/fake_stream_monkeypatch.py).2. Load it via callbacks and flag models in model_info: model_list: - model_name: my-broken-model litellm_params: model: openai/gpt-5.2 api_base: https://gateway.internal/v1 api_key: os.environ/GATEWAY_KEY model_info: supports_native_streaming: false # <- the per-model switch litellm_settings: callbacks: fake_stream_monkeypatch.callbackHow it works------------The OpenAI SDK handler (llms/openai/openai.py) consultsprovider_config.should_fake_stream() on every request and, when true,forces a non-streaming upstream call and replays it as a stream. Theprovider config class is resolved per-request through late-bound moduleattributes, which is what makes these patches effective. We overrideshould_fake_stream() to key off model_info.supports_native_streaming,which the Router registers from config.yaml into litellm's model-infolookup.get_provider_chat_config short-circuits on model NAME before reaching itsgeneric provider map, so all three resolution targets are patched: - litellm.OpenAIGPTConfig default openai-provider models - litellm.OpenAIGPT5Config gpt-5* / gpt-5.<x>* names - litellm.openaiOSeriesConfig o1*/o3*/... names (pre-built instance!)Caveats-------- model_info must be on the deployment actually being served. If store_model_in_db is enabled and the model was added via the Admin UI, set supports_native_streaming there; the YAML copy is not used. Verify with GET /model/info.- This relies on internal (if stable) litellm behavior. Re-verify on version bumps: send stream=true to a flagged model and assert the upstream request body contains no "stream" key."""fromtypingimportOptionalimportlitellmfromlitellm.integrations.custom_loggerimportCustomLoggerfromlitellm.llms.openai.chat.gpt_5_transformationimportOpenAIGPT5Configfromlitellm.llms.openai.chat.gpt_transformationimportOpenAIGPTConfigfromlitellm.llms.openai.chat.o_series_transformationimportOpenAIOSeriesConfigdef_flagged(model: Optional[str], provider: str) ->bool:
"""True if this model is marked supports_native_streaming: false."""ifnotmodel:
returnFalsetry:
info=litellm.get_model_info(model=model, custom_llm_provider=provider)
returninfo.get("supports_native_streaming") isFalseexceptException:
# unknown model -> default to real streaming, never fake-stream by accidentreturnFalseclass_FakeStreamMixin:
defshould_fake_stream(self, model, stream, custom_llm_provider=None):
if_flagged(model, custom_llm_provideror"openai"):
returnstreamisTruereturnsuper().should_fake_stream(model, stream, custom_llm_provider)
classGatewayOpenAIConfig(_FakeStreamMixin, OpenAIGPTConfig):
passclassGatewayGPT5Config(_FakeStreamMixin, OpenAIGPT5Config):
passclassGatewayOSeriesConfig(_FakeStreamMixin, OpenAIOSeriesConfig):
passlitellm.OpenAIGPTConfig=GatewayOpenAIConfiglitellm.OpenAIGPT5Config=GatewayGPT5Configlitellm.openaiOSeriesConfig=GatewayOSeriesConfig() # instance, not classclassDummyCallback(CustomLogger):
"""No-op. Exists only so `litellm_settings.callbacks` can reference this module, which triggers the monkeypatches at proxy startup."""callback=DummyCallback()
Same issue here - I am using Gemma-4B-it served by litert-lm w/ OpenAI API compat. It cannot handle streaming but I cannot disable it due to this issue.
Activity
segevfiner commentedon Jun 11, 2026
Here is what Claude created in the end of plenty of back and forth to get it working and convincing it to dive deep enough into LiteLLM code. If we just add that should_fake_stream check to the base config class or to those specific config classes, this feature will be builtin and I won't need this monkeypatch.
leedsjb commentedon Jun 12, 2026
Same issue here - I am using Gemma-4B-it served by litert-lm w/ OpenAI API compat. It cannot handle streaming but I cannot disable it due to this issue.
github-actions commentedon Sep 11, 2026
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
segevfiner commentedon Sep 11, 2026
Bump