Repository navigation
[Feature]: Force enable fake streaming per model #5416
Description
The Feature
I would like to disable streaming on a per-model basis in my proxy yaml config, and have LiteLLM do a non-streaming request upstream (while returning a fake-streaming response to the client).
Basically what was mentioned here: #61 (comment)
Motivation, pitch
When streaming on OpenAI, their moderation filter makes the responses insanely slow. Like, I sometimes timeout after >10 minutes.
I don't want to change my client-side code, as I do want other providers (like Azure OpenAI) in the same model group to stream (as the perf impact isn't as bad).
Twitter / LinkedIn details
Reacted by Jonne
Activity
mikewirth commentedon Jan 23, 2025
This would be useful for me as well, as I'm trying to use https://mockgpt.wiremock.io/ through LiteLLM.
MockGPT doesn't support streaming, but I want to use it when load testing to replace GPT-4o. I can't control if my client application sends stream=True to LiteLLM.
tan-yong-sheng commentedon Feb 2, 2025
Facing similar problem when I try to use GitHub/o1-mini on openwebui... I hope I could disable the stream per model basis.
github-actions commentedon May 4, 2025
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.
feat(router): add force_non_streaming per-deployment config option
feat(router): add force_non_streaming per-deployment config option