Repository navigation
Add timeout and max_retries to the OpenAI provider - #593
inamdarmihir wants to merge 3 commits into
Conversation
|
Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA). View this failed invocation of the CLA check for more information. For the most up to date status, view the checks section at the bottom of the pull request. |
d14eaad to
ebc5917
Compare
Both were accepted through language_model_params but never reached the OpenAI client: leftover kwargs are filtered to an allowlist of request parameters, so users hitting client timeouts on large inputs had no way to raise the limit. Accept them as constructor arguments and pass them to openai.OpenAI only when set, leaving the SDK defaults untouched otherwise. Related to google#82
ebc5917 to
445744c
Compare
|
Your branch is 1 commits behind git fetch origin main
git merge origin/main
git pushNote: Enable "Allow edits by maintainers" to allow automatic updates. |
|
Your branch is 8 commits behind git fetch origin main
git merge origin/main
git pushNote: Enable "Allow edits by maintainers" to allow automatic updates. |
|
Your branch is 1 commits behind git fetch origin main
git merge origin/main
git pushNote: Enable "Allow edits by maintainers" to allow automatic updates. |
Description
#82 asks for a way to set the LLM call timeout when large inputs time out. Gemini exposes
http_options/retry settings and Ollama has atimeoutargument, butOpenAILanguageModelhas no equivalent:timeoutandmax_retriespassed vialanguage_model_paramsland in_extra_kwargs, are dropped by the request-parameter allowlist ininfer(), and never reachopenai.OpenAI(...).This adds
timeoutandmax_retriesas keyword arguments onOpenAILanguageModel.__init__and forwards them to theopenai.OpenAIclient only when set, so SDK defaults are unchanged otherwise. They are placed aftermax_workersand before**kwargs, so existing positional calls are unaffected. This also applies to OpenAI-compatible endpoints configured withbase_url.Usage:
Verified through
factory.create_modelwith the real SDK:timeout=7.5, max_retries=0yields a client with those values; without them the client keeps the SDK defaults (Timeout(connect=5.0, read=600, ...),max_retries=2).Overlap note: #548 also touches the
openai.OpenAI(...)construction (addsdefault_headers); the two should merge with a trivial rebase.Fixes #82
Feature
How Has This Been Tested?
Added tests that the client receives
timeout/max_retrieswhen set and is constructed exactly as before when they are not.tests/create_provider_plugin_test.pyhas 3 failures onmainin my environment (offline editable install / build isolation) unrelated to this change.Checklist:
pylintover the affected code.