Repository navigation
fix: omit max_tokens for OpenAI-compatible requests when unset - #9123
Merged
Merged
Conversation
DOsinga
approved these changes
May 12, 2026
DOsinga
left a comment
Collaborator
There was a problem hiding this comment.
Clean, well-scoped fix that solves a real pain point for local provider users. The conditional omission of max_tokens when unset is the right approach — lets the server decide its own limits rather than imposing an arbitrary 4096 cap. Test follows existing patterns. LGTM.
Contributor
|
Thanks for fixing this! :) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
For local OpenAI-compatible providers (
llama_swap,lmstudio) the OpenAI request builder always setsmax_tokens/max_completion_tokensto whateverModelConfig::max_output_tokens()returns. When the user-supplied model has no canonical record andGOOSE_MAX_TOKENSis unset, that value is the legacy 4096 fallback, and local servers honor it and truncate non-trivial responses mid-tool-use. The user has no way to find this without discoveringGOOSE_MAX_TOKENS.This change skips emitting the field in
providers/formats/openai.rs::create_requestwhenmodel_config.max_tokensisNone, letting the server pick its own ceiling. Behavior is unchanged whenGOOSE_MAX_TOKENSis set or the model has a canonical record (which populatesmax_tokensviawith_canonical_limits).Only the shared OpenAI request builder is changed. The Ollama provider also routes through this
create_request(re-exported atformats/ollama.rs:24-26); for unknown models withGOOSE_MAX_TOKENSunset,apply_ollama_optionswill simply leaveoptions.num_predictunset, letting the Ollama server pick its own default — same fix, applied for free. Other format builders (anthropic, databricks, snowflake, openai_responses) still callmax_output_tokens()and so still send the 4096 fallback; those APIs requiremax_tokens, so they're intentionally untouched.Testing
New unit test
test_create_request_omits_max_tokens_when_unsetasserts that an unknown model withmax_tokens: Noneproduces a payload containing neithermax_tokensnormax_completion_tokens. Existing tests still pass.Related Issues
Fixes #9007