Skip to content

fix: omit max_tokens for OpenAI-compatible requests when unset - #9123

Merged
DOsinga merged 1 commit into
aaif-goose:mainfrom
SarthakB11:fix/issue-9007
May 12, 2026
Merged

DOsinga merged 1 commit into
aaif-goose:mainfrom
SarthakB11:fix/issue-9007

Conversation

@SarthakB11

Copy link
Copy Markdown
Contributor

Summary

For local OpenAI-compatible providers (llama_swap, lmstudio) the OpenAI request builder always sets max_tokens / max_completion_tokens to whatever ModelConfig::max_output_tokens() returns. When the user-supplied model has no canonical record and GOOSE_MAX_TOKENS is unset, that value is the legacy 4096 fallback, and local servers honor it and truncate non-trivial responses mid-tool-use. The user has no way to find this without discovering GOOSE_MAX_TOKENS.

This change skips emitting the field in providers/formats/openai.rs::create_request when model_config.max_tokens is None, letting the server pick its own ceiling. Behavior is unchanged when GOOSE_MAX_TOKENS is set or the model has a canonical record (which populates max_tokens via with_canonical_limits).

Only the shared OpenAI request builder is changed. The Ollama provider also routes through this create_request (re-exported at formats/ollama.rs:24-26); for unknown models with GOOSE_MAX_TOKENS unset, apply_ollama_options will simply leave options.num_predict unset, letting the Ollama server pick its own default — same fix, applied for free. Other format builders (anthropic, databricks, snowflake, openai_responses) still call max_output_tokens() and so still send the 4096 fallback; those APIs require max_tokens, so they're intentionally untouched.

Testing

New unit test test_create_request_omits_max_tokens_when_unset asserts that an unknown model with max_tokens: None produces a payload containing neither max_tokens nor max_completion_tokens. Existing tests still pass.

cargo test -p goose --lib --no-default-features --features rustls-tls \
  providers::formats::openai providers::ollama

Related Issues

Fixes #9007

@DOsinga DOsinga left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clean, well-scoped fix that solves a real pain point for local provider users. The conditional omission of max_tokens when unset is the right approach — lets the server decide its own limits rather than imposing an arbitrary 4096 cap. Test follows existing patterns. LGTM.

@DOsinga
DOsinga added this pull request to the merge queue May 12, 2026
Merged via the queue into aaif-goose:main with commit b23b95c May 12, 2026
21 checks passed
@monroewilliams

Copy link
Copy Markdown
Contributor

Thanks for fixing this! :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Default max_output_tokens of 4096 is unreasonably low

3 participants