Skip to content

fix(ai): fit thinking budgets to the output limit - #51157

Merged
rekram1-node merged 1 commit into
v2from
fit-thinking-budgets
Sep 24, 2026
Merged

rekram1-node merged 1 commit into
v2from
fit-thinking-budgets

Conversation

@rekram1-node

@rekram1-node rekram1-node commented Sep 24, 2026 •

Copy link
Copy Markdown
Collaborator

Issue for this PR

N/A. Found while testing default output limits for #51021.

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Protocols that send an explicit thinking budget now cap it at half the output limit. Thinking counts against the output limit, so a budget near or above it leaves the answer, a tool call or a summary with no room.

ProviderShared.fitThinkingBudget is the single rule:

  • a budget at or under half the limit passes through unchanged;
  • special values such as Gemini's -1 (dynamic) and 0 (off) pass through;
  • requests without an output limit pass through;
  • the result never goes below a provider's minimum.

It applies in the four protocols that read a typed budget:

Protocol Field Provider behavior
Anthropic Messages thinking.budget_tokens Rejects a budget that is not below max_tokens. Keeps Anthropic's 1,024 minimum.
Alibaba Chat thinking_budget Rejects a budget that is not below max_completion_tokens; the answer gets what remains.
Alibaba Messages thinking.budget_tokens Builds its own thinking, so it never went through the Anthropic path.
Gemini (Google, Vertex) thinkingConfig.thinkingBudget Accepts a budget above the limit, but thinking can use the whole limit. Live on Gemini 2.5 Pro with a 1,024 limit and an 8,192 budget: 1,021 thinking tokens and an empty answer (MAX_TOKENS). Uses a 512 minimum, because Gemini 2.5 rejects budgets under a per-model minimum (512 on Flash-Lite, 128 on Pro).

Main steps are mostly unaffected, because variant budgets are usually at most half the model's output limit. The cap applies to compaction summaries (32k), near-full contexts, and budgets derived from the whole output limit.

Not covered:

  • OpenRouter passes reasoning through untyped.
  • Bedrock Converse and NVIDIA receive budgets as raw body fields that are merged after the protocol runs.

How did you verify your code works?

  • Unit tests: fit tests for Anthropic (including the minimum), Alibaba Chat and Messages, and Gemini (including -1 and the 512 minimum). The full packages/ai suite passes.

  • Gemini minimums, live on AI Studio:

    Model Rejected (400) Accepted
    2.5 Flash-Lite 1, 500, 511: "choose a value between 512 and 24576" 512
    2.5 Pro 100, 127: "choose a value between 128 and 32768" 128
    2.5 Flash — 1, 500

    All three accept 512 under a 1,000 limit, which is what the fit now sends there.

  • Live with the default output limits from feat(core): fit output limits to the context window #51021 applied: a title, tool-using steps and a manual compaction.

    Model Steps (limit / budget) Compaction (limit / budget) Before
    Alibaba qwen3.7-plus max 131,072 / 65,536 32,000 / 16,000 Answer cut after 8 tokens; compaction rejected
    Google and Vertex gemini-2.5-pro max 65,536 / 32,768 32,000 / 16,000 Budget above the summary limit
    Anthropic claude-haiku-4-5 max 64,000 / 31,999 32,000 / 16,000 31,999 under 32,000 left about one token for the summary

    All requests returned 200, and every compaction completed.

Screenshots / recordings

N/A

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

@rekram1-node
rekram1-node merged commit c903774 into v2 Sep 24, 2026
10 checks passed
@rekram1-node
rekram1-node deleted the fit-thinking-budgets branch September 24, 2026 17:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant