Repository navigation
fix(core): leave headroom when fitting output to the context window - #59934
Conversation
There was a problem hiding this comment.
The fix looks good. Fitting a normal turn's maxTokens to 99% of the window means a small difference between our count and the provider's no longer pushes the request past the window.
I checked it by running the updated runner test against the base source, where it fails, and with this PR, where it passes. I also called outputLimit directly with the numbers from #58463. For Kimi K2.6 with a ~27k prompt, the output limit goes from 234,994 to 232,372. For the DeepSeek case it goes from 249,555 (the value that overshot by 6) to 239,069. The runner, compaction, generate and model-request tests pass (282), and tsc -b passes.
Public API: unchanged. outputLimit gets an optional headroom parameter that defaults to 0, and it's only used inside core.
One non-blocking note: compaction requests still ask for exactly the room left (in fitOutput), so a summary request near the window can still be rejected by a strict provider for being a few tokens over. Leaving that out of this PR is reasonable.
Related: #50574
Issue for this PR
Closes #58463
Type of change
What does this PR do?
Once the prompt is large enough,
outputLimitlowersmax_tokensto exactly the room left in the window:context - measured - estimated * 1.15. Our count runs a little below the provider's (chat template tokens, tool schemas), so the request can overshoot the window by a few tokens, and strict providers reject it. The issue thread has real examples: Kimi K2.6 through OpenRouter ("maximum context length is 262144 tokens. However, you requested about 262223") and DeepSeek (6 tokens over).This is easy to hit on models whose output limit is close to their window. Kimi K2.6 is 262k context / 236k output, so the output limit starts being fitted from about a 26k prompt, and every request after that sits right at the edge.
The fix leaves 1% of the window unused when a normal turn's output limit is fitted (2.6k on Kimi, about 10k on a 1M model). Requests that aren't up against the window are unchanged.
Compaction requests are left as they are. The summary is sized against its own reserve, and the existing tests pin that it gets the whole reserve, so I kept this change to normal turns.
The issue also says the compaction ceiling is above the request ceiling on 1M models.
max_tokensalready shrinks pastcontext - 256k, so that alone doesn't fail a request. The real failure is the missing headroom, which is what this changes.How did you verify your code works?
fits the output limit to the prompt sizeinsession-runner.test.tsso it checks for the 1% gap through the real runner path.bun test test/session-runner.test.ts test/session-compaction.test.ts test/session-generate.test.ts test/session-model-request.test.tsfrompackages/core: 282 pass.bun typecheckinpackages/corepasses.Visual
Checklist
Added by the opencode bot (AI-generated).
Related: #50574