Skip to content

fix(core): leave headroom when fitting output to the context window - #59934

Merged
neriousy merged 1 commit into
devfrom
output-headroom
Oct 28, 2026
Merged

neriousy merged 1 commit into
devfrom
output-headroom

Conversation

@neriousy

@neriousy neriousy commented Oct 28, 2026 •

Copy link
Copy Markdown
Member

Issue for this PR

Closes #58463

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Once the prompt is large enough, outputLimit lowers max_tokens to exactly the room left in the window: context - measured - estimated * 1.15. Our count runs a little below the provider's (chat template tokens, tool schemas), so the request can overshoot the window by a few tokens, and strict providers reject it. The issue thread has real examples: Kimi K2.6 through OpenRouter ("maximum context length is 262144 tokens. However, you requested about 262223") and DeepSeek (6 tokens over).

This is easy to hit on models whose output limit is close to their window. Kimi K2.6 is 262k context / 236k output, so the output limit starts being fitted from about a 26k prompt, and every request after that sits right at the edge.

The fix leaves 1% of the window unused when a normal turn's output limit is fitted (2.6k on Kimi, about 10k on a 1M model). Requests that aren't up against the window are unchanged.

Compaction requests are left as they are. The summary is sized against its own reserve, and the existing tests pin that it gets the whole reserve, so I kept this change to normal turns.

The issue also says the compaction ceiling is above the request ceiling on 1M models. max_tokens already shrinks past context - 256k, so that alone doesn't fail a request. The real failure is the missing headroom, which is what this changes.

How did you verify your code works?

  • Updated fits the output limit to the prompt size in session-runner.test.ts so it checks for the 1% gap through the real runner path.
  • bun test test/session-runner.test.ts test/session-compaction.test.ts test/session-generate.test.ts test/session-model-request.test.ts from packages/core: 282 pass.
  • bun typecheck in packages/core passes.

Visual

<svg xmlns="http://www.w3.org/2000/svg" width="240" height="72" viewBox="0 0 240 72" fill="none">
  <style>
    .p{animation:p 4s ease-in-out infinite}
    .o{animation:o 4s ease-in-out infinite}
    @keyframes p{0%,15%{width:96px}55%,85%{width:150px}100%{width:96px}}
    @keyframes o{0%,15%{x:116px;width:96px}55%,85%{x:170px;width:42px}100%{x:116px;width:96px}}
    @media (prefers-reduced-motion:reduce){.p,.o{animation:none}}
  </style>
  <rect x="20" y="24" width="200" height="24" rx="5" stroke="currentColor" stroke-opacity=".35"/>
  <rect class="p" x="20" y="24" width="150" height="24" rx="5" fill="currentColor" fill-opacity=".55"/>
  <rect class="o" x="170" y="24" width="42" height="24" rx="3" fill="currentColor" fill-opacity=".22"/>
  <line x1="212" y1="18" x2="212" y2="54" stroke="currentColor" stroke-opacity=".6" stroke-dasharray="2 2"/>
  <text x="20" y="64" font-family="system-ui,sans-serif" font-size="9" fill="currentColor" fill-opacity=".6">prompt</text>
  <text x="200" y="14" font-family="system-ui,sans-serif" font-size="9" fill="currentColor" fill-opacity=".6">headroom</text>
</svg>

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

Added by the opencode bot (AI-generated).

Related: #50574

@opencodejr opencodejr Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The fix looks good. Fitting a normal turn's maxTokens to 99% of the window means a small difference between our count and the provider's no longer pushes the request past the window.

I checked it by running the updated runner test against the base source, where it fails, and with this PR, where it passes. I also called outputLimit directly with the numbers from #58463. For Kimi K2.6 with a ~27k prompt, the output limit goes from 234,994 to 232,372. For the DeepSeek case it goes from 249,555 (the value that overshot by 6) to 239,069. The runner, compaction, generate and model-request tests pass (282), and tsc -b passes.

Public API: unchanged. outputLimit gets an optional headroom parameter that defaults to 0, and it's only used inside core.

One non-blocking note: compaction requests still ask for exactly the room left (in fitOutput), so a summary request near the window can still be rejected by a strict provider for being a few tokens over. Leaving that out of this PR is reasonable.

Related: #50574

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant