Repository navigation
Conversation
![opencodejr[bot]](https://avatars.githubusercontent.com/in/5497873?s=60&v=4){kind=small}
There was a problem hiding this comment.
The headroom in outputLimit and fitOutput fixes the exact-fill rejections from #58463. I called outputLimit from both checkouts with the issue's DeepSeek 1M, Kimi 262k and Claude 200k numbers. On base, those requests leave 300–1,170 tokens free in the window by our own estimate, which a small provider undercount overruns. With this PR they leave 4.5k–12k. The three changed session test files pass (281 tests), and packages/core type-checks.
Public API: no schema, SDK, HTTP or CLI changes. Only the default of compaction.buffer changes: 15% of the window (at least 16k) becomes 16k plus min(10k, 5% of the window). Windows under 32k keep 15%. Existing configs still work as before.
One concern, inline: tokenRatio pairs measured responses across agent and variant changes, which measuredPrompt deliberately excludes.
Related: #50574
| * characters. chars/4 runs low on dense code and JSON for some tokenizers, about 1.6× for Claude. Never below 1, so | ||
| * an overcount only compacts a little early, and 1 until two responses since the last compaction have been measured. | ||
| */ | ||
| const tokenRatio = (context: SessionContext.Loaded) => { |
There was a problem hiding this comment.
tokenRatio only checks the model id, but measuredPrompt also requires the same variant and agent because "they shape the request". When the agent changes between two measured responses (e.g. plan → build), the difference in system prompt and tool definitions ends up in counted, while estimated only covers the appended messages. Early in a session, a single pair like that, with a tail of about 1k estimated tokens, can push the ratio to the 3× cap. That ratio then triples the estimate for the next large tool result, which can start compaction early and cut max_tokens well below what fits. A negative delta in the other direction pulls the ratio down. Could the ratio only use consecutive pairs with the same agent and variant (and model) as the current request?
Issue for this PR
Closes #58463
Supersedes #59934, which only covered normal turns.
Type of change
What does this PR do?
This reworks how the context window is budgeted. I replayed about 41k real steps from local sessions through a port of the current rules. Four things go wrong today:
W − output,max_tokensis the exact room left, so strict providers (Anthropic, OpenRouter) reject the request whenever the estimate runs low. Kimi K2.6 via OpenRouter hits this from a 26k prompt on. Claude on a 200k window hits it at about 136k, so in practice compaction is forced by a rejection before it ever reaches its trigger.fitOutput's floor branch uses the room left with no margin. Fixing only normal turns (fix(core): leave headroom when fitting output to the context window #59934) moves the rejection there: in the replay, 39% of Claude 200k compactions then fell back to text.mwith probabilityE[(G − m)⁺] / E[G]for step growthG. So 15% is wasted on 1M windows (133k between trigger and cache-hit limit) and thin on small ones.Changes:
outputLimitnever claims the last 1% of the window (at least 4k, at most 5%), and keeps a 25% margin on the unmeasured tail. This covers normal turns, the summary request's starting limit, and both branches offitOutput.estimatePromptscales the unmeasured tail by a ratio learned from the history: between consecutive measured responses, how many tokens the provider counted for what chars/4 estimated. It is clamped to [1, 3], and falls back to 1 until two responses since the last compaction have been measured. No new state.deliversends the history before the kept tail unchanged up to the cache-hit limit, not just up to the trigger. When it can, it sizes that history from the measured count of the response that ends it. The margin and the kept tail now add up instead of covering the same overshoots.W − 16k − min(10k, 5% W)instead ofW − max(15% W, 16k). That is 974k instead of 850k on 1M windows and 174k instead of 170k on 200k. Windows under 32k keep the old 15%.In the replay on strict providers (they reject prompt +
max_tokenspast the window):Trade-offs: the summary's output limit on a 200k window drops from 30k to 22k, which is still above the largest summaries. Windows between 32k and 200k compact a few thousand tokens earlier, for example 44.8k instead of 48k on 64k. Larger context per request also costs more per step, since every request rereads its prefix.
Not included: a cost-oriented trigger mode, and the 64k floor on background compaction (#58818).
Before and after
These come from a simulator that ports the budgeting rules before and after this PR and replays real step sizes from local sessions (about 41k steps). The source for the simulator and the charts is in a separate repo, and I can publish it.
Why the estimate needs calibrating. For the same tool output, Claude needs about 1.6× the tokens chars/4 predicts, Gemini about 1.26×. The 15% margin covers GPT and almost nothing else.
Where the window goes. Compaction starts later on large windows. The window never fills to the last token, because 1% (at least 4k) is never claimed by
max_tokens. Purple marks wheremax_tokensis fitted to the room left. For Kimi that already happens from a 26k prompt.One session, before and after. Before, Claude 200k never reaches its trigger: the first request whose output is fitted to the window gets rejected, at about 70%, and that rejection forces the compaction. Kimi via OpenRouter is rejected on more than half of its requests and compacts every couple of steps at about 15% of the window. After, both run to the trigger and compact with a cache hit.
Requests rejected at the window edge. This is the share of fitted requests a strict provider rejects, replaying every real step. Headroom on normal turns alone (#59934) still leaves 6% of Claude requests rejected, and moves the failure to the summary request.
How did you verify your code works?
bun run script/test.tsinpackages/core: 6901 pass. The 7 failures are shell-tool tests that also fail ondevhere (macOS bash 3.2).--rerun-each 10 --randomize: all pass.bun run checkpasses.Visual
Checklist