fix(core): start background compaction a fixed lead before the threshold - #60049

Issue for this PR

Closes #58818

Type of change

• Bug fix
• New feature
• Refactor / code improvement
• Documentation

What does this PR do?

Changes when experimental.background_compaction starts a summary and when it drops one. Three rules change.

1. Start a fixed 60k tokens before the threshold, not at 85% of it, and drop the 64k minimum. A background summary takes 30–60 s to write, and the session only waits if the conversation reaches the threshold first. How much it grows in that time doesn't depend on the window: a few thousand tokens at the median, and 25k or more about one time in twenty, when large file reads arrive together.

Today's rule is right only by coincidence:

• On a 200k window, 85% is 26k ahead, which is about right.
• On 1M, it is 128k ahead, much too early. The held summary keeps getting replaced, and many sessions end before the threshold.
• On a 128k GPT window, it is 13k ahead, too late.
• On windows whose threshold is under 64k, it never starts (Background compaction: hardcoded 64k floor makes it unreachable on smaller context windows #58818).

On small windows, the lead is capped at half the room since the cycle's first response, so a new summary still covers more than the previous one.

2. Don't start from an expired prompt cache. After a break longer than the cache lifetime, the next step started a background summary from the same cold prefix the next request was about to write again, so the whole prompt was written twice. I hit this locally: an Opus session at 823k tokens, an 85-minute break, and a background summary that wrote 825,304 tokens to the cache instead of reading them (about $4.13 instead of $0.16). It now waits for a response whose cache is still warm (the model's cache.ttl, 5 minutes by default), the way it already waits for the first response after a model switch.

3. Allow a verbatim tail of up to 40% of the budget instead of 30k. Once 30k tokens (twice keep) arrived after the held summary, it was replaced, or at the threshold rejected in favor of a foreground compaction. That happened at the same 30k on every window, and on 1M windows most held summaries were replaced before they were used. Typical tails are 10–20k, so the limit only matters for bursts of large tool inputs or pasted text. At 40%, a swapped-in context still stays well under the threshold.

To check these numbers, I replayed about 41k steps from my local sessions through a simulation of this code. It uses only token counts and timings, no content, and draws summary latencies from measured requests.

On top of the summaries per compaction above, sessions that end before the threshold leave summaries that are never used: about 1.5 per compaction on 1M windows today, 0.6 with this PR.

The trade-off is on windows from 128k to 262k, where a summary now starts earlier:

• More summaries are paid for sessions that end before the threshold. A summary of a shorter prefix is also cheaper, so total summary spend on Claude 200k only goes from 13.0¢ to 14.0¢ per compaction.
• More of the conversation stays verbatim after the swap: on Claude 200k, 41k tokens right after compaction instead of 27k. The model keeps more context, but every request carries more tokens. Cost per step goes up 5.6% on Claude 200k, 4.3% on Kimi and 2.4% on GPT 128k, and on Claude 200k compaction comes every 78 steps instead of 85.

On 1M windows the cost per step barely moves (+0.1% on Opus, +0.6% on GPT), and summary spend drops from 91¢ to 37¢ per compaction on Opus.

Charts

Both sides run the same simulated sessions; only the background compaction rules differ. Everything that changes, better in green and worse in red:

Where a summary starts. Today it starts 128k before the threshold on a 1M window and 13k before it on GPT 128k:

When compaction runs in one session on a 200k window. Dev waits for the summary at 4 of 16 compactions, this PR at none. Each compaction keeps more verbatim, so in the same time the PR compacts 17 times:

The same on a 1M window. Compaction runs at the same moments; dev starts its summary 128k early and throws one away:

How long the session waits when compaction is due:

What the summaries paid for turn into. On 1M windows dev pays for 3.3 summaries per compaction and uses one. On smaller windows this PR pays for more that are never used, because more sessions end within the longer lead:

How much of the context one compaction summarizes and how much it keeps:

Not in this PR:

• Keeping the old summary until its replacement is ready made no measurable difference once the tail limit was raised, so it stays one summary at a time.
• Swapping a ready summary in when the user returns after the cache expired would avoid writing the whole prompt again, but it compacts early, so I left it out.

How did you verify your code works?

• New tests in session-compaction.test.ts. They fail on dev and pass here:
  • the fixed lead on a 1M window;
  • a window whose threshold is under 64k;
  • waiting for a warm cache, with the default lifetime and with cache.ttl: "1h";
  • a 35k tail that is now kept.
• Updated tests:
  • The bursts in "fell too far behind" and "no longer fits the conversation" now exceed the new limit.
  • The cache-prefix fixture starts its cycle lower, so a background summary starts.
• Runs: bun test on the session test files in packages/core (713 pass) and bun run check from the root.
• Docs: updated compaction.mdx to describe the new start point, the cache wait and the tail limit.

Visual

<svg xmlns="http://www.w3.org/2000/svg" width="240" height="72" viewBox="0 0 240 72" fill="none">
  <style>
    .fill{transform-box:fill-box;transform-origin:left center;animation:grow 6s linear infinite}
    .sum{transform-box:fill-box;transform-origin:left center;animation:write 6s linear infinite}
    @keyframes grow{0%{transform:scaleX(.25)}70%{transform:scaleX(1)}73%,100%{transform:scaleX(.25)}}
    @keyframes write{0%,57%{transform:scaleX(0);opacity:0}58%{transform:scaleX(0);opacity:1}67%,70%{transform:scaleX(1);opacity:1}73%,100%{transform:scaleX(1);opacity:0}}
    @media (prefers-reduced-motion:reduce){.fill,.sum{animation:none}}
  </style>
  <rect x="16" y="34" width="208" height="10" rx="3" stroke="currentColor" stroke-opacity=".35"/>
  <rect class="fill" x="16" y="34" width="180" height="10" rx="3" fill="currentColor" fill-opacity=".45"/>
  <line x1="172" y1="30" x2="172" y2="48" stroke="currentColor" stroke-opacity=".45"/>
  <line x1="196" y1="24" x2="196" y2="52" stroke="currentColor" stroke-opacity=".75" stroke-dasharray="2 2"/>
  <rect x="172" y="17" width="24" height="6" rx="2" stroke="currentColor" stroke-opacity=".3"/>
  <rect class="sum" x="172" y="17" width="24" height="6" rx="2" fill="currentColor" fill-opacity=".7"/>
  <text x="167" y="22.5" text-anchor="end" font-family="system-ui,sans-serif" font-size="8" fill="currentColor" fill-opacity=".6">summary</text>
  <text x="168" y="62" text-anchor="end" font-family="system-ui,sans-serif" font-size="8" fill="currentColor" fill-opacity=".6">60k before</text>
  <text x="196" y="62" text-anchor="start" font-family="system-ui,sans-serif" font-size="8" fill="currentColor" fill-opacity=".6">threshold</text>
</svg>

Checklist

• I have tested my changes locally
• I have not included unrelated changes in this PR

Added by the opencode bot (AI-generated).

Related: #50777