Skip to content

compaction.buffer: flat token count can't serve mixed model windows (no percentage or per-model form) #59994

Closed as duplicate

Description

@JustinKeltner

What

compaction.buffer accepts only an absolute token count (NonNegativeInt). Its default is proportional — limit.input - max(15% of limit.input, 16_000) — but the moment buffer is set explicitly that proportional behaviour is discarded and one absolute number applies to every model in the config.

Why that's a problem

A single role commonly mixes model windows of very different sizes (e.g. a 131K local model and a 1M cloud model in the same opencode.jsonc). One flat reserve cannot serve both:

model usable input buffer: 30000 reserve as share of input
local 131K 98,304 68,304 30.5%
cloud 1M 980,000 950,000 3.1%

A buffer large enough to protect a slow local model's summarization request pushes that model down to ~52% of its window; a buffer small enough to be negligible for the cloud model leaves the local summarizer no headroom. There is no value that is correct for both.

Evidence from a real deployment

Raising buffer 10,000 → 30,000 fixed timeouts on one single-GPU local model. Because the key is global, it silently moved every model's trigger on that role: the local 131K model dropped from ~84% of its usable input to ~52% of its window. Nothing in the config said 52% — the flat buffer did. Diagnosing it required reading calculateCeiling() and the config backup history.

Related: a flat buffer also interacts badly with a per-model limit.input pin, because the trigger is limit.input - buffer while the UI gauge divides by limit.context. Two models with identical buffer values show completely different "percent used" at the moment they compact.

Proposal

Any one of these would remove the footgun:

  1. Accept a percentage, applied per model against limit.input:
    "compaction": { "buffer": "10%" }
  2. Per-model override, mirroring the existing limit block:
    "provider": { "x": { "models": { "m": {
      "limit": { ... },
      "compaction": { "buffer": 10000 }
    } } } }
  3. At minimum, warn when buffer is set on a config whose models span very different windows. Today an unknown key under compaction is silently dropped, so a typo'd percentage key looks configured and does nothing.

Repro

{
  "compaction": { "auto": true, "buffer": 30000, "keep": { "tokens": 15000 } },
  "provider": {
    "local": { "models": { "small": { "limit": { "context": 131072,  "input": 98304,  "output": 32768  } } } },
    "cloud": { "models": { "large": { "limit": { "context": 1000000, "input": 980000, "output": 131072 } } } }
  }
}

Compaction then fires at 68,304 for small (52% of its window) and 950,000 for large (95% of its window).

Version

v2.1.4 (also present on dev): packages/schema/src/config/compaction.ts declares buffer: NonNegativeInt; RESERVE_SHARE = 0.15 and RESERVE_MIN = 16_000 are module constants in packages/core/src/session/compaction.ts, not config keys

Activity

  1. opencodejr commented on Oct 28, 2026

    @opencodejr
    Contributor

    Thanks for the detailed write-up and the worked numbers!

    I think this is a duplicate of #43703, which asks for the same thing: a compaction buffer per model, so models with very different windows don't have to share one global value. Possible duplicates and related items, ordered from most to least likely:

    The warning you suggest for unknown keys under compaction being dropped silently (option 3) isn't covered by these. If you'd like it tracked, a separate, focused issue would be best.

    The bot will close this issue as a duplicate in 1 day unless you reply explaining how your problem is different.

  2. opencodejr commented on Oct 29, 2026

    @opencodejr
    Contributor

    Closing this as a duplicate of #43703, since there's been no reply in the past day. #43703 covers the same problem: a single global compaction.buffer can't serve models with very different context windows. Open PR #56443 adds a per-model and per-provider buffer for it. Please follow those for updates.

    If your case turns out to be different, or you'd like the warning about unknown compaction keys being dropped tracked on its own, please comment here or open a focused issue. Thanks again for the detailed report!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions