Skip to content

feat(core)!: redesign auto-compaction thresholds with three-tier ladder#4345

Merged
LaZzyMan merged 14 commits into
mainfrom
lazzy/auto-compaction-redesign-clean
May 25, 2026
Merged

feat(core)!: redesign auto-compaction thresholds with three-tier ladder#4345
LaZzyMan merged 14 commits into
mainfrom
lazzy/auto-compaction-redesign-clean

Conversation

@LaZzyMan

Copy link
Copy Markdown
Collaborator

Summary

Redesigns auto-compaction triggering from a single proportional threshold (70% of window) to a three-tier ladder (warn / auto / hard) that combines a proportional floor with absolute-byte reservations near the window edge. Pairs with a hard-tier rescue that forces compaction one send before the API would reject an oversized prompt.

  • auto = max(0.7 × window, effectiveWindow − 13K) — proactive cheap-gate
  • warn = max(0.6 × window, auto − 20K) — UX hint tier
  • hard = max(effectiveWindow − 3K, auto) — force-compaction trigger
  • effectiveWindow = window − SUMMARY_RESERVE (20K) — input + summary budget
  • COMPACT_MAX_OUTPUT_TOKENS = 20K — pinned summary cap (matches SUMMARY_RESERVE)

Small windows (32K / 64K) automatically fall back to the proportional branch — large windows are dominated by the absolute branch, capping wasted reservation at ~33K instead of 30% of the window.

Migration / breaking

  • chatCompression.contextPercentageThreshold removed. The new ladder is computed internally from the context window; the field is silently ignored with a one-line deprecation warning at startup.
  • CompressOptions.hasFailedCompressionAttempt: booleanconsecutiveFailures: number (SDK breaking change documented in design doc with migration guide).
  • Thinking disabled (includeThoughts: false) on compression side query — per-provider thinking-budget semantics are inconsistent.

Why a new PR

PR #4168 ran 12 review rounds and accumulated 6303 LOC of which ~2700 was review-driven scope creep (observability throttles, hostile-provider hardening, new escape-hatch feature, i18n baselines, cross-file constant unification, etc.) — much of it self-inflicted regression chains. The PR became un-reviewable.

This PR ships only the original spec + early-discovered real bugs (R1-R5) — 4288 LOC, 6 clean commits. The load-bearing review-driven refinements from R6-R12 are deferred to focused follow-up issues so each can be reviewed in isolation:

  • Hard-rescue counter bound (hardRescueFailureCount) — prevents infinite rescue retries on un-compressible history
  • <state_snapshot> envelope extraction — data-retention fix for scratchpad content
  • chatCompression.disabled escape hatch — opt-out for compliance / audit sessions
  • lastCandidatesTokenCount plumbing — closes one-response under-estimate in steady-state
  • Hostile-provider data hardening — NaN / Infinity / negative guards on usage fields

Old PR's full archaeology preserved at tag pr-4168-archive-pre-revert.

Design doc

docs/design/auto-compaction-threshold-redesign.md (included in this PR).

Test plan

  • npx vitest run --root packages/core src/services/chatCompressionService.test.ts src/core/geminiChat.test.ts src/services/tokenEstimation.test.ts → 172/172 passing
  • npm run typecheck --workspace=packages/core clean
  • npm run build --workspace=packages/core clean
  • E2E with real model — to be redone on this branch

LaZzyMan added 6 commits May 20, 2026 12:02
Replaces the single 70% proportional threshold with a three-tier ladder
(warn/auto/hard) that combines proportional fallback with absolute
reservation. Large-window models (>=128K) now reserve ~33K instead of
30% of the window, freeing tens of thousands of context tokens that the
old formula wasted.

Other improvements bundled in the same redesign:

- Compression sideQuery now disables thinking and caps maxOutputTokens
  at 20K, matching claude-code so the buffer math is predictable across
  providers (Anthropic/OpenAI/Gemini handle thinking budgets
  inconsistently)
- Failure handling upgraded from one-shot permanent lock to a 3-strike
  circuit breaker; reactive overflow still latches immediately
- New estimatePromptTokens helper closes the lag-by-one-turn and
  first-send-is-0 gaps in lastPromptTokenCount
- Hard-tier rescue pulls reactive overflow recovery forward to before
  the API call, saving an oversized round-trip
- /context command displays the three-tier ladder + current tier
- tipRegistry's context-* tips track the new thresholds instead of
  fixed 50/80/95 percentages

BREAKING CHANGE: chatCompression.contextPercentageThreshold setting is
removed. Settings files containing the field log a one-line deprecation
warning at startup and the value is ignored; behaviour is now controlled
by built-in thresholds via the new computeThresholds() function.

Design: docs/design/auto-compaction-threshold-redesign.md
Plan: docs/plans/2026-05-14-auto-compaction-threshold-redesign.md
…ss test

A pre-existing test case at chatCompressionService.test.ts:678 still
passed `hasFailedCompressionAttempt: false` in the CompressOptions
shape; rebasing onto current main surfaced this as a typecheck error
because the field was renamed to `consecutiveFailures` (Task 7 of the
three-tier ladder migration). Update to `consecutiveFailures: 0` —
semantically equivalent, the test asserts the side-query is called
when `force: true`, no other behaviour change.
Adds a defensive guard in ChatCompressionService.compress() that detects
when the side-query summary hit COMPACT_MAX_OUTPUT_TOKENS (20K). In that
case the summary is likely truncated mid-content, so we drop it and
return NOOP rather than persist a half-summary. The next send re-tries;
reactive overflow still catches the catastrophic case where the API
rejects the next request as too large.

Documented in the design doc as risk #2; the bot reviewer on PR #4168
correctly pushed for it to land alongside the threshold redesign rather
than as a follow-up since the new 20K cap is what makes truncation
likely in the first place.
The Task 11 redesign updated the non-interactive text formatter
(formatContextUsageText) but left ContextUsage.tsx — the interactive
React component that real /context users see — unchanged. As a result
the TUI still showed the old single "Autocompact buffer" line and none
of the new warn/auto/hard ladder.

Adds a "Compaction thresholds" section after the per-category breakdown:
  - Effective window
  - Warn / Auto / Hard threshold rows with a ▶ marker on the row the
    current usage has crossed
  - Current tier label coloured by severity (safe→green, warn/auto→
    yellow, hard→red)

The existing progress bar legend (Used / Free / Autocompact buffer)
is preserved because it's tied to the three-segment progress bar
visualisation; the new section adds the absolute numbers + tier badge
on top of that.

Caught by the tmux e2e test (PR #4168 ci-monitor follow-up). Pre-fix
the assertion 'Compaction thresholds' missed completely from the TUI;
post-fix the new section renders correctly for fresh and live sessions
on 1M / 200K / 128K windows.
Behavior fixes:
- MAX_TOKENS truncation guard now returns COMPRESSION_FAILED_EMPTY_SUMMARY
  instead of NOOP so the consecutive-failure breaker actually trips after
  repeated max-length summaries (R1.1).
- Reactive overflow failure increments consecutiveFailures by 1 instead
  of latching to MAX in one shot, so a transient network blip doesn't
  permanently disable auto-compaction. The hard-tier rescue resets the
  counter, which remains the designated recovery path (R1.2).
- /context current-tier classification uses rawOverhead (system + tools +
  memory + skills) as the tier input when API data is not yet available,
  rather than 0 — large inherited contexts no longer silently show 'safe'
  (R2.2).

Performance:
- sendMessageStream computes effectiveTokens ONCE and passes it through
  TryCompressOptions.precomputedEffectiveTokens, so the cheap-gate inside
  service.compress doesn't redo the estimation. Also fixes the
  imageTokenEstimate inconsistency between the rescue and cheap-gate
  paths (R1.3 + R1.4).
- Steady-state path (lastPromptTokenCount > 0) skips the costly
  getHistory(true) clone — estimatePromptTokens only needs the user
  message in that branch.

Code hygiene:
- BYTES_PER_TOKEN → CHARS_PER_TOKEN (inputs are char counts, not byte
  counts; CJK text would mislead under the old name) (R3.1).
- Drop dead getContextUsagePercent helper + index re-export — no callers
  in source after the threshold rewire (R1.5).
- Add a comment on estimatePromptTokens' first-send fallback documenting
  the ~15-20K under-estimate (system prompt + tools + skills) and that
  reactive overflow is the safety net (R3.3).

Tests:
- New CLI ContextUsage.test.tsx exercises the React renderer for the
  three-tier section: section presence, ▶ marker placement per tier,
  current-tier label coloring (R1.6).
- New chatCompressionService.test.ts case pins that a stale
  contextPercentageThreshold: 0 value in user settings no longer
  short-circuits compaction (R2.1).
- New tokenEstimation.test.ts case covers functionResponse (distinct
  nested-parts branch from functionCall) (R3.5).
- New geminiChat.test.ts integration test exercises the real
  ChatCompressionService — not a mock — for the first-send-after-
  inherited-history scenario where lastPromptTokenCount=0 and only the
  full-history estimate can cross the auto threshold (R3.4).

Declined: R3.2 (change `>=` to `>` on the MAX_TOKENS guard). The current
operator catches the at-cap case as suspicious, which is intentional —
landing exactly at the output cap is far more likely truncation than
clean stop given p99.99 ≈ 17K. With R1.1 in place, persistent truncations
trip the breaker after MAX_CONSECUTIVE_FAILURES so the worst case is
bounded.
- R5.1: tighten /context tier comment + TODO. The rawOverhead-based fix
  doesn't cover `--continue` restores with many history messages (since
  rawOverhead excludes messagesTokens). UI may still show 'safe' for one
  render until the first send. Documented inline and added a TODO to plumb
  chat history into collectContextData for same-source-of-truth as the
  cheap-gate.
- R5.2a: add TODO(finish_reason) at the truncation guard. The `>= cap`
  heuristic false-positives on legitimate at-cap summaries; the proper
  signal is finish_reason which runSideQuery doesn't surface today.
- R5.2b: split telemetry — new CompressionStatus.COMPRESSION_FAILED_OUTPUT_TRUNCATED
  enum value. Distinct from EMPTY_SUMMARY so logs/telemetry can tell
  prompt-quality failures (tune prompt / splitter) from capacity failures
  (raise cap / shrink splitter input). isCompressionFailureStatus()
  treats both as failures so the breaker behavior is unchanged.
- R5.3: expand consecutiveFailures JSDoc to clarify it tracks
  "non-force, non-hard-rescue consecutive failures" — hard-rescue resets
  the counter and force=true skips increments, so the counter is the
  "regular path" health signal only; reactive overflow is the real
  safety net for the force-only paths.
- R5.4: document the CompressOptions field rename
  (hasFailedCompressionAttempt: boolean → consecutiveFailures: number)
  as an SDK breaking change in the design doc with migration guide.
@github-actions

Copy link
Copy Markdown
Contributor

📋 Review Summary

This PR redesigns auto-compaction triggering from a single proportional threshold to a three-tier ladder (warn/auto/hard) combining proportional floors with absolute-byte reservations. The implementation is well-documented, thoroughly tested, and addresses real issues from the previous PR #4168. Overall assessment: solid engineering with minor areas for improvement.

🔍 General Feedback

  • Excellent documentation: The design doc clearly explains the problem space, trade-offs, and numerical choices. The threshold ladder diagram is particularly helpful.
  • Good separation of concerns: Pure computeThresholds() function makes the logic testable and reusable.
  • Thoughtful migration path: Deprecation warning for removed config field is user-friendly.
  • Test coverage: 172/172 tests passing with dedicated tests for new token estimation logic.
  • Positive aspects:
    • Conservative token estimation approach (char/4 ratio) is safe for triggering compaction earlier
    • Defensive guard for max-length summaries prevents silent truncation
    • Pre-computed effective tokens optimization avoids redundant history cloning
  • Architectural decision: Keeping internal constants (PCT, buffers) rather than exposing all as config is appropriate for this complexity level

🎯 Specific Feedback

🟡 High

  • packages/core/src/services/chatCompressionService.ts:539 - The guard for compressionOutputTokenCount >= COMPACT_MAX_OUTPUT_TOKENS uses a heuristic that can false-positive on legitimate summaries exactly at the cap. The TODO comment acknowledges this needs proper finish_reason plumbing. Recommendation: Consider this a follow-up blocker if the false-positive rate is high — marking safe compressions as failures could trigger the circuit breaker unnecessarily.

  • packages/core/src/services/tokenEstimation.ts:67-75 - The fallback path when lastPromptTokenCount === 0 admits it "MISSES the system prompt (~8-15K), tool definitions (~5K), skill content, and cache headers — typically ~15-20K of under-estimate." Recommendation: This relies on reactive overflow as a "safety net" — consider adding a debug log when this path triggers so operators can detect systematic under-estimation patterns.

🟢 Medium

  • packages/core/src/services/chatCompressionService.ts:513 - The sideQuery config still shows includeThoughts: true in the comment but the actual code has includeThoughts: false. Recommendation: Update the comment to match the implementation to avoid confusion.

  • packages/core/src/core/geminiChat.ts:547-550 - The hasFailedCompressionAttempt flag is still being set in the old single-failure pattern (this.hasFailedCompressionAttempt = true), but the new design uses consecutiveFailures counting. Recommendation: Verify this is intentional backward compatibility or update to use the new counter.

  • packages/cli/src/services/tips/tipRegistry.ts:36-65 - The tip thresholds (50/80/95%) are now decoupled from the auto-compaction thresholds (60/70% + absolute buffers). Recommendation: Consider aligning tip percentages with the new threshold tiers or at least documenting why they differ.

🔵 Low

  • packages/core/src/services/chatCompressionService.ts:53 - The COMPRESSION_TOKEN_THRESHOLD constant is now unused (replaced by DEFAULT_PCT in the ladder). Recommendation: Remove this deprecated constant to avoid confusion.

  • packages/core/src/services/tokenEstimation.ts:20 - The CHARS_PER_TOKEN = 4 constant could benefit from a reference to the claude-code source or empirical validation data in the comment.

  • packages/cli/src/ui/components/views/ContextUsage.tsx:62-83 - The progress bar shows the autocompact buffer visually, but the buffer size varies by window size (absolute vs proportional branch). Recommendation: Consider computing the actual buffer percentage from computeThresholds() rather than using a fixed visual representation.

  • packages/core/src/config/config.ts:270-271 - The ChatCompressionSettings.contextPercentageThreshold field should be marked as @deprecated in the JSDoc to match the runtime deprecation warning.

✅ Highlights

  • Three-tier threshold ladder: Excellent solution to the "wasted reservation" problem on large windows. The max(proportional, absolute) pattern is elegant and handles edge cases gracefully.
  • Token estimation compensation: The estimatePromptTokens() function correctly closes both gaps (missing current user message +首轮为零) identified in the design doc.
  • Pre-computed effective tokens optimization: The precomputedEffectiveTokens parameter in CompressOptions shows thoughtful performance consideration — avoids duplicate getHistory(true) cloning on every send.
  • Defensive summary truncation guard: The check for max-length output with distinct COMPRESSION_FAILED_OUTPUT_TRUNCATED status enables better observability and debugging.
  • Pure function design: computeThresholds() has no side effects, making it easy to test and reason about.
  • Comprehensive test coverage: New tokenEstimation.test.ts covers edge cases including inlineData, functionCall, and functionResponse estimation.

@github-actions

github-actions Bot commented May 20, 2026

Copy link