feat(compaction): opt-in hard-floor for incremental compaction below contextThreshold by 100yenadmin · Pull Request #619 · Martian-Engineering/lossless-claw

100yenadmin · 2026-05-06T18:54:45Z

TL;DR

Adds an opt-in config flag respectThresholdAsHardFloor (default false, no behavior change for existing users). When enabled, evaluateIncrementalCompaction short-circuits with reason="below-context-threshold-floor" whenever currentTokenCount < contextThreshold * tokenBudget, regardless of cache state, leaf-trigger fires, or activity-band heuristics. The deferred-mode debt-recording path at engine.ts:6548 is gated under the same predicate to prevent silent leakage of the floor through the deferred drain.

This is a policy enhancement, not a bug fix. The default behavior continues to allow cold-cache catch-up passes to fire below threshold (the existing design that some users want). Users who instead want strict "never compact below X%" semantics — typically because of bursty/idle conversation patterns where step-away gaps cause cold-mode firing to bleed context — can now opt in with one flag.

The problem this addresses, visualized

Without the flag (existing behavior):
   ┌─────────────────────────────────────────────────────────────────┐
   │  step-away 20 min  →  cache cold  →  next turn               │
   │      ↓                                                        │
   │  cold-mode catch-up fires (maxColdCacheCatchupPasses leaf passes) │
   │      ↓                                                        │
   │  context drops by ~2× chunkSize even though %-used was low    │
   │      ↓                                                        │
   │  step-away 5 more min  →  cache cold again  →  fires again    │
   │      ↓                                                        │
   │  pattern repeats; context drains during idle gaps             │
   └─────────────────────────────────────────────────────────────────┘

With respectThresholdAsHardFloor=true:
   ┌─────────────────────────────────────────────────────────────────┐
   │  evaluateIncrementalCompaction first checks:                  │
   │    currentTokenCount  vs  floor (= contextThreshold × budget) │
   │      │                                                        │
   │      ├── below floor → return shouldCompact=false             │
   │      │                  reason="below-context-threshold-floor"│
   │      │                  (cache-aware/leaf-trigger gates skipped)│
   │      │                                                        │
   │      └── at/above floor → existing logic runs unchanged       │
   │                            (budget-trigger, hot-cache gates,  │
   │                             cold catch-up, etc.)              │
   └─────────────────────────────────────────────────────────────────┘

Behavior matrix

Flag	currentTokenCount	Decision
`false` (default)	any	existing logic — all gates and triggers behave as before
`true`	undefined	falls through to existing logic (no floor enforced when count is unknown)
`true`	< `contextThreshold × tokenBudget`	short-circuit: `shouldCompact=false reason="below-context-threshold-floor"`
`true`	≥ `contextThreshold × tokenBudget`	falls through to existing logic
`true` (deferred mode)	< floor	also skips `recordDeferredCompactionDebt` (prevents drain leak)
`true` (deferred mode)	≥ floor	existing deferred-debt recording behavior

Diff overview

3 commits:

1c99131 feat(compaction): add opt-in respectThresholdAsHardFloor flag
- src/db/config.ts: add respectThresholdAsHardFloor: boolean to LcmConfig, parser entry with env-var support (LCM_RESPECT_THRESHOLD_AS_HARD_FLOOR), default false.
- src/engine.ts: 30-line guard at the top of evaluateIncrementalCompaction, returns through the existing logIncrementalCompactionDecision so the new reason shows up in standard telemetry.
- Tests in test/engine.test.ts and test/config.test.ts.
434eb28 fix(compaction): plug deferred-mode hard-floor leak; add boundary tests
- In deferred mode (engine.ts:6548-6603), debt is recorded based on thresholdDecision.shouldCompact || rawLeafTrigger?.shouldCompact — both use unguarded compaction.evaluate / evaluateLeafTrigger. Without this fix, debt would accumulate from below-floor leaf-trigger fires and later drain via paths that don't re-check the floor.
- Adds the same predicate guard around recordDeferredCompactionDebt and the deferred-drain scheduling, with an explanatory log line: [lcm] afterTurn: skipping deferred compaction debt below context-threshold floor ….
- Adds boundary tests: exact equality at floor, undefined currentTokenCount with flag enabled, and an integration test for the deferred-mode skip.
8cc4e63 feat(manifest): expose respectThresholdAsHardFloor in plugin config schema and uiHints
- openclaw.plugin.json: adds respectThresholdAsHardFloor: { type: "boolean" } to configSchema.properties (required so OpenClaw's host-side validator with additionalProperties: false accepts the new field). Adds a uiHints entry with label and help text so the field shows up in the OpenClaw plugin settings UI.

Naming conventions

Flag: respectThresholdAsHardFloor — follows the existing pattern of compound-clause boolean flags.
Env var: LCM_RESPECT_THRESHOLD_AS_HARD_FLOOR — matches the existing LCM_* env pattern for plugin config overrides.
Reason string: "below-context-threshold-floor" — follows the hyphenated convention of "budget-trigger", "hot-cache-defer", "hot-cache-budget-headroom", "below-leaf-trigger".
Telemetry: routed through the existing logIncrementalCompactionDecision so the new reason appears in the same [lcm] incremental compaction decision: … lines operators already grep.

What is intentionally NOT changed

compactFullSweep and compactUntilUnder are not affected. Forced compaction (manualCompaction: true, overflow recovery, explicit compact({force: true})) bypasses evaluateIncrementalCompaction entirely and continues to work above the soft floor as before. This matters for emergency overflow recovery, which should always be allowed.
Default behavior is preserved exactly — when the flag is false (the default), zero lines of pre-existing logic execute differently. Verified via the regression test evaluateIncrementalCompaction preserves default behavior when respectThresholdAsHardFloor is disabled.

Tests

test/engine.test.ts — 5 new tests:
- evaluateIncrementalCompaction short-circuits below contextThreshold when respectThresholdAsHardFloor is enabled — verifies short-circuit and that evaluateLeafTrigger/evaluate are NOT called.
- evaluateIncrementalCompaction allows existing compaction logic at/above contextThreshold even when hard-floor is enabled — verifies budget-trigger still fires above floor.
- evaluateIncrementalCompaction preserves default behavior when respectThresholdAsHardFloor is disabled — regression guard for the default path.
- evaluateIncrementalCompaction allows existing logic at exact floor boundary even with hard-floor enabled — boundary test for currentTokenCount === minTokenFloor (uses < not <=, so the boundary admits compaction).
- evaluateIncrementalCompaction falls through to existing logic when currentTokenCount is undefined even with hard-floor enabled — verifies the type-guard fall-through case.
- afterTurn in deferred mode skips deferred-debt recording when below contextThreshold floor — integration test for the deferred-mode leak fix (commit 434eb28).
test/config.test.ts — 3 path coverages:
- Default false in the hardcoded-defaults test.
- Plugin-config override path.
- Env-var override path (LCM_RESPECT_THRESHOLD_AS_HARD_FLOOR=true).

Full suite: 860/860 passing.

Independence

PR #619 is independent of PR #621. They address different concerns:

feat(compaction): opt-in hard-floor for incremental compaction below contextThreshold #619 (this PR): policy enhancement — opt-in hard-floor for incremental compaction.
fix(afterTurn): evaluate compaction even when ingestBatch is empty #621: bug fix — afterTurn early-return on empty ingestBatch skips compaction evaluation entirely (different code path).

No merge-order dependency. Either PR can land first.

Test plan

npm run build — bundles cleanly (689kb).
npm test — 860/860 passing.
Default-false preserves existing behavior across all 855 pre-existing tests (zero regressions).
Adversarial review (3 reviewers, separately checking flag interaction, regression risk, and mechanism) — all cleared.
Manifest schema accepts the new field; verified with the host's config validator (additionalProperties: false).

When enabled, evaluateIncrementalCompaction short-circuits with reason="below-context-threshold-floor" whenever currentTokenCount < contextThreshold * tokenBudget, regardless of cache state, leaf trigger, or activity band. Motivation: cold-cache catch-up passes can compact context away during idle gaps even when overall context usage is well below the configured threshold. Step-away patterns (idle 20 min → cold turn fires 2× leaf passes → idle 5 min → another 2× passes) can bleed conversations down to the freshTail. Users who want a strict "never compact below X%" policy now have a single flag to enforce it. Default false preserves existing behavior. Configurable via plugin config or env var LCM_RESPECT_THRESHOLD_AS_HARD_FLOOR=true.

Adversarial review found that respectThresholdAsHardFloor only gated evaluateIncrementalCompaction. In deferred mode, the after-turn flow records compaction debt based on `thresholdDecision.shouldCompact || rawLeafTrigger?.shouldCompact`, both of which use unguarded `compaction.evaluate` and `compaction.evaluateLeafTrigger` calls. Without this fix, debt would accumulate from below-floor leaf-trigger fires and later drain via paths that do not re-check the floor — silently leaking the hard-floor semantics. Fix: skip deferred debt recording entirely when the hard-floor flag is enabled and currentTokenCount is below the floor. Logs a single info line so operators can verify suppression. Tests added: - Boundary: currentTokenCount === minTokenFloor (exact equality; strict `<` permits compaction at the boundary, by design). - Boundary: currentTokenCount undefined with flag enabled (falls through to existing logic). - Integration: afterTurn in deferred mode with hard-floor enabled skips deferred-debt recording when below floor (regression guard for the leak). Suite: 864/864 passing (was 861).

100yenadmin · 2026-05-06T19:00:14Z

Adversarial review pass — one HIGH finding fixed

Ran a 3-agent parallel adversarial review on the initial commit. Two reviewers independently flagged a HIGH-severity leak in deferred mode: the hard-floor only short-circuited evaluateIncrementalCompaction, but the after-turn flow at engine.ts:6548 records deferred debt based on thresholdDecision.shouldCompact || rawLeafTrigger?.shouldCompact — both of which use unguarded compaction.evaluate / compaction.evaluateLeafTrigger calls. Below-floor leaf-trigger fires would accumulate debt that later drained via paths that didn't re-check the floor.

Fixed in 434eb28 by gating deferred-debt recording on the same floor predicate. Operators get one info-level log line per skipped recording so suppression is observable.

Other findings + dispositions

Severity	Finding	Disposition
HIGH	Deferred-mode debt leak	Fixed in `434eb28`
MEDIUM	New `"below-context-threshold-floor"` reason isn't recognized by `shouldForceDeferredPromptCacheLeafCompaction` (engine.ts:2400) or the deferred `maintenance.reason === "threshold"` check (engine.ts:3196)	Working as intended — both correctly return false / no-match for the new reason since the floor short-circuit is meant to be terminal, not retried
LOW	Boundary tests: exact equality, undefined currentTokenCount with flag enabled	Added (2 new tests in `434eb28`)
LOW	`activityBand: "low"` hardcoded in short-circuit telemetry	Working as intended; documented via the patch comment
✓	Manual / forced compaction (`compact({force:true})`, `compactUntilUnder`, `compactFullSweep`)	Confirmed bypass `evaluateIncrementalCompaction` and continue to work as intended

Suite

864/864 passing (3 new tests added since the initial commit).

…chema and uiHints The host-side OpenClaw config validator uses the plugin manifest's configSchema (additionalProperties: false). Without adding the new field there, hot-reload rejects the config with 'must NOT have additional properties'.

afterTurn early-returned at engine.ts:6294-6300 when deduplicateAfterTurnBatch removed every new message — typically because afterTurnTranscriptReconcile (or per-message engine.ingest calls during the turn) had already imported the new content before the dedup check ran. Long-running conversations could accumulate well past contextThreshold without ever triggering an incremental leaf pass, deferred-compaction debt record, or budget-trigger run. Symptom: every afterTurn log ends with "nothing to ingest" and zero [lcm] incremental compaction decision events appear, even when the host's [context-overflow-precheck] is firing emergency truncation. Replace the early-return with a fall-through: when ingestBatch is empty the actual ingest call is skipped, but token-budget resolution, conversation lookup, and compaction evaluation still run. The existing ingest-failure return path is preserved. Tests: - afterTurn evaluates compaction when ingestBatch is empty (regression guard for the early-return; with mocked dedup-eats-all and over-threshold telemetry, deferred debt is recorded) - afterTurn skips compaction work when empty AND below threshold (existing no-op behavior preserved) Suite: 860/860 passing.

…ed paths Empirical evidence on a live DB (conv 1872) showed 9 leaf compactions plus 1 condensed compaction (eating 27 leaves) fired AFTER the respectThresholdAsHardFloor flag was armed at 2026-05-11 18:00. The condensed pass alone compressed ~467K source tokens into 2024 summary tokens. Total damage: ~616K source tokens collapsed into ~7.7K of summary while the operator believed the floor was holding. Root cause: PR Martian-Engineering#619 v1 only short-circuited evaluateIncrementalCompaction and the afterTurn deferred-debt-recording path. The actual compaction entry points (CompactionEngine._compactLeafImpl, compactFullSweep, and the inner condensed-pass loops inside _compactLeafImpl) had separate gates that consulted only `tokensBefore > threshold || leafTrigger. shouldCompact` — leaf-trigger still fired below floor. Fix: - CompactionConfig adds `respectThresholdAsHardFloor?: boolean` - engine.ts plumbs config.respectThresholdAsHardFloor into CompactionConfig - _compactLeafImpl: hard-floor short-circuit at entry (before leaf-trigger check); plus per-iteration check in the inner condensed loop so a legitimate leaf pass that drops runningTokens below floor does NOT proceed to condensed passes - compactFullSweep: matching hard-floor short-circuit at entry. Phase 1 (leaf) and Phase 2 (condensed) loops already break on `passTokensAfter <= threshold` / `previousTokens > threshold`, which is correct under the hard-floor semantics — they stop the moment we cross the boundary. After the fix, every compaction entry point honors the commit-message promise of PR Martian-Engineering#619 v1: "never compact when currentTokenCount < contextThreshold * tokenBudget, regardless of cache state, leaf trigger, or activity band." Tests: 236/236 passing on test/engine.test.ts.

PR follow-up to Martian-Engineering#619. Adds the lower-level plumbing for the "accordion" compaction cadence (90% trigger → 35% target → repeat) that pairs with the openclaw session_before_compact intercept hook. Three layers changed: 1. CompactionEngine.compact / compactFullSweep - New optional `stopAtTokens` param on both. When set, both Phase 1 (leaf) and Phase 2 (condensed) loops break the moment running tokens drop to or below this value, regardless of force/threshold. Lets the accordion target stop AT the operator's floor instead of overshooting to the freshTailMaxTokens floor (~9% of budget). - Validated to positive finite numbers; bad values are normalized to null (no precise stop) and legacy behavior is preserved. 2. CompactionEngine.compactUntilUnder - Now forwards `stopAtTokens: targetTokens` to compact() ONLY when the resolved target is strictly less than tokenBudget. This preserves the existing auth-recovery semantics that depend on force=true running the sweep to exhaustion (circuit-breaker test verifies multi-pass invocation). 3. LcmContextEngine.compact (CompactionExecutionParams) - New optional `compactionTargetFraction?: number` in (0, 1]. - When set + valid, becomes the effective target: targetTokens = floor(fraction * tokenBudget) and routes through the convergence loop (compactUntilUnder) which respects targetTokens. compactFullSweep is explicitly NOT used for fraction-target because it has no notion of a caller-supplied target (without the new stopAtTokens forwarding, which only compactUntilUnder uses). - Bad values (≤0, >1, non-finite, undefined) are silently ignored and the legacy compactionTarget enum is used. Tests: - 5 new validation tests in test/compaction-target-fraction.test.ts - 1615/1615 passing across the suite (including the circuit-breaker auth-recovery test that depends on legacy behavior when targetTokens == tokenBudget). Backward compat: - All new params are optional with sane defaults - When unset / invalid, behavior is byte-identical to pre-PR - Existing callers (afterTurn, deferred drain, overflow recovery) don't pass the new params and get unchanged semantics

…ompact PR follow-up to Martian-Engineering#619. Implements the lossless-claw side of the session_before_compact integration with pi-coding-agent (verified by a prior research agent: the SDK exposes this event as a true interception point, not a notification). When openclaw's PR #1 wires this to codex's compaction lifecycle, codex's GPT summarization call is BYPASSED entirely and our lossless compaction output is used as the replacement_history instead. Method signature on LcmContextEngine: interceptCompaction(request) → Promise< | { handled: true, summary, tokensAfter, firstKeptEntryId, tokensBefore } | { handled: false, reason: string } > Flow: 1. Five guard rails (ignored / stateless / no-target-fraction-configured / pre-compaction abort / mid-compaction abort) → handled:false 2. engine.compact() with force=true + the operator-configured compactionTargetFraction (defaults to 0.35 under OAuth profile via CODEX_OAUTH_DEFAULTS — applied at config layer in step 1's commit). 3. engine.assemble() to get the post-compaction AgentMessage[]. 4. Guard 5: if assembled.messages is empty, return handled:false so codex falls back to its native compaction (don't hand codex a near-empty summary). 5. Serialize AgentMessage[] → single string via the new helper serializeAssembledMessagesForCompaction. Each message becomes a `[role]\n<body>` block; arrays of content blocks join text blocks plainly and JSON-encode tool_use/tool_result/image blocks to preserve structure. 6. Return handled:true with the summary, echoing firstKeptEntryId and tokensBefore unchanged for codex's history-shape invariants. Defensive properties: - Never throws across the SDK boundary (try/catch wraps step 2-5; errors return handled:false with describeLogError). - Respects AbortSignal before and during compaction. - Honors all existing session-exclusion rules. Tests (9 new in test/intercept-compaction.test.ts): - handled:false for unset / invalid compactionTargetFraction - handled:false for ignored / stateless sessions - handled:false on pre-compaction abort - handled:false when LCM has no assembled context (e.g. unseeded conv) - Never throws on pathological inputs - Validation surface matches (0, 1] contract 1631/1631 tests passing across the suite. Backward compat: New method is purely additive. interceptCompaction is ONLY called by openclaw when PR #1 wires the session_before_compact handler; pre-PR-#1 plugin behavior is byte-identical.

PR follow-up to Martian-Engineering#619. Completes the OAuth profile by connecting detection at plugin registration time to the config resolver, so CODEX_OAUTH_DEFAULTS are applied when a Codex-configured host registers. Two changes: 1. detectCodexOAuthSync helper in plugin/index.ts - Best-effort SYNCHRONOUS detection at register-time - ANY-of signal detection: openclawDefaultModel prefix, summaryProvider/Model, expansionProvider/Model, largeFileSummaryProvider/Model - Strict prefix matching with case-insensitive + whitespace-trim handling - Defensive against non-string inputs - Why sync (not async): plugin register() is sync-shaped today; awaiting modelAuth.resolveApiKeyForProvider() would require touching many call sites. The v1 simplification accepts a slight false- positive surface (API-key Codex users get the OAuth profile unless they opt out via codexOAuthProfile: "off"). API-key codex is rare; trade-off favors getting OAuth users on the correct cadence today. - V2 plan: thread async detection through register() for strict `resolveApiKeyForProvider().mode === "oauth"` checking. 2. RuntimeModelAuthResult.mode field - Added optional mode?: string to the local type alias. - Mirrors `ResolvedProviderAuth.mode` in plugin-sdk/src/agents/model-auth-runtime-shared.d.ts. - Forward-compatible: when V2 lands, this field becomes the strict signal without typedef churn. 3. createLcmDependencies plumbing - Calls detectCodexOAuthSync(envSnapshot, pluginConfig) at the top of dep-building (after modelAuth resolves). - Passes the boolean to resolveLcmConfigWithDiagnostics(env, pc, oauthProfileActive) as the new third arg. - Failure path: wrapped in try/catch; detection errors log a warning and fall back to oauthProfileActive=false (legacy defaults). Plugin registration never blocks on detection. 4. Startup banner - When diagnostics.codexOAuthProfileApplied is true, logs an info-level banner once per process documenting the locked defaults (0.90 / 0.35 / hard-floor / deferred) and how to opt out. Operators can verify the profile is active from gateway logs. Tests (8 new in test/codex-oauth-detection.test.ts): - No signals → false (default) - openclawDefaultModel prefix match - summaryProvider / Model match - expansionProvider / Model + largeFileSummary* match - Non-string inputs (defensive) - Whitespace + case-insensitive normalization - ANY-OF semantics — one signal sufficient - Strict prefix — "openai-codex-mini/foo" does NOT match 1640/1640 tests passing across the suite. Backward compat: - Detection failure path leaves oauthProfileActive=false → legacy defaults preserved. - Hosts that don't use Codex see no behavior change. - Codex hosts can opt out via `codexOAuthProfile: "off"`.

PR follow-up to Martian-Engineering#619. Exposes the two new config fields via the plugin manifest so they appear in the openclaw plugin UI (with uiHints) and pass gateway-side schema validation (configSchema). uiHints additions: - codexOAuthProfile — explains "auto" detection + opt-out - compactionTargetFraction — explains pairing with contextThreshold for the accordion cadence configSchema additions: - codexOAuthProfile: string enum ["auto", "off"] - compactionTargetFraction: number in (0, 1] (exclusiveMinimum 0) 48 schema properties total. JSON parses cleanly. 1640/1640 tests passing (no test changes — schema is data, not code).