[Bug]: Compaction safeguard drops messages without summarizing, causing loss of recent context #2418
Description
(Lost last lines of conversation during auto-compaction. clawdbot just didnt know what my last question was.)
Summary
During agent conversations, auto-compaction can cause loss of recent context. The agent doesn't remember what was said in the previous message(s), even though those messages should have been preserved or at
least summarized.
Steps to reproduce
- Have a long conversation that approaches the context window limit
- Continue chatting until auto-compaction triggers (context crosses
contextWindow - reserveTokens) - The compaction safeguard in
compaction-safeguard.tskicks in - On the next turn, the agent has lost context about recent messages
Expected behavior
- Recent messages should be preserved intact (controlled by
firstKeptEntryId) - Older messages should be summarized, not dropped
- The agent should remember the gist of the conversation even after compaction
Actual behavior
Messages are being dropped entirely (not summarized) when the compaction safeguard's pruning logic triggers. The agent loses context about what was discussed, including recent questions.
Root cause analysis
The bug is in src/agents/pi-extensions/compaction-safeguard.ts lines 182-204:
const summarizableTokens =
estimateMessagesTokens(messagesToSummarize) + estimateMessagesTokens(turnPrefixMessages);
const newContentTokens = Math.max(0, Math.floor(tokensBefore - summarizableTokens));
const maxHistoryTokens = Math.floor(contextWindowTokens * 0.5);
if (newContentTokens > maxHistoryTokens) {
const pruned = pruneHistoryForContextShare({
messages: messagesToSummarize,
maxContextTokens: contextWindowTokens,
maxHistoryShare: 0.5, // HARDCODED
parts: 2,
});
messagesToSummarize = pruned.messages; // Dropped messages are GONE
}
Bug flow:
1. tokensBefore = actual tokens from API
2. summarizableTokens = estimate of history tokens (uses estimateMessagesTokens)
3. If estimate is too low (underestimate) → newContentTokens becomes too high
4. This incorrectly triggers the pruning condition (newContentTokens > maxHistoryTokens)
5. pruneHistoryForContextShare() drops older chunks entirely
6. Only remaining messages get summarized - dropped messages have no summary at all
The issue in src/agents/compaction.ts pruneHistoryForContextShare() (lines 319-327):
while (keptMessages.length > 0 && estimateMessagesTokens(keptMessages) > budgetTokens) {
const chunks = splitMessagesByTokenShare(keptMessages, parts);
if (chunks.length <= 1) break;
const [dropped, ...rest] = chunks; // Dropped = gone forever
droppedChunks += 1;
droppedMessages += dropped.length;
droppedTokens += estimateMessagesTokens(dropped);
keptMessages = rest.flat();
}
The dropped chunks are never summarized - they're just removed from messagesToSummarize.
Issues identified
1. Token estimation inaccuracy: estimateMessagesTokens() can underestimate actual tokens, triggering unnecessary pruning
2. Hardcoded threshold: maxHistoryShare: 0.5 is not configurable
3. Destructive pruning: Dropped messages get no summary at all - they should at least be summarized before removal
Suggested fixes
1. Make maxHistoryShare configurable via agents.defaults.compaction.maxHistoryShare
2. Summarize before dropping: Instead of just removing chunks in pruneHistoryForContextShare, summarize them first and prepend to the final summary
3. Add safety margin to token estimation comparison: The SAFETY_MARGIN = 1.2 is used elsewhere but not in the safeguard's comparison logic
4. Log warning when pruning triggers: Currently logs at line 196-200, but could be more prominent
Workarounds
- Increase agents.defaults.compaction.reserveTokensFloor to trigger compaction earlier
- Enable agents.defaults.compaction.memoryFlush to persist context before compaction
- Use /compact manually before context gets too full
Environment
- Clawdbot version: (current main)
- OS: Linux
- Install method: source
Logs or screenshots
Console warning when pruning triggers:
Compaction safeguard: new content uses X% of context; dropped N older chunk(s) (M messages) to fit history budget.
This warning indicates messages were dropped without being summarized.