fix(telegram): add cross-turn dedup cache to prevent duplicate messages after context compaction#39456
fix(telegram): add cross-turn dedup cache to prevent duplicate messages after context compaction#39456eveiljuice wants to merge 2 commits intoopenclaw:mainfrom
Conversation
shouldSkipAssistantText() only deduplicated within the same assistantMessageIndex. After context compaction the index resets, so when the model re-emits the same text from the compaction summary it bypasses the dedup check and gets delivered as a new message. Added recentDeliveredTexts — a bounded rolling cache (max 20 entries, 1h TTL) that persists across assistant message turns within a session. Before delivering text, normalized characters are compared against recently delivered hashes. Matches are skipped. Includes unit tests covering hash building, cross-turn duplicate detection, TTL expiration, whitespace/casing normalization, cache eviction and capacity limits. Closes #37702 Related: #38434, #33308, #33453, #33592, #37697, #30316
Greptile SummaryThis PR introduces a bounded rolling cache ( Two issues found:
Confidence Score: 3/5
Last reviewed commit: c1e5ad4 |
| // 53-bit FNV-1a-inspired hash (fits in a JS safe integer). | ||
| let h = 0x811c9dc5; | ||
| for (let i = 0; i < normalized.length; i++) { | ||
| h ^= normalized.charCodeAt(i); | ||
| h = Math.imul(h, 0x01000193); | ||
| } | ||
| // Combine prefix + length + full-text hash for uniqueness. | ||
| return `${normalized.slice(0, 200)}|${normalized.length}|${(h >>> 0).toString(36)}`; |
There was a problem hiding this comment.
Hash described as 53-bit but is actually 32-bit
The comment claims a "53-bit FNV-1a-inspired hash", but Math.imul performs 32-bit integer multiplication with 32-bit truncation, and h >>> 0 produces a 32-bit unsigned result. The hash has only 2³² (~4.3 billion) possible values, not 2⁵³. Saying "fits in a JS safe integer" is true (32 < 53), but labelling it "53-bit" misrepresents the hash's entropy and could mislead future readers about collision resistance.
| // 53-bit FNV-1a-inspired hash (fits in a JS safe integer). | |
| let h = 0x811c9dc5; | |
| for (let i = 0; i < normalized.length; i++) { | |
| h ^= normalized.charCodeAt(i); | |
| h = Math.imul(h, 0x01000193); | |
| } | |
| // Combine prefix + length + full-text hash for uniqueness. | |
| return `${normalized.slice(0, 200)}|${normalized.length}|${(h >>> 0).toString(36)}`; | |
| // 32-bit FNV-1a-inspired hash (Math.imul truncates to 32 bits; fits comfortably in a JS safe integer). | |
| let h = 0x811c9dc5; | |
| for (let i = 0; i < normalized.length; i++) { | |
| h ^= normalized.charCodeAt(i); | |
| h = Math.imul(h, 0x01000193); | |
| } | |
| // Combine prefix + length + full-text hash for uniqueness. | |
| return `${normalized.slice(0, 200)}|${normalized.length}|${(h >>> 0).toString(36)}`; |
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/pi-embedded-helpers/messaging-dedupe.ts
Line: 35-42
Comment:
**Hash described as 53-bit but is actually 32-bit**
The comment claims a "53-bit FNV-1a-inspired hash", but `Math.imul` performs 32-bit integer multiplication with 32-bit truncation, and `h >>> 0` produces a 32-bit unsigned result. The hash has only 2³² (~4.3 billion) possible values, not 2⁵³. Saying "fits in a JS safe integer" is true (32 < 53), but labelling it "53-bit" misrepresents the hash's entropy and could mislead future readers about collision resistance.
```suggestion
// 32-bit FNV-1a-inspired hash (Math.imul truncates to 32 bits; fits comfortably in a JS safe integer).
let h = 0x811c9dc5;
for (let i = 0; i < normalized.length; i++) {
h ^= normalized.charCodeAt(i);
h = Math.imul(h, 0x01000193);
}
// Combine prefix + length + full-text hash for uniqueness.
return `${normalized.slice(0, 200)}|${normalized.length}|${(h >>> 0).toString(36)}`;
```
How can I resolve this? If you propose a fix, please make it concise.| void Promise.resolve() | ||
| .then(() => params.onBlockReply?.(payload)) | ||
| .then(() => { | ||
| // Record in cross-turn dedup cache only after successful delivery. | ||
| // Recording before send would suppress retries on transient failures. | ||
| if (opts?.sourceText) { | ||
| recordDeliveredText(opts.sourceText, state.recentDeliveredTexts); | ||
| } | ||
| }) | ||
| .catch((err) => { | ||
| log.warn(`block reply callback failed: ${String(err)}`); | ||
| }); |
There was a problem hiding this comment.
Async recording creates a dedup window exactly when cross-turn dedup matters most
recordDeliveredText is called inside the second .then() — it only executes after onBlockReply (the Telegram send) resolves. For Telegram, onBlockReply is a network call and can take hundreds of milliseconds. If context compaction triggers and a new assistant turn begins during that window, isRecentlyDelivered returns false for the in-flight text (it has not been recorded yet), and the duplicate gets through — defeating the entire purpose of the cross-turn dedup.
The scenario this PR is fixing (#37702) is exactly this interleave:
- Turn N emits text → Telegram send starts (async, in-flight)
- Compaction fires → Turn N+1 starts with a fresh
assistantMessageIndex - Turn N+1 re-emits the same text →
isRecentlyDeliveredsees an empty cache → delivers duplicate - Turn N's
.then()finally fires and records — too late
Note the inconsistency within this PR: pushAssistantText (line 193) records synchronously before any delivery, so it does provide protection. Only the emitBlockChunk → emitBlockReplySafely path is affected.
A safer approach is to record the hash synchronously before calling emitBlockReplySafely (accepting the trade-off that a transient delivery failure would suppress the retry, which is less harmful than delivering a visible duplicate to the user):
// In emitBlockChunk, before calling emitBlockReplySafely:
recordDeliveredText(chunk, state.recentDeliveredTexts);
emitBlockReplySafely({ text: cleanedText, ... }); // no sourceText neededAnd remove the async post-delivery recording from emitBlockReplySafely.
Prompt To Fix With AI
This is a comment left during a code review.
Path: src/agents/pi-embedded-subscribe.ts
Line: 114-125
Comment:
**Async recording creates a dedup window exactly when cross-turn dedup matters most**
`recordDeliveredText` is called inside the second `.then()` — it only executes after `onBlockReply` (the Telegram send) resolves. For Telegram, `onBlockReply` is a network call and can take hundreds of milliseconds. If context compaction triggers and a new assistant turn begins during that window, `isRecentlyDelivered` returns `false` for the in-flight text (it has not been recorded yet), and the duplicate gets through — defeating the entire purpose of the cross-turn dedup.
The scenario this PR is fixing (#37702) is exactly this interleave:
1. Turn N emits text → Telegram send starts (async, in-flight)
2. Compaction fires → Turn N+1 starts with a fresh `assistantMessageIndex`
3. Turn N+1 re-emits the same text → `isRecentlyDelivered` sees an empty cache → delivers duplicate
4. Turn N's `.then()` finally fires and records — too late
Note the inconsistency within this PR: `pushAssistantText` (line 193) records **synchronously** before any delivery, so it does provide protection. Only the `emitBlockChunk` → `emitBlockReplySafely` path is affected.
A safer approach is to record the hash **synchronously** before calling `emitBlockReplySafely` (accepting the trade-off that a transient delivery failure would suppress the retry, which is less harmful than delivering a visible duplicate to the user):
```ts
// In emitBlockChunk, before calling emitBlockReplySafely:
recordDeliveredText(chunk, state.recentDeliveredTexts);
emitBlockReplySafely({ text: cleanedText, ... }); // no sourceText needed
```
And remove the async post-delivery recording from `emitBlockReplySafely`.
How can I resolve this? If you propose a fix, please make it concise.There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c1e5ad407d
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| assistantTexts.push(text); | ||
| rememberAssistantText(text); | ||
| // Record in cross-turn cache so post-compaction replays are caught. | ||
| recordDeliveredText(text, state.recentDeliveredTexts); |
There was a problem hiding this comment.
Avoid caching undelivered text during non-streaming retries
Recording recentDeliveredTexts inside pushAssistantText marks text as already delivered even when onBlockReply is not set and nothing has been sent to the channel yet; if auto_compaction_end later arrives with willRetry=true, resetForCompactionRetry() clears assistantTexts but keeps this cache, so replayed text in the retry attempt is filtered by shouldSkipAssistantText() and the final response can be truncated or empty. This regression is specific to non-block-reply flows that hit compaction retry.
Useful? React with 👍 / 👎.
Move recordDeliveredText() from the async post-delivery callback in emitBlockReplySafely to the synchronous path in emitBlockChunk, before the Telegram send. This closes the race window where context compaction could trigger a new assistant turn while the delivery is still in-flight, bypassing the dedup cache entirely. Trade-off: if the send fails transiently the text remains in the cache, but the 1-hour TTL ensures it won't suppress the same content forever. This matches the synchronous recording already done in pushAssistantText. Also fixes the hash comment: Math.imul + >>> 0 produce a 32-bit hash, not 53-bit. Addresses review feedback from greptile-apps.
|
Addressed review feedback (commit 19bceed): 1. Fixed async recording race condition Moved This now matches the synchronous recording already done in Trade-off: if the send fails transiently, the text remains in the cache. The 1-hour TTL ensures it won't suppress the same content forever. 2. Fixed hash comment
|
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 19bceede9f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| // This matches the synchronous recording in pushAssistantText. | ||
| // Trade-off: if the send fails transiently the text stays in the cache, | ||
| // but the 1-hour TTL ensures it won't suppress the same text forever. | ||
| recordDeliveredText(chunk, state.recentDeliveredTexts); |
There was a problem hiding this comment.
Parse directive chunks before marking them delivered
emitBlockChunk now records every chunk in recentDeliveredTexts before it is parsed by replyDirectiveAccumulator.consume(...). When a chunk is directive-only (for example [[reply_to_current]]), nothing is emitted yet, but the hash is cached; if auto-compaction triggers willRetry=true, the retry can skip that directive chunk via cross-turn dedupe, so the accumulator never rebuilds pending reply metadata and subsequent text is emitted without the intended replyTo* target. This regression only needs a compaction retry plus split directive/text chunks to mis-thread replies.
Useful? React with 👍 / 👎.
Problem
shouldSkipAssistantText()inpi-embedded-subscribe.tsonly deduplicates within the sameassistantMessageIndex. After context compaction, the index resets, so when the model re-emits the same text from the compaction summary, it bypasses the dedup check and gets delivered as a new Telegram message (#37702).Solution
Added
recentDeliveredTexts— a bounded rolling cache (max 20 entries, 1h TTL) that persists across assistant message turns within a session. Before delivering text, normalized characters are compared against recently delivered hashes. Matches are skipped.Key design decisions:
recordDeliveredTextis called after successfulemitBlockReplySafelydelivery to avoid suppressing retries on transient failures.Files Changed
src/agents/pi-embedded-helpers/messaging-dedupe.ts— new exports:buildDeliveredTextHash,isRecentlyDelivered,recordDeliveredText,RecentDeliveredEntrysrc/agents/pi-embedded-helpers.ts— re-export new functionssrc/agents/pi-embedded-subscribe.handlers.types.ts— addrecentDeliveredTextsto state typesrc/agents/pi-embedded-subscribe.ts— integrate cross-turn dedup inshouldSkipAssistantText(),pushAssistantText(),emitBlockChunk(),finalizeAssistantTexts(), andemitBlockReplySafely()Tests
Added
src/agents/pi-embedded-helpers/messaging-dedupe.test.tscovering:Related Issues
Closes #37702
Related: #38434, #33308, #33453, #33592, #37697, #30316
Split from #38644 per @steipete request (triage call).