Skip to content

fix(imessage): strip U+FFFD garbage chars from echo text key normalization#62191

Closed
maguilar631697 wants to merge 4 commits into
openclaw:mainfrom
maguilar631697:fix/imessage-echo-loop-self-chat
Closed

fix(imessage): strip U+FFFD garbage chars from echo text key normalization#62191
maguilar631697 wants to merge 4 commits into
openclaw:mainfrom
maguilar631697:fix/imessage-echo-loop-self-chat

Conversation

@maguilar631697

@maguilar631697 maguilar631697 commented Apr 6, 2026

Copy link
Copy Markdown

Summary

U+FFFD replacement characters and C0/C1 control characters injected by imsg when extracting text from NSAttributedString (attributedBody column) break echo cache text-key matching, causing duplicate message delivery.

The echo cache stores clean outbound text, but the reflected inbound copy carries garbage character prefixes — normalizeEchoTextKey() only strips \r\n, so the keys never match and the echo slips through as a new inbound message.

Fix: Strip [\ufffd\ufffe\uffff\u0000-\u0008\u000b\u000c\u000e-\u001f\u007f-\u009f]+ before comparison in normalizeEchoTextKey().

Previously included fixes (now upstream)

The original PR included two additional fixes that have since been merged independently by the OpenClaw team (#61619, #63868, #63980, #63989, #64000):

  • TTL 4s → 30s — shipped in 2026.4.10 (SENT_MESSAGE_TEXT_TTL_MS = 30e3)
  • destination_caller_id self-chat detection — shipped in 2026.4.10 with isSelfChat + isAmbiguousSelfThread

This PR now contains only the remaining unmerged fix.

Changes

  • extensions/imessage/src/monitor/echo-cache.ts: Strip U+FFFD/control chars in normalizeEchoTextKey

Related Issues

Fixes #61312, #61821

Test Plan

  • Confirmed echo cache miss on messages with U+FFFD prefixes (unpatched)
  • Applied patch to compiled dist — garbage-prefixed echoes now correctly matched and suppressed
  • Clean text (no U+FFFD) still matches normally — no regression

@openclaw-barnacle openclaw-barnacle Bot added channel: imessage Channel integration: imessage size: XS labels Apr 6, 2026
@greptile-apps

greptile-apps Bot commented Apr 6, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes three genuine iMessage echo-loop bugs (wrong isSelfChat classification, short 4 s TTL, and U+FFFD mismatch in the echo cache). The TTL increase and the normalizeEchoTextKey regex change are correct. However, two regressions block the primary fix from working.

  • destination_caller_id never declared in IMessagePayload (extensions/imessage/src/monitor/types.ts): inbound-processing.ts line 218 reads params.message.destination_caller_id, but the type has no such field. TypeScript will fail to compile, and at runtime the value is always undefined, so !destCallerId is always true — collapsing the new self-chat guard back to the old broken senderMatchesChatId-only check.
  • deliver.ts scope double-adds "imessage:" prefix: target for a DM is already "imessage:<sender>" (from buildIMessageInboundContext), so the new \${accountId}:imessage:${target}`produces"accountId:imessage:imessage:", which never matches buildIMessageEchoScope's "accountId:imessage:". For groups the mismatch is identical. The existing deliver.test.ts` assertion (line 148) catches this. The original one-liner was correct.

Confidence Score: 3/5

Not safe to merge — two P1 regressions leave the echo-loop fix non-functional and break the TypeScript build.

The missing destination_caller_id field causes a compile failure and a silent runtime no-op that neutralises the isSelfChat fix. The scope double-prefix in deliver.ts defeats all echo-cache deduplication after merge. Both defects are on the directly changed path and must be fixed before the PR can ship.

extensions/imessage/src/monitor/types.ts (missing destination_caller_id declaration) and extensions/imessage/src/monitor/deliver.ts (incorrect scope prefix)

Comments Outside Diff (1)

  1. extensions/imessage/src/monitor/types.ts, line 10-27 (link)

    P1 destination_caller_id missing from IMessagePayload

    inbound-processing.ts reads params.message.destination_caller_id (line 218 of the changed file), but IMessagePayload has no such field. TypeScript will reject this at compile time, and even if the build somehow passed, the runtime value would always be undefined — making !destCallerId unconditionally true and collapsing the new guard back to the old broken senderMatchesChatId-only check. The field must be declared here for Bug 1's fix to have any effect.

    Prompt To Fix With AI
    This is a comment left during a code review.
    Path: extensions/imessage/src/monitor/types.ts
    Line: 10-27
    
    Comment:
    **`destination_caller_id` missing from `IMessagePayload`**
    
    `inbound-processing.ts` reads `params.message.destination_caller_id` (line 218 of the changed file), but `IMessagePayload` has no such field. TypeScript will reject this at compile time, and even if the build somehow passed, the runtime value would always be `undefined` — making `!destCallerId` unconditionally `true` and collapsing the new guard back to the old broken `senderMatchesChatId`-only check. The field must be declared here for Bug 1's fix to have any effect.
    
    
    
    How can I resolve this? If you propose a fix, please make it concise.
Prompt To Fix All With AI
This is a comment left during a code review.
Path: extensions/imessage/src/monitor/types.ts
Line: 10-27

Comment:
**`destination_caller_id` missing from `IMessagePayload`**

`inbound-processing.ts` reads `params.message.destination_caller_id` (line 218 of the changed file), but `IMessagePayload` has no such field. TypeScript will reject this at compile time, and even if the build somehow passed, the runtime value would always be `undefined` — making `!destCallerId` unconditionally `true` and collapsing the new guard back to the old broken `senderMatchesChatId`-only check. The field must be declared here for Bug 1's fix to have any effect.

```suggestion
export type IMessagePayload = {
  id?: number | null;
  guid?: string | null;
  chat_id?: number | null;
  sender?: string | null;
  is_from_me?: boolean | null;
  text?: string | null;
  reply_to_id?: number | string | null;
  reply_to_text?: string | null;
  reply_to_sender?: string | null;
  created_at?: string | null;
  attachments?: IMessageAttachment[] | null;
  chat_identifier?: string | null;
  chat_guid?: string | null;
  chat_name?: string | null;
  participants?: string[] | null;
  is_group?: boolean | null;
  destination_caller_id?: string | null;
};
```

How can I resolve this? If you propose a fix, please make it concise.

---

This is a comment left during a code review.
Path: extensions/imessage/src/monitor/deliver.ts
Line: 31-34

Comment:
**Echo-cache scope double-adds `"imessage:"` prefix**

For a DM, `target` is already `"imessage:<sender>"` (set by `buildIMessageInboundContext` as `` imessageTo = `imessage:${decision.sender}` ``). Prepending another `"imessage:"` produces scope `"accountId:imessage:imessage:<sender>"`, which never matches the key `buildIMessageEchoScope` returns (`"accountId:imessage:<sender>"`). For groups `target` is `"chat_id:<n>"` and the new scope `"accountId:imessage:chat_id:<n>"` likewise never matches `"accountId:chat_id:<n>"`. The existing test (`deliver.test.ts` line 148) already asserts the correct format — `"acct-3:chat_id:30"` for `target: "chat_id:30"` — and would catch this. The original `scope = \`${accountId ?? ""}:${target}\`` was already correct; the added prefix silently breaks echo deduplication for every conversation.

```suggestion
  const scope = `${accountId ?? ""}:${target}`;
```

How can I resolve this? If you propose a fix, please make it concise.

Reviews (1): Last reviewed commit: "fix(imessage): prevent echo loop from se..." | Re-trigger Greptile

Comment thread extensions/imessage/src/monitor/deliver.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d07d423d42

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread extensions/imessage/src/monitor/deliver.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: b565954865

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread extensions/imessage/src/monitor/inbound-processing.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 93c8eac0d1

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread extensions/imessage/src/monitor/echo-cache.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c4b5e9e06a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread extensions/imessage/src/monitor/echo-cache.ts Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 04b7cd501a

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread extensions/imessage/src/monitor/echo-cache.ts Outdated
@maguilar631697

Copy link
Copy Markdown
Author

Re: P2 on TTL increase — acknowledged tradeoff. The 30s window does widen the false-positive path for text-only matching on short repeated messages like "ok".

However, the 4s TTL causes echo loops that flood users with 2-10+ duplicate replies — a much worse failure mode. The text-only matching path only fires when there's no usable message ID (normalized to null/ok/unknown), which is uncommon in normal flow.

A follow-up could add a minimum text length threshold to skip text-only matching for very short strings, but keeping that separate from this fix.

@maguilar631697
maguilar631697 force-pushed the fix/imessage-echo-loop-self-chat branch from 04b7cd5 to 30b0af3 Compare April 11, 2026 14:20
@maguilar631697 maguilar631697 changed the title fix(imessage): prevent echo loop from self-chat false positive, short TTL, and U+FFFD mismatch fix(imessage): increase echo TTL to 30s and strip U+FFFD garbage from echo text keys Apr 11, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 30b0af3990

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread extensions/imessage/src/monitor/echo-cache.ts Outdated
@maguilar631697
maguilar631697 force-pushed the fix/imessage-echo-loop-self-chat branch from 30b0af3 to 8d5bd51 Compare April 11, 2026 14:28
@maguilar631697 maguilar631697 changed the title fix(imessage): increase echo TTL to 30s and strip U+FFFD garbage from echo text keys fix(imessage): strip U+FFFD and control chars from echo text keys Apr 11, 2026
@maguilar631697
maguilar631697 force-pushed the fix/imessage-echo-loop-self-chat branch from 8d5bd51 to 782f8fa Compare April 11, 2026 14:48
@maguilar631697 maguilar631697 changed the title fix(imessage): strip U+FFFD and control chars from echo text keys fix(imessage): strip U+FFFD garbage chars from echo text key normalization Apr 11, 2026
@maguilar631697

Copy link
Copy Markdown
Author

Rebased — review comments no longer applicable

This PR has been rebased to only include the remaining unmerged fix: U+FFFD garbage char normalization in normalizeEchoTextKey() (echo-cache.ts).

The previous fixes for TTL (4s → 30s) and destination_caller_id self-chat detection have been removed from this PR — they were independently merged upstream in #61619, #63868, #63980, #63989, #64000 and ship in 2026.4.10.

The three P1 review comments from Greptile and ChatGPT Codex all reference files (deliver.ts, inbound-processing.ts) that are no longer in the diff:

  • deliver.ts scope double-prefix — file no longer changed
  • inbound-processing.ts null guard on chatIdentifier — file no longer changed

The only change remaining is a one-line regex addition in echo-cache.ts to strip U+FFFD/C0/C1 control characters before echo text-key comparison. See #61312, #61821.

cryptoassest and others added 4 commits April 28, 2026 18:42
…ation

imsg extracts message text from the SQLite attributedBody column via
NSAttributedString, which injects U+FFFD replacement characters and C0/C1
control characters around rich-text spans. The echo cache stores clean
outbound text, but the reflected copy carries these garbage characters,
defeating text-based deduplication. See: openclaw#61312, openclaw#61821.

Fixes openclaw#61312, openclaw#61821
@vincentkoc
vincentkoc force-pushed the fix/imessage-echo-loop-self-chat branch from bce01df to eec4bb7 Compare April 28, 2026 18:42
@vincentkoc

Copy link
Copy Markdown
Member

ProjectClownfish pushed a narrow repair to this branch so the original contributor path can stay canonical.

Source PR: #62191
Validation: pnpm test extensions/imessage/src/monitor/monitor-provider.echo-cache.test.ts; pnpm check:changed
Contributor credit is preserved in the branch history and PR context.

@clawsweeper

clawsweeper Bot commented Apr 28, 2026

Copy link
Copy Markdown
Contributor

Codex review: keeping this open for maintainer follow-up; there is still a little grit to resolve.

Current main and the latest release still lack the requested iMessage echo-cache normalization for leading U+FFFD/control-byte corruption, so this PR is not obsolete. The PR remains a relevant implementation candidate, but the provided current diff appears to need a regex syntax repair before it can land safely.

Best possible solution:

Keep this PR open for maintainer follow-up. The best path is to repair the leading-only regex using syntax-safe U+FFFD/U+FFFE/U+FFFF and C0/C1 ranges, keep the interior-control preservation tests, coordinate the overlap with #63581, and then land this or an equivalent iMessage plugin fix in extensions/imessage/src/monitor/echo-cache.ts.

What I checked:

Likely related people:

  • vincentkoc: Current main's changelog credits @vincentkoc on the earlier iMessage self-chat echo dedupe work in fix(imessage): dedupe reflected self-chat duplicates #38440, and the PR discussion says vincentkoc pushed the latest narrow repair to this branch. (role: feature-history owner and recent PR maintainer; confidence: high; commits: eec4bb724d08, #38440; files: extensions/imessage/src/monitor/echo-cache.ts, extensions/imessage/src/monitor/self-chat-dedupe.test.ts, CHANGELOG.md)
  • neeravmakwana: Merged iMessage self-chat and destination_caller_id fixes fix(imessage): avoid DM self-chat false positives #61619 and fix(imessage): require destination_caller_id for self-chat classification (#63980) #63989 are directly adjacent to the echo-cache path and are credited in the current changelog; current main source still carries that destination_caller_id behavior. (role: adjacent merged-feature contributor; confidence: medium; commits: #61619, #63989; files: extensions/imessage/src/monitor/inbound-processing.ts, extensions/imessage/src/monitor/types.ts, extensions/imessage/src/monitor/self-chat-dedupe.test.ts)
  • Peter Steinberger [email protected]: Local blame/log on the available current-main history attributes the current echo-cache.ts snapshot and latest release changelog consolidation to Peter Steinberger, making him a plausible maintainer routing fallback for this area. (role: recent maintainer on current main snapshot; confidence: low; commits: 3aac8e650c50, be8c24633aaa; files: extensions/imessage/src/monitor/echo-cache.ts, extensions/imessage/src/monitor/monitor-provider.echo-cache.test.ts, CHANGELOG.md)

Remaining risk / open question:

  • The provided current PR diff appears to contain an invalid JavaScript regex range; this should be repaired before maintainer landing or CI will likely fail at parse time.
  • The PR overlaps imessage: strip NUL bytes from echo-cache text normalization #63581 around echo-cache prefix cleanup, so maintainers should combine or coordinate the U+FFFD/control-prefix and NUL-prefix handling to avoid competing normalizers.
  • No tests were run because this was a required read-only review; the decision is based on source, tag, discussion, PR-diff, and read-only Node syntax inspection.

Codex review notes: model gpt-5.5, reasoning high; reviewed against 955b4df0939f.

@openclaw-clownfish

Copy link
Copy Markdown
Contributor

ProjectClownfish could not safely update this branch, so it opened a narrow replacement PR instead.

Replacement PR: #73891
Source PR: #62191
Contributor credit is preserved in the replacement PR body and changelog plan.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

channel: imessage Channel integration: imessage clawsweeper Tracked by ClawSweeper automation size: XS

Projects

None yet

Development

Successfully merging this pull request may close these issues.

iMessage echo loop in self-chat: U+FFFD from attributedBody breaks echo cache text matching

3 participants