Skip to content

fix(agents): block message-tool spam loops defeated by volatile message ids#89109

Merged
steipete merged 3 commits into
openclaw:mainfrom
openperf:fix/89090-loopdetect-message-volatile-id
Jun 7, 2026
Merged

fix(agents): block message-tool spam loops defeated by volatile message ids#89109
steipete merged 3 commits into
openclaw:mainfrom
openperf:fix/89090-loopdetect-message-volatile-id

Conversation

@openperf

@openperf openperf commented Jun 1, 2026

Copy link
Copy Markdown
Member

Summary

  • Problem: Issue Bug: loopDetection cannot block message tool loops — volatile messageId in result defeats all critical-level detection paths #89090 reports that loopDetection detects repeated message.send calls (ping_pong, warning level) but can never escalate to a block. A model stuck in a tool-call loop keeps sending duplicate messages — 30+ duplicates reported — until runRetries.max (default 160) is exhausted; the user only sees a flood of identical messages and the warning the model ignores.
  • Root Cause: In tool-loop-detection.ts, hashToolOutcome hashes the full result for the send tools. A successful send returns through jsonResult(payload), so the per-call delivery ids land in both the result details (e.g. messageId, receipt, the gateway runId) and the result text (the stringified payload). The hash therefore differs on every call. All three critical-block paths — the global circuit breaker, ping_pong critical, and generic no-progress — require consecutive identical resultHash values (getNoProgressStreak / noProgressEvidence), which is structurally impossible for a tool whose result carries a unique id. Detection fires only at warning level because it also keys on argsHash, which is stable.
  • Fix: Hash a stable, id-stripped projection of the result, scoped to delivery results only — sessions_send and the outbound-message actions (send, broadcast, reply, thread-reply, sendWithEffect, sendAttachment, upload-file, sticker, poll) of the multi-action message tool.
    • isVolatileSendResult gates the path: sessions_send and the outbound-message actions above on the multi-action message tool get the stripped hash, while message's mutate/query actions (read, search, edit, react, unsend, list-pins, thread-list, …) keep full-result hashing because their ids/timestamps are real progress. Provider-docked send tools (channel plugins registered as named tools, e.g. telegram/discord) return the same volatile-id shape and are matched via the shared isMessagingToolSendAction detector, so loops on them escalate too; SEND_LIKE_MESSAGE_ACTIONS stays intentionally broader than that helper's terminal-send set (broadcast/reply/sticker/poll are volatile but not terminal sends).
    • stripVolatileSendIds recursively removes VOLATILE_SEND_RESULT_KEYS (messageId, message_id, messageIds, platformMessageId, platformMessageIds, fileId, fileKey, pollId, receipt, runId, idempotencyKey, ts, timestamp, sentAt, deliveredAt, createdAt) from details at any depth — so it also neutralizes the nested ids in a broadcast payload (results[].result.messageId).
    • It also drops a message object's own bare id (detected via isMessageDeliveryObject: a string id + text plus conversation/direction/senderId/accountId) so plugin-shaped delivery results normalize too, while a stable nested route/conversation.id is preserved.
    • The stringified text is not hashed for delivery results (it is the payload restated and carries the same volatile ids); the stable delivery facts (ok / status / channel / target) remain.
    • The loop detector's own veto result (buildBlockedToolResult with deniedReason: "tool-loop") gets no result hash, so a critical block stays sticky instead of the block resetting the streak. Other blocked results (plugin/approval vetoes) keep their hash, so repeated identical denials still escalate normally.
    • This runs at the shared hashToolOutcome layer, so it covers the message tool, sessions_send, and provider-docked channel tools uniformly (direct, gateway, and plugin-handled send results). Repeated identical-args sends now produce a stable resultHash, so the no-progress / critical paths fire and the loop is blocked.
    • Why this is safe (no false positives): no-progress requires BOTH argsHash and resultHash to repeat. The args hash already carries the message content and target, so two genuinely different messages have different argsHash and are never flagged — dropping the volatile ids only restores detection for true same-args resends.
  • What changed:
    • src/agents/tool-loop-detection.ts — add SEND_LIKE_MESSAGE_ACTIONS, VOLATILE_SEND_RESULT_KEYS, stripVolatileSendIds, isVolatileSendResult, isMessageDeliveryObject, and isLoopVetoResult; branch delivery results to the id-stripped hash inside hashToolOutcome and give blocked/veto results no hash. isVolatileSendResult delegates provider-docked tool names to the shared isMessagingToolSendAction (from embedded-agent-messaging.ts), unifying the send-action source of truth; the denylist also covers platformMessageId/platformMessageIds.
    • src/agents/tool-loop-detection.test.ts — fifteen regression cases on the real jsonResult result shape (stable hash across ids in details+text, nested broadcast ids, escalation to critical for message send, sibling reply and upload-file (fresh file id) actions, sessions_send, and a provider-docked telegram send, no false-positive on distinct messages, a non-send message read whose id/timestamp progress must stay distinct, a stable nested route id that must survive the strip, and a critical block that stays sticky after the veto result is recorded).
  • What did NOT change (scope boundary):
    • Generic / exec / process / poll detector semantics, non-send message action hashing, thresholds, and retry caps are untouched — nothing was loosened or raised; this restores the existing block behavior for delivery results only.
    • No config surface (no schema, defaults, doctor migrations, or docs/reference/config).
    • No plugin surface (no plugin SDK, manifest, extensions/* api/runtime-api, registry/loader).
    • Gateway protocol unchanged.

Reproduction

  1. Configure an agent with a model prone to tool-call loops on a channel that returns a per-send message id (e.g. Feishu).
  2. The model loops on message.send with identical arguments.
  3. Before this PR: ping_pong fires at warning level but never reaches critical/block; the user receives 30+ duplicate messages until run-retries are exhausted.
  4. After this PR: the repeated identical send is recognized as no-progress and escalates to a critical block.

Deterministic source reproduction (added as regression tests): record repeated sends with identical args but changing delivery ids in the result, then call detectToolCallLoop — before the fix it can never reach a critical branch.

Real behavior proof

Behavior addressed (#89090 — send loops can now be blocked, not just warned): repeated identical message.send (and sessions_send) escalates to a critical block instead of spamming until run-retry exhaustion.

Real environment tested (Linux, Node — Vitest against the production loop-detection module using the real jsonResult result shape; no live channel run): verified via tool-loop-detection.test.ts, plus core production type-check and a scoped test type-check over the two changed files, plus format check. A live multi-channel gateway loop was not run.

Exact steps or command run after this patch (repo-root): node scripts/run-vitest.mjs src/agents/tool-loop-detection.test.ts; node scripts/run-tsgo.mjs -p tsconfig.core.json (core types) and a scoped tsgo over the changed production + test files (test types); oxfmt --check on the changed files.

Evidence after fix (Vitest output for the touched test file):

 tool-loop-detection.test.ts   Tests  58 passed (58)

Fifteen new cases pass, all built on the real jsonResult result shape (volatile id present in both details and the stringified text): a stable result hash across two sends whose ids differ; stable hash across broadcast results with per-call nested ids; escalation to critical for message send, reply, upload-file, sessions_send, and a provider-docked telegram send despite every id differing; a plugin-shaped result whose message object carries a bare per-send id; no stuck result when the sends carry distinct content (different args); a non-send message read whose results differ only by id/timestamp staying distinct (full hashing preserved); a stable nested route id that survives the strip; a changing conversation.id that stays below critical; and a critical block that stays sticky after the veto result is recorded.

Observed result after fix (identical-args send loops reach critical/block): detectToolCallLoop reports stuck at critical for repeated identical sends, while distinct-content sends remain unflagged.

What was not tested (live channel/gateway runaway loop): a real Feishu/gateway loop session was not reproduced live; the proof is the deterministic module-level reproduction the issue itself describes.

Repro confirmation: with the production delivery branch removed on an otherwise-identical tree, the four send behavior cases fail (stable-hash, nested-broadcast, and both escalation cases); and when the branch is instead broadened to every message action (dropping the params.action gate), the non-send read case fails — so both the fix and its action-scoping are covered by tests that fail on the shape the runtime actually produces rather than passing vacuously. All cases pass with the scoped fix.

Risk / Mitigation

  • Risk: over-stripping result fields could collide distinct outcomes. Mitigation: only a fixed set of volatile delivery-id keys is removed (recursively); stable facts (status, channel, target, action) and the argsHash gate remain, so distinct messages stay distinguishable (covered by the no-false-positive regression test).
  • Risk: behavior change could surface previously-hidden blocks. Mitigation: the change is scoped to the send family and merely lets the existing critical/no-progress thresholds apply as designed; no threshold, cap, or other detector/tool is altered.

Change Type (select all)

  • Bug fix

Scope (select all touched areas)

  • Agents / tools

Linked Issue/PR

Fixes #89090

@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: S labels Jun 1, 2026
@clawsweeper

clawsweeper Bot commented Jun 1, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs maintainer review before merge. Reviewed June 5, 2026, 2:13 AM ET / 06:13 UTC.

Summary
The PR changes agent tool-loop result hashing to ignore volatile delivery IDs for send-like message outcomes and adds regression tests for repeated send escalation, non-send preservation, provider-docked sends, and loop-veto stickiness.

PR surface: Source +99, Tests +339. Total +438 across 2 files.

Reproducibility: yes. Current main hashes volatile send IDs in the resultHash while the critical blockers require identical resultHash values, and the PR adds deterministic regression cases for that exact source-level failure path.

Review metrics: none identified.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🐚 platinum hermit
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

Risk before merge

  • [P1] A sibling PR for the same reported bug remains open with broader Gateway/qa-channel runtime proof, so merging both branches could create duplicate hash-normalization paths or review drift.
  • [P1] The volatile-field denylist directly controls user-visible message blocking: missing a volatile delivery field can let spam continue, while over-stripping can block legitimate repeated sends with materially different delivery outcomes.
  • [P1] This branch reports deterministic module-level proof but not a live Gateway/channel runaway-loop proof, leaving some user-visible message-delivery behavior to maintainer judgment before landing.

Maintainer options:

  1. Choose one canonical fix branch (recommended)
    Compare this PR with fix(agents): stabilize message send loop detection #89143 and land only the branch whose hashing scope, veto handling, and runtime proof maintainers want to own.
  2. Keep this branch and add runtime proof
    If maintainers prefer this implementation, add or link a Gateway/qa-channel proof that repeated sends stop after the critical threshold without suppressing distinct sends.
  3. Pause this duplicate path
    If the sibling PR is chosen as canonical, leave this PR unmerged and close it only after the selected fix lands or clearly supersedes it.

Next step before merge

  • [P2] Maintainers need to choose the canonical fix path for this MEMBER-authored PR and its sibling rather than queueing an automated repair.

Security
Cleared: The diff only changes agent loop-detection code and tests; it does not add dependencies, scripts, workflow permissions, package metadata, credential handling, or artifact downloads.

Review details

Best possible solution:

Land exactly one canonical volatile-send hashing fix for the shared loop detector, ideally with Gateway/qa-channel or equivalent runtime proof, then close the linked bug and close or supersede the duplicate PR.

Do we have a high-confidence way to reproduce the issue?

Yes. Current main hashes volatile send IDs in the resultHash while the critical blockers require identical resultHash values, and the PR adds deterministic regression cases for that exact source-level failure path.

Is this the best way to solve the issue?

Yes, fixing the shared outcome hash layer is a maintainable solution for this bug, but this branch is not clearly the sole best landing path while the sibling PR with broader runtime proof remains open.

AGENTS.md: found and applied where relevant.

Codex review notes: model gpt-5.5, reasoning high; reviewed against d7759c6a35ac.

Label changes

Label justifications:

  • P1: The PR targets a real agent/channel spam loop where duplicate outbound messages can continue until the retry cap is exhausted.
  • merge-risk: 🚨 message-delivery: The changed result-hash projection decides when repeated visible message sends are blocked, so an incorrect scope can either allow duplicate delivery or suppress legitimate sends.
  • rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🐚 platinum hermit and patch quality is 🐚 platinum hermit.
  • status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Not applicable: The external-contributor proof gate does not apply to this MEMBER PR; the body still supplies deterministic module-level proof and explicitly notes that no live channel/gateway runaway loop was run.
Evidence reviewed

PR surface:

Source +99, Tests +339. Total +438 across 2 files.

View PR surface stats
Area Files Added Removed Net
Source 1 99 0 +99
Tests 1 340 1 +339
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 2 439 1 +438

What I checked:

Likely related people:

  • steipete: Blame and GitHub history show repeated recent work across tool-loop detection, embedded messaging helpers, and agent runtime refactors that shape this behavior. (role: recent area contributor; confidence: high; commits: 86872e088006, bf1634b17afe, dbfe5a252cd2; files: src/agents/tool-loop-detection.ts, src/agents/embedded-agent-messaging.ts, src/agents/agent-tools.before-tool-call.ts)
  • efpiva: Recent history includes a focused fix to loop-detection observation and post-compaction guard behavior, which is adjacent to the no-progress outcome tracking this PR changes. (role: adjacent loop-detection contributor; confidence: medium; commits: 4c4825679b69; files: src/agents/tool-loop-detection.ts, src/agents/agent-tools.before-tool-call.ts)
  • vincentkoc: Recent before-tool-call and policy work touched the same hook path that records tool outcomes and blocks critical tool loops. (role: adjacent before-tool-call contributor; confidence: medium; commits: be967545c500, 8bff73cfb07f; files: src/agents/agent-tools.before-tool-call.ts, src/agents/tool-loop-detection.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. P1 High-priority user-facing bug, regression, or broken workflow. labels Jun 1, 2026
@openperf
openperf force-pushed the fix/89090-loopdetect-message-volatile-id branch from 7b0ec30 to f4a29c9 Compare June 1, 2026 14:38
@clawsweeper clawsweeper Bot added the merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. label Jun 1, 2026
@openperf
openperf force-pushed the fix/89090-loopdetect-message-volatile-id branch 4 times, most recently from ede8650 to 59b2c14 Compare June 1, 2026 16:31
@clawsweeper clawsweeper Bot added status: 🛠️ actively grinding The PR author has acted after the latest ClawSweeper review and work remains. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 🛠️ actively grinding The PR author has acted after the latest ClawSweeper review and work remains. labels Jun 1, 2026
@openperf
openperf force-pushed the fix/89090-loopdetect-message-volatile-id branch from 59b2c14 to 93816f1 Compare June 1, 2026 16:54
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jun 1, 2026
@openperf
openperf force-pushed the fix/89090-loopdetect-message-volatile-id branch 2 times, most recently from 8913738 to fdd7479 Compare June 2, 2026 01:11
@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. labels Jun 2, 2026
@openperf
openperf force-pushed the fix/89090-loopdetect-message-volatile-id branch from fdd7479 to 428e585 Compare June 2, 2026 02:07
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jun 2, 2026
@openperf
openperf force-pushed the fix/89090-loopdetect-message-volatile-id branch from 428e585 to 6dd37c8 Compare June 2, 2026 02:51
@clawsweeper clawsweeper Bot added rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. and removed rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. labels Jun 2, 2026
@openperf
openperf force-pushed the fix/89090-loopdetect-message-volatile-id branch 2 times, most recently from d2399da to 99b9c0c Compare June 2, 2026 04:52
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR. and removed rating: 🦐 gold shrimp Decent PR readiness signal, but merge confidence is limited. status: ⏳ waiting on author ClawSweeper has contributor-facing work open and is waiting for author action. labels Jun 2, 2026
@openperf
openperf force-pushed the fix/89090-loopdetect-message-volatile-id branch from 7456425 to 1f4246e Compare June 5, 2026 06:06
@steipete steipete self-assigned this Jun 7, 2026
openperf and others added 3 commits June 7, 2026 08:14
loopDetection only warned on repeated message.send calls and could never
escalate to a block: each send returns a fresh messageId, so the full
result hash differed every call and the no-progress / critical paths
(global circuit breaker, ping_pong critical, generic no-progress) never
fired. A looping model could spam identical messages until runRetries.max.

Hash message-tool outcomes from stable delivery facts only, dropping
volatile id keys, mirroring the existing exec / process stable-hash cases.
The args hash already separates genuinely distinct sends, so this cannot
mask a different message.

Fixes openclaw#89090
@steipete
steipete force-pushed the fix/89090-loopdetect-message-volatile-id branch from 1f4246e to 7b65fee Compare June 7, 2026 07:19
@steipete

steipete commented Jun 7, 2026

Copy link
Copy Markdown
Contributor

Land-ready after maintainer fixup.

What changed after review:

  • Added src/agents/tools/message-tool.test.ts coverage that drives the real wrapped message tool through repeated qa-channel send results with volatile message object ids/timestamps.
  • Verified the wrapper blocks before dispatching another visible send once loop detection reaches the critical threshold, and stays blocked on the next attempt.

Local proof:

  • pnpm exec oxfmt --write src/agents/tools/message-tool.test.ts
  • git diff --check
  • node scripts/run-vitest.mjs src/agents/tool-loop-detection.test.ts src/agents/tools/message-tool.test.ts -> 2 files, 157 tests passed
  • .agents/skills/autoreview/scripts/autoreview --mode local --parallel-tests "node scripts/run-vitest.mjs src/agents/tool-loop-detection.test.ts src/agents/tools/message-tool.test.ts" -> clean, tests passed
  • .agents/skills/autoreview/scripts/autoreview --mode branch --base origin/main --parallel-tests "node scripts/run-vitest.mjs src/agents/tool-loop-detection.test.ts src/agents/tools/message-tool.test.ts" -> clean, tests passed

GitHub proof on 7b65fee261b157fb0c66b8ba299e9900849e4b91:

Known proof gaps: no additional live Feishu/Lark/Telegram send was run in this session; the PR has hash-layer coverage, wrapped message-tool runtime coverage, and the repository real-behavior proof check is green.

@steipete
steipete merged commit 697d2d0 into openclaw:main Jun 7, 2026
168 of 170 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. P1 High-priority user-facing bug, regression, or broken workflow. rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. size: L status: 👀 ready for maintainer look ClawSweeper has no concrete contributor-facing blocker left for this PR.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: loopDetection cannot block message tool loops — volatile messageId in result defeats all critical-level detection paths

2 participants