Skip to content

Bug: genericRepeat critical/circuit-breaker never fires when exec results vary slightly #93917

Description

@zj0001

Bug: genericRepeat critical/circuit-breaker never fires when exec results vary slightly

Summary

The genericRepeat detector in tool-loop-detection.ts has two detection paths:

  1. Warning path (line ~418): uses recentCount — counts all calls with identical args, regardless of result. ✅ Works correctly.
  2. Critical path (line ~407): uses noProgressStreak — requires identical result hash across consecutive calls. ❌ Fails for exec commands.
  3. Global circuit breaker (line ~348): also uses noProgressStreak. ❌ Same failure mode.

Root cause

getNoProgressStreak() (the function used by critical and circuit-breaker) requires the result hash to be identical across consecutive calls. For exec tool calls, hashExecToolOutcome() includes the full output text in the hash. When a command fails repeatedly (e.g., SSH timeout, docker connection refused), the error output often varies slightly — different timestamps, different error messages, different connection states — causing the result hash to differ each time.

This means noProgressStreak is always 1, never reaching the critical threshold (default 20) or circuit breaker (default 30), even after hundreds of identical failed calls.

Real-world evidence

From a production OpenClaw session (aa7aef13), 1252 total exec calls were made in one session. The top repeated commands:

Count Command
430x sleep 3 && docker exec alist /opt/alist/alist admin set admin123
164x sleep 8 && ssh -q ... "curl -s http://127.0.0.1:9233/json/version"
132x NapCat send message via curl
131x Cleanup temp files
57x Python Telethon script

The loop detection config was set aggressively (warningThreshold: 5, criticalThreshold: 10, globalCircuitBreakerThreshold: 15), yet none of the critical/breaker thresholds fired. The warning path did fire but the model (DeepSeek v4 Pro) ignored the text warning.

This eventually led to thread pool exhaustion and a Gateway restart — the exact failure mode loop detection was designed to prevent.

Proposed fix

For exec/bash tools where output is inherently variable, the critical path should not require identical result hashes. Options:

Option A: For exec tools, use recentCount (same as warning path) for critical/breaker thresholds too, since "same command called N times" is itself the danger signal regardless of output variation.

Option B: Make hashExecToolOutcome() more lenient — hash only {status, exitCode, timedOut} without the full output text, so that "failed with exit code 1" always produces the same hash regardless of error message variation.

Option C: Add a separate execRepeatThreshold that uses recentCount and bypasses the result-hash requirement entirely for shell commands.

Environment

  • OpenClaw version: 2026.6.6
  • Config: tools.loopDetection.enabled: true, warningThreshold: 5, criticalThreshold: 10, globalCircuitBreakerThreshold: 15
  • Model: xopdeepseekv4pro via headroom provider

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:crash-loopCrash, hang, restart loop, or process-level availability failure.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions