Skip to content

fix(core): stop repeated tool call loops and stalled streams - #53032

Closed
davtur19 wants to merge 2 commits into
anomalyco:devfrom
davtur19:loop-protection
Closed

davtur19 wants to merge 2 commits into
anomalyco:devfrom
davtur19:loop-protection

Conversation

@davtur19

@davtur19 davtur19 commented Oct 3, 2026 •

Copy link
Copy Markdown

Issue for this PR

Closes #45442

Type of change

  • Bug fix
  • New feature
  • Refactor / code improvement
  • Documentation

What does this PR do?

Two protections for runs that never terminate on their own:

  • Two-strike doom-loop detection: before a tool call runs, the runner compares it against the assistant's tool calls since the last user message. A third identical call (same tool, same input) fires the existing doom_loop permission as a warning; if it repeats once more after the warning, the turn stops with a session error. Below the threshold nothing changes, and an allowing doom_loop rule still executes the call as before.
  • Stall watchdog: when a provider stream goes silent for 90 seconds after its first event, the step is cut off and retried instead of hanging, covering the missing heartbeat timeout called out in the issue. Tool execution time does not count toward the silence window, and time to first event is not treated as a stall.

How did you verify your code works?

bun test in packages/core: 233 tests pass in the touched files (session-runner, session-step, doom-loop), full suite matches vanilla upstream/v2. bun typecheck passes.

Screenshots / recordings

Not a UI change.

Checklist

  • I have tested my changes locally
  • I have not included unrelated changes in this PR

Davide added 2 commits October 4, 2026 00:18
Ports the fork's stall watchdog: a provider stream that goes silent — no events, no error — never fails on its own and hangs the step until the user stops it. The drain races a silence check over the step's stream clock (90 seconds, polled every 5); silence before the first event is a slow time to first token and silence while a tool call is pending is a long local or hosted execution idling the stream by design, and neither counts. A stall fails the attempt with the incomplete-stream classification so recovery reuses the established paths — pre-output stalls retry, post-output stalls continue — instead of the fork's TransientTurnError, bounded by the step-level retry schedule and its renewed phases.
@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Thanks for your contribution!

This PR doesn't have a linked issue. All PRs must reference an existing issue.

Please:

  1. Open an issue describing the bug/feature (if one doesn't exist)
  2. Add Fixes #<number> or Closes #<number> to this PR description

See CONTRIBUTING.md for details.

@github-actions

github-actions Bot commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

The following comment was made by an LLM, it may be inaccurate:

@a88web3

a88web3 commented Oct 28, 2026

Copy link
Copy Markdown

I built an equivalent runner-level guard for a self-hosted 2.x deployment and read this diff closely. The layer is right — a permission-only check has nothing firing it on the 2.x line, so doing this in step.ts is the correct place. Two concrete gaps and one design suggestion, all from the code in this PR.

1. Detection requires three adjacent tool parts, which the common real-world loop shape defeats

const doomLoopDetected = (content, call) => {
  const self = content.findIndex((item) => item.type === "tool" && item.id === call.id)
  if (self < DOOM_LOOP_THRESHOLD - 1) return false
  const input = JSON.stringify(call.input)
  return content
    .slice(self - (DOOM_LOOP_THRESHOLD - 1), self + 1)
    .every((item) => item.type === "tool" && item.state.status !== "streaming" && item.name === call.name && JSON.stringify(item.state.input) === input)
}

.every(item => item.type === "tool" && …) over the slice means the executing call and the two immediately preceding content parts must all be tool parts. A single text part anywhere in that run — a thought, a one-line preamble — breaks it and detection returns false forever.

This is not hypothetical. @agentgoblin already reported the mechanism in #45442 ("the subagent inserts some thoughts between launches … the system does not consider an identical command repeated five times to be something bad"). I hit the same shape in a plain top-level session, not a subagent: a real 2.0.25 run degenerated into 55 identical cycles over ~2h07m, where every assistant turn contained a short text preamble followed by exactly one read with byte-identical arguments. Flattened, the content is text, tool, text, tool, …, so no three adjacent tool parts ever exist and this PR's detector never fires. The loop ran to a manual interrupt.

Suggested change: count identical (name, input) signatures across the turn instead of requiring adjacency — e.g. a Map<signature, count> scoped to the step, incremented before execution, with the same two-strike behaviour on the 3rd and 4th. It catches both shapes, and it is exact rather than window-limited.

2. DOOM_LOOP_MESSAGES = 16 plus a store read on every tool call

gateDoomLoop runs store.messages({ …, limit: 16 }) before each tool execution. Two consequences: the 16-message cap bounds how far back prior strikes are visible in a long turn, and every tool call now costs an extra history read plus a decode-failure branch. A per-step counter is O(1), has no window bound, and survives retries in the same step without re-deriving state from projected history. (I do like that deriving the prior strike from the projected warning marker makes it survive restarts — worth keeping that property if you move to a counter, e.g. by seeding the counter from the marker on the first call.)

3. Add a progress signal, or healthy long turns get stopped

The window is "since the last user message" with no notion of progress. A legitimate autonomous task that iterates on one file — read it, edit it, run tests, read it again to verify — accumulates identical read signatures across the turn with real progress every cycle. Under this PR it gets warned on the 3rd and the turn is stopped on the 4th.

Proposed reset conditions, all already observable inside the runner:

  1. a completed mutating tool call (edit, write, apply_patch);
  2. a bash call whose command matches a recognizable test invocation (pytest, python -m pytest, npm test, bun test, cargo test, go test);
  3. a non-empty workspace file diff for the step — the runner already computes one to publish changed files, so this is free, and it also covers progress made by MCP/plugin tools the guard knows nothing about.

This also resolves the tension @agentgoblin raised about legitimately repeating a command to monitor something: a monitoring loop produces no mutation, no test and no file change, so none of the three resets fire and it is still caught. The reset only forgives cycles that demonstrably changed something.

Verified against a real local model in an isolated harness (2.0.25, self-hosted OpenAI-compatible endpoint, single 27B model):

  • Must not be stopped — read a, read a, write, read a, read a: four identical reads of the same file, all completed, guard never fired, run exited 0. Without a progress reset the third read is blocked.
  • Must be stopped — four identical read-only calls with no mutation, no test and no file change: the 3rd returned a recovery message, the 4th ended the turn, durable history preserved so the session could be resumed.
  • Plus a normal multi-file workflow (3 distinct reads → write → 2 reads) completing with zero guard markers, and a 16-distinct-large-read turn that also triggered a real automatic compaction and continued afterwards.

One smaller note: naming the expected next action in the strike message measurably helped. A bare "change your approach" often produced the same call again; "reuse the prior result and take the next edit/test/verify step" gave the model a way out. Small sample, but the direction was consistent.

Provenance, so nobody over-reads this

All numbers above are from an isolated single-model E2E harness and unit tests, not from production interception — our guard was deployed after the 55-cycle incident, so it has never fired in production and I am not claiming otherwise. The 55-cycle trace is a real 2.x session; its shape (cycle count, per-cycle tool fingerprint, preamble-before-tool ordering, compaction cadence) is what I based gap 1 on.

Happy to send the count-by-signature + progress-reset variant as a commit against loop-protection if that is useful. FWIW the branch currently reports mergeable: false against dev, so it needs a rebase regardless.

@github-actions

github-actions Bot commented Nov 4, 2026

Copy link
Copy Markdown
Contributor

Automated PR Cleanup

Thank you for contributing to opencode.

Due to the high volume of PRs from users and AI agents, we periodically close older PRs using automated criteria so maintainers can focus review time on the most active and community-supported contributions.

This PR was closed because it matched the following cleanup criteria:

  • The PR was created more than 1 month ago
  • The PR had fewer than 2 positive reactions
  • Positive reactions are counted as thumbs-up, heart, celebration, or rocket reactions on the PR

PRs created within the last month are not affected by this cleanup.

If you believe this PR was closed incorrectly, or if you are still actively working on it, please leave a comment explaining why it should be reopened. A maintainer can review and reopen it if appropriate.

Thanks again for taking the time to contribute.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

subagent: infinite loop of identical tool calls for ~50min with no loop protection, uncontrollable token burn

2 participants