Skip to content

[RFC]: Agent stall detection watchdog (reply-chain monitor) #23867

Description

@kami-saia

Summary

A per-session watchdog that detects when an agent promises action but fails to follow through — says "I'll edit the file" but never invokes the tool. It injects a system nudge to resume the broken reply chain.

Problem

Agents sometimes say they'll do something and then don't. No tool call, no follow-up — just dead air. The user's only recourse is manually sending "continue" to nudge the agent. This is the most common failure mode in multi-turn tool-use conversations.

State Machine

              Agent sends text (non-empty)
DISARMED ─────────────────────────────────────► ARMED (timer starts)
    ▲                                              │
    │  User message / NO_REPLY / empty text        │ Timeout (no activity)
    │  / streaming delta                           ▼
    └────────────────────────────────────────── FIRED (inject nudge)

Transitions

Event → State Notes
Agent sends non-empty text ARMED Timer starts. Agent made a claim — clock is ticking.
Agent sends NO_REPLY / empty DISARMED Explicit sign-off.
Streaming delta received DISARMED Agent is alive and generating.
User message arrives DISARMED User's turn.
Tool activity (result received) Reset timer Work in progress, not stalled.
Timeout expires FIRED Inject system nudge.

Key Behaviors

  • Streaming resets the watchdog. An agent actively generating tokens is not stalled. Deltas disarm; the watchdog only re-arms on the final text.
  • Tool results reset the timer but don't change state. A tool call completing means work is happening — extend the deadline, don't disarm.
  • maxRetries prevents loops. After N nudges with no resolution, stop firing.

Configuration

{
  agents: {
    defaults: {
      watchdog: {
        enabled: true,
        timeoutMs: 30000,
        maxRetries: 3,
        message: "[System] Reply chain broken (stall detected). Resume your task or reply NO_REPLY."
      }
    }
  }
}

Implementation

  • Session-level timer, managed alongside the agent runner
  • Nudge delivered as a system event into the session
  • Nudges appear in transcript for debugging
  • Should respect explicit user "pause"/"stop" signals

Production Evidence

Running for ~2 weeks on a multi-channel setup (Discord + heartbeat + cron). 30s timeout balances false-positive avoidance with responsiveness. maxRetries: 3 is sufficient — if 3 nudges don't work, the agent is truly stuck.

Addresses #11776 (Layer 2: agent-level stall detection).

Metadata

Metadata

Assignees

No one assigned

    Labels

    staleMarked as stale due to inactivity

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions