Skip to content

Gateway watchdog fails to kill stuck model calls; stall recovery is none #94650

Description

@ArisMontclair

Problem

A model call to the Xiaomi provider hung indefinitely (149s observed). The gateway's built-in ~120s idle/stream watchdog did NOT fire. The diagnostic system detected the stall but had recovery=none, meaning it logged the problem every 30 seconds without taking action.

Impact

Total downtime: ~44 minutes (18:02 to 18:46). The 149s model hang was just the start. Because the gateway couldn't kill the stuck run:

  1. Messages queued behind it (queueDepth=2, then 12 after restarts)
  2. Manual gateway restarts replayed all queued messages
  3. Each replay created duplicate agent turns that got stuck or looped
  4. Required clearing the session transcript and multiple restarts to recover

Expected Behavior

  1. The ~120s watchdog should kill model calls that produce no response within the window, regardless of stream state
  2. The stall diagnostic with active_work_without_progress should have an automatic recovery option (abort the stuck run)
  3. Gateway restarts should NOT replay messages that were already processed or are duplicates

Logs

18:08:28 [diagnostic] stalled session: lastProgress=embedded_run:started lastProgressAge=149s recovery=none
18:08:58 [diagnostic] stalled session: lastProgressAge=179s recovery=none
18:09:28 [diagnostic] stalled session: lastProgressAge=209s recovery=none
18:09:58 [diagnostic] stalled session: lastProgressAge=239s recovery=none

Gateway kept diagnosing the stall every 30 seconds for 4+ minutes without killing it.

Environment

Config

timeoutSeconds was NOT SET on the provider. The implicit ~120s watchdog should have killed the request. It didn't.

Request

  1. Fix the watchdog to actually fire on stuck model calls
  2. Add automatic abort to the stall diagnostic when recovery=none becomes recovery=abort
  3. Add message dedup on gateway restart to prevent replay storms

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:linked-pr-openClawSweeper found an open linked pull request for this issue.clawsweeper:needs-live-reproClawSweeper needs live local, crabbox, or manual validation to confirm this issue.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.impact:auth-providerAuth, provider routing, model choice, or SecretRef resolution may break.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🐚 platinum hermitGood issue quality with a plausible reproduction path needing some confirmation.staleMarked as stale due to inactivity

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions