Skip to content

Diagnostic recovery should not be suppressed by stuck-session warning backoff #92742

Description

@zhuyankarl

Summary

During a QQ direct incident on 2026-06-13, a stale active run blocked the agent:main:direct:c_total lane. The user message arrived at 19:49:59, but Gateway did not begin processing it until 20:06:11. First visible reply was sent at 20:07:44, for an end-to-end first response delay of about 17m45s.

The diagnostic subsystem did detect the stuck/long-running session, but the same warning backoff path also suppressed follow-up recovery checks until much later. In the observed incident, abort_embedded_run did not release the lane until 20:00:42.

Evidence

  • Logs repeatedly showed stuck session / long-running session, queueDepth=1.
  • Logs showed queued_behind_active_work.
  • The lane was not released promptly even though stale active work was recoverable.
  • Gateway systemd memory peak reached about 5.9-6.0 GB during the broader incident.
  • Local process observation also showed bursts of openclaw-hooks processes; earlier same-day notes observed D-state hooks consuming several GB.

Local Hotfix Applied

In dist/diagnostic-BxXuHqy7.js, the warning backoff conditions were changed from:

if (params.ageMs < nextWarnAgeMs) return;

to:

if (params.ageMs < nextWarnAgeMs && !recoveryEligible) return;

for both session.stuck and session.long_running.

This keeps log throttling while still allowing recovery-eligible sessions to run the recovery path.

Validation

  • node --check passed for the patched file.
  • Gateway restarted successfully.
  • Gateway remained active after restart.

Proposed Upstream Fix

  1. Separate diagnostic log backoff from recovery scheduling.
  2. Add a regression test where:
    • the first stuck warning is emitted;
    • subsequent warnings are throttled;
    • the session becomes recovery-eligible before the next warn interval;
    • recovery still runs.
  3. Add a QQ direct fast-ack / quick-reply path for simple status messages so a user-visible acknowledgement can be sent without waiting for full long-context agent execution.
  4. Add watchdog/telemetry for long-lived or D-state openclaw-hooks processes.

Related Follow-Ups

  • QQ platform message sends should return after platform 200 OK when no additional post-send work is required.
  • Hook relay lifecycle should track and clean stuck child processes.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:queueable-fixClawSweeper marked this issue as an existing queue_fix_pr work candidate.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.impact:session-stateSession, memory, transcript, context, or agent state can drift or corrupt.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions