Skip to content

[Bug]: Embedded run timeout fails to clean up session/lane state when tool execution is in-flight #14228

Description

@roccoren

Summary

When an embedded run times out (600s) while a tool call (e.g., exec running SSH commands) is in-flight, the abort signal fails to properly clean up the run state. The session remains active=true permanently, blocking the lane and preventing all subsequent messages from being processed.

This is a sibling bug to #13341 (which covers the same symptom when compaction is in-flight). In this case, the timeout fires during tool execution, not compaction.

Reproduction Steps

  1. Have an agent session that performs long-running exec tool calls (e.g., SSH into a remote VM, monitoring a process)
  2. The tool call takes longer than expected, causing the total run to exceed timeoutSeconds (default 600s)
  3. The timeout fires (embedded run timeout logged as WARNING)
  4. The finally cleanup block never executes
  5. The session lane is permanently blocked

Observed Behavior (3 occurrences in 3 hours)

16:23:23 embedded run start: runId=ed1f4099 sessionId=50284e3a
16:28:43 embedded run prompt end: durationMs=320111
16:32:13 run active check: active=true
16:33:23 embedded run timeout: timeoutMs=600000
         ❌ No "session state: prev=processing new=idle"
         ❌ No "run cleared"
         ❌ No "lane task done"
16:38:00 ← Lane permanently blocked, all new messages queued but never dequeued

This pattern repeated 3 times within 3 hours on the same session:

  • 16:33 — timeout during tool exec → stuck → required full restart
  • 17:32 — timeout during tool exec → stuck → required full restart
  • 18:12 — timeout during tool exec → stuck → required full restart

Expected Behavior

When the embedded run timeout fires during tool execution:

  • The abort signal should propagate to the in-flight tool call
  • clearActiveEmbeddedRun() should be called (session returns to idle)
  • The lane task should complete, unblocking the lane
  • Subsequent messages should be processed normally

Root Cause (Hypothesis)

Similar to #13341: the tool execution await is likely not wrapped in abortable(), so when abortRun(true) fires, the abort signal never interrupts the pending tool call. The finally block containing cleanup logic never executes.

The fix from #13341 (wrapping waitForCompactionRetry() in abortable()) should be extended to cover tool execution awaits as well — or better yet, ensure all async operations within the run loop are abort-aware.

Environment

  • OpenClaw version: 2026.2.1
  • OS: Ubuntu (Linux 6.14.0-1017-azure x64)
  • Installation: npm install -g openclaw
  • Channel: Discord (thread session)
  • Agent model: github-copilot/claude-opus-4.5 (model-independent)
  • Tool involved: exec (SSH to remote VM)
  • Run timeout: 600s (default)

Workaround

Full gateway restart is the only fix:

systemctl --user restart openclaw-gateway

SIGUSR1 / hot reload does not work (in-memory state survives).
After restart, the affected session needs /reset to clear accumulated context.

Impact

High — Sessions doing legitimate long-running tool work (SSH, file transfers, process monitoring) hit this repeatedly. In our case, 3 occurrences in 3 hours, each requiring manual intervention.

Related

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingstaleMarked as stale due to inactivity

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions