Skip to content

fix: refresh subagent idle timeout on successful tool calls#94133

Closed
kumaxs wants to merge 1 commit into
openclaw:mainfrom
kumaxs:fix/subagent-idle-timeout-tool-activity
Closed

fix: refresh subagent idle timeout on successful tool calls#94133
kumaxs wants to merge 1 commit into
openclaw:mainfrom
kumaxs:fix/subagent-idle-timeout-tool-activity

Conversation

@kumaxs

@kumaxs kumaxs commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Problem

When a subagent (sessions_spawn with mode: "run") performs long-running tool calls (web_fetch, read, exec, etc.), the LLM stream is idle. If no LLM token arrives within 120 seconds, the session is terminated with:

LLM idle timeout (120s): no response from model

This is particularly impactful for research/analysis subagents that perform multi-step data collection before generating their final report. The subagent may have completed all its data gathering and be about to synthesize the output, but the 120s window expires and kills the session.

agents.defaults.subagents.runTimeoutSeconds (default 900s / 15min) only controls total wall-clock time, not idle time between LLM turns — so a subagent can be well within its total budget but still get killed because no LLM output happened during tool calls.

Fix

A tool-activity heartbeat mechanism that keeps the session alive when tools are actively returning results:

  1. New module tool-activity-heartbeat.ts — provides onToolActivity() / notifyToolActivity() modelled on the existing llm-request-activity.ts pattern
  2. llm-idle-timeout.tsstreamWithIdleTimeout now subscribes to both LLM-request activity AND tool-activity via armTimer(), so either signal resets the idle watchdog
  3. attempt.ts — calls notifyToolActivity(runAbortController.signal) after every completed tool call (both success and error branches)

The 30-second heartbeat window is intentionally shorter than the 120s default idle timeout, giving slow tool calls multiple chances to refresh before the watchdog fires.

How it works

  1. Tool call starts → LLM stream is idle
  2. Tool completes → notifyToolActivity(runAbortController.signal) fires → idle timer resets (30s window)
  3. Next tool call starts within 30s → timer stays alive
  4. Agent returns to LLM → next() resets the timer via existing onLlmRequestActivity
  5. If neither tool nor LLM makes progress for 120s → timeout (as expected)

No new config fields. No breaking changes. Just a smarter timeout origin.

Related

When a subagent performs long-running tool calls (e.g. web_fetch, read,
exec), the LLM stream is idle. If no LLM token arrives within 120 seconds,
the session is terminated with LLM idle timeout.

This fix adds a tool-activity heartbeat mechanism:
- A new module tool-activity-heartbeat.ts provides a subscription/
  notification pair (onToolActivity / notifyToolActivity) modelled on the
  existing llm-request-activity pattern.
- streamWithIdleTimeout (llm-idle-timeout.ts) now subscribes to both
  LLM-request activity AND tool-activity, resetting the idle timer via
  armTimer() on either signal.
- attempt.ts calls notifyToolActivity(runAbortController.signal) after
  every completed tool call (both success and failure branches), so the
  idle watchdog sees activity even when the model is silent.

The 30-second heartbeat window (TOOL_ACTIVITY_HEARTBEAT_MS = 30_000) is
intentionally shorter than the default 120s idle timeout, giving slow
tool calls multiple chances to refresh before the watchdog fires.

Fixes openclaw#94124
@openclaw-barnacle openclaw-barnacle Bot added agents Agent runtime and tooling size: S triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels Jun 17, 2026
@kumaxs

kumaxs commented Jun 17, 2026

Copy link
Copy Markdown
Contributor Author

@clawsweeper review

@clawsweeper

clawsweeper Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper

clawsweeper Bot commented Jun 17, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 1, 2026, 3:31 PM ET / 19:31 UTC.

Summary
The PR adds an AbortSignal-keyed tool-activity heartbeat and notifies it from the embedded runner Tool Search catalog executor so subagent LLM idle timers can reset after tool completion.

PR surface: Source +61. Total +61 across 3 files.

Reproducibility: yes. for the PR defects at source level: the signal-key mismatch, dropped between-wait activity, and missed direct-tool path are visible from the current code and PR diff. I did not run a live subagent timeout reproduction in this read-only review.

Review metrics: none identified.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #94124
Summary: This PR is a flawed candidate fix for the canonical subagent LLM idle-timeout issue; a newer open PR is the stronger current candidate, while broader timeout and completion-delivery reports remain distinct or partial overlaps.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🧂 unranked krab
Proof: 🧂 unranked krab
Patch quality: 🧂 unranked krab
Result: blocked until real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • Rework the heartbeat around run-scoped state or a single shared lifecycle key.
  • Cover shared direct tools and cataloged/deferred tools with focused idle-timeout tests.
  • [P1] Add redacted terminal output, logs, recording, or linked artifact from a real after-fix subagent run.

Proof guidance:

  • [P1] Needs real behavior proof before merge: No after-fix real subagent run evidence is present for this branch; tests or CI alone would not satisfy the external PR proof gate. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

  • [P1] The listener and notifier use different AbortSignal objects, so the heartbeat can be a no-op on the reported path.
  • [P1] The PR emits a one-shot completion event without persistent run-scoped activity state, so activity between provider waits can be dropped.
  • [P1] The hook is limited to the Tool Search catalog executor and misses normal direct tools such as read and exec through agent-core.
  • [P1] The PR lacks after-fix real behavior proof for this exact branch.

Maintainer options:

  1. Rework and prove this branch before merge (recommended)
    Use one run-scoped heartbeat state, emit from shared visible and deferred tool execution paths, add focused idle-timeout tests, and require redacted real subagent proof for this exact branch.
  2. Coordinate with the newer candidate
    If maintainers prefer the proof-positive runId-scoped approach, close or pause this branch only after preserving useful attribution and landing the newer candidate.

Next step before merge

  • [P1] Contributor and maintainer follow-up is needed because this branch needs code rework plus contributor-supplied real behavior proof, and a stronger open candidate already exists.

Security
Cleared: The diff adds only in-process timeout signaling and does not change dependencies, workflows, package scripts, credentials, permissions, or external code execution.

Review findings

  • [P1] Use one lifecycle key for tool heartbeats — src/agents/embedded-agent-runner/run/llm-idle-timeout.ts:300
  • [P1] Persist activity across provider waits — src/agents/embedded-agent-runner/run/attempt.ts:3681
  • [P2] Emit heartbeats from shared tool execution — src/agents/embedded-agent-runner/run/attempt.ts:3681
Review details

Best possible solution:

Land #95536 or rework this branch to the same run-scoped shared tool-execution heartbeat with focused tests and real subagent proof; keep #94124 open until a fix merges.

Do we have a high-confidence way to reproduce the issue?

Yes, for the PR defects at source level: the signal-key mismatch, dropped between-wait activity, and missed direct-tool path are visible from the current code and PR diff. I did not run a live subagent timeout reproduction in this read-only review.

Is this the best way to solve the issue?

No. A heartbeat is the right direction, but this implementation is not the best fix because it is keyed to the wrong lifecycle object and does not cover the shared tool execution boundary.

Full review comments:

  • [P1] Use one lifecycle key for tool heartbeats — src/agents/embedded-agent-runner/run/llm-idle-timeout.ts:300
    onToolActivity registers the listener under streamAbortController.signal, but the notifier calls notifyToolActivity(runAbortController.signal). Because the new listener map is keyed by AbortSignal, tool completion will not reach armTimer() for the reported timeout path.
    Confidence: 0.95
  • [P1] Persist activity across provider waits — src/agents/embedded-agent-runner/run/attempt.ts:3681
    This fires only when the cataloged tool has finished, but the previous provider stream listener can already be cleaned up and the next provider wait has not registered yet. Without run-scoped stored activity, the heartbeat can be dropped between LLM turns.
    Confidence: 0.88
  • [P2] Emit heartbeats from shared tool execution — src/agents/embedded-agent-runner/run/attempt.ts:3681
    The new notification is only in the Tool Search catalog executor, while normal direct tools execute through packages/agent-core/src/agent-loop.ts. The reported read and exec style tools can bypass this heartbeat unless the shared path is wired or explicitly proven irrelevant.
    Confidence: 0.84

Overall correctness: patch is incorrect
Overall confidence: 0.92

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 24d94a54a516.

Label changes

Label justifications:

  • P1: The PR targets an active embedded-agent availability bug where long-running subagent tool work can be killed by the LLM idle watchdog.
  • merge-risk: 🚨 availability: Merging the current heartbeat wiring could leave active subagent runs timing out while making the availability bug appear fixed.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🧂 unranked krab.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs real behavior proof before merge: No after-fix real subagent run evidence is present for this branch; tests or CI alone would not satisfy the external PR proof gate. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Source +61. Total +61 across 3 files.

View PR surface stats
Area Files Added Removed Net
Source 3 61 0 +61
Tests 0 0 0 0
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 3 61 0 +61

What I checked:

Likely related people:

  • osolmaz: Authored the latest merged first-event timeout work touching the same LLM idle-timeout surface. (role: recent idle-timeout contributor; confidence: high; commits: 530d576cc642; files: src/agents/embedded-agent-runner/run/llm-idle-timeout.ts)
  • vincentkoc: Recent history shows repeated changes to embedded-runner idle-timeout and attempt surfaces. (role: recent embedded-runner contributor; confidence: high; commits: 7f6a93eb8efd, 7b74d7332f70, bfc5e4929145; files: src/agents/embedded-agent-runner/run/llm-idle-timeout.ts, src/agents/embedded-agent-runner/run/attempt.ts)
  • lanzhi-lee: Authored the provider idle-timeout fix that changed the same timeout resolver and tests. (role: adjacent idle-timeout contributor; confidence: medium; commits: 43d0aaec3d86; files: src/agents/embedded-agent-runner/run/llm-idle-timeout.ts, src/agents/embedded-agent-runner/run/llm-idle-timeout.test.ts)
  • fuller-stack-dev: Recent Tool Search directory-mode and post-tool behavior work overlaps the cataloged-tool boundary touched here. (role: adjacent tool execution contributor; confidence: medium; commits: 965fa05df38b, 0314819f918a; files: src/agents/tool-search.ts, src/agents/embedded-agent-runner/run/attempt.ts, packages/agent-core/src/agent-loop.ts)
What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. labels Jun 18, 2026
@wangmiao0668000666

Copy link
Copy Markdown
Contributor

Hey @kumaxs — I dug into the ClawSweeper review and wanted to flag the three P1 findings with concrete pointers, plus how #95536 already addresses each one. Not opening a competing PR; just trying to help the canonical issue get unblocked whichever PR lands.

The 3 P1 findings, with the exact mismatch

1. Signal-key mismatch (confidence 0.95) — current code is a no-op

// src/agents/embedded-agent-runner/run/llm-idle-timeout.ts:299
const unsubscribeToolActivity = onToolActivity(streamAbortController.signal, armTimer);

// src/agents/embedded-agent-runner/run/attempt.ts:3678
notifyToolActivity(runAbortController.signal);

streamAbortController.signal and runAbortController.signal are two different AbortSignal objects. The new module keys the listener map by AbortSignal:

const toolActivityListeners = new WeakMap<AbortSignal, Set<() => void>>();

So notifyToolActivity(runAbortController.signal) looks up listeners registered with runAbortController.signal, but the only listener is registered against streamAbortController.signal. The notification never reaches armTimer(). The heartbeat is effectively dead code on the bug path.

2. Activity dropped between stream waits (confidence 0.88)

onToolActivity is registered inside streamWithIdleTimeout's cleanupIterator. The lifecycle is:

  • register listener → wait for stream → cleanupIterator unregisters listener
  • Tool completes after stream cleanup → notify fires with zero listeners

So even if the signal keys matched, the notification would still miss because the listener has already been torn down by the time the tool completes.

3. Cataloged-tools only (confidence 0.84)

The two notifyToolActivity(...) calls at attempt.ts:3678 and :3694 are inside the Tool Search catalog executor wrapper. Direct tools (read, exec, web_fetch) execute through packages/agent-core/src/agent-loop.ts executePreparedToolCall — that path bypasses this notification entirely.

How #95536 addresses all three

For awareness (so it's clear there's already a working shape on the table):

Finding #95536's solution
Signal-key mismatch Drops signal-keyed WeakMap, uses runId-keyed Map<runId, Set<listener>> in src/shared/tool-activity-heartbeat.ts
Dropped between waits Adds getLastToolActivityMs(runId) — timestamps persist across stream lifecycles, consumed once via a streamFirstArmDone guard
Cataloged-only Wraps the shared effectiveTools execute path in attempt.ts rather than the catalog executor

So #95536 already has the architectural pieces; it just needs proof: sufficient (real subagent run evidence) to clear the same needs-real-behavior-proof gate.

Paths forward (your call)

A few options I see, roughly ordered by blast radius:

  1. Coordinate with fix(agents): add tool-activity heartbeat to keep subagent alive during tool calls #95536 — ask @xydt-tanshanshan whether they'd accept a Co-authored-by: or a co-PR with real proof on top of their runId-scoped design. Most likely to clear both gates fastest.
  2. Adopt the runId-keyed pattern in this PR — re-key the WeakMap by runId, persist last-activity-ms across stream waits, wrap the shared direct-tool path. Bigger diff but matches the reviewer guidance and could replace fix(agents): add tool-activity heartbeat to keep subagent alive during tool calls #95536.
  3. Close in favor of fix(agents): add tool-activity heartbeat to keep subagent alive during tool calls #95536 — if you'd rather not grind a re-key, marking this as superseded by fix(agents): add tool-activity heartbeat to keep subagent alive during tool calls #95536 unblocks the canonical issue. Your diagnosis and PoC still count.

Whatever you pick, I'd suggest pinging ClawSweeper once the patch is updated so the re-review queue picks it up — the sweep.yml dispatch has been slow on this shard.

If it's useful, I can draft a small repro harness script that exercises a subagent with a 3-minute mock tool call and shows model_call:started → tool → tool → done progression under the new heartbeat. Just let me know.

@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added stale Marked as stale due to inactivity triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. labels Jul 16, 2026
@openclaw-barnacle

Copy link
Copy Markdown

Closing due to inactivity.
If you believe this PR should be revived, post in #clawtributors on Discord to talk to a maintainer.
That channel is the escape hatch for high-quality PRs that get auto-closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

agents Agent runtime and tooling merge-risk: 🚨 availability 🚨 May cause crashes, hangs, restart loops, stalls, or process outages. P1 High-priority user-facing bug, regression, or broken workflow. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: S stale Marked as stale due to inactivity status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants