fix(agents): keep cron cloud idle watchdog enabled by bek91 · Pull Request #94445 · openclaw/openclaw

bek91 · 2026-06-18T06:01:39Z

Summary

Fixes cron-triggered embedded agent runs skipping the default LLM idle watchdog when no explicit timeout is configured.
Targets the reported Codex harness gpt-5.5 main-agent cron wedge: an isolated agent:main:cron:* run stayed in activeWorkKind=model_call with queued work and recovery=none because cron disabled the implicit model-idle fuse.
Keeps the existing timeout precedence intact: explicit run timeout, provider request timeout, and agents.defaults.timeoutSeconds still win over the implicit default.
Preserves the local-provider opt-out so local models can continue long prompt evaluation without being aborted by the remote-model silence guard.
Adds focused regression coverage for both cron no-timeout behavior and cron local-provider behavior.
Out of scope: changing lane scheduling, queued-work diagnostics, recovery policy, fallback policy, or the configured timeout model.

Reviewers should focus on whether cron-triggered Codex/remote model calls should share the same default idle guard as other embedded runs when no explicit timeout is set, and whether the local-provider exception remains narrow enough.

Linked context

Maintainer-reported installed-runtime incident on OpenClaw 2026.6.8 with the Codex harness using gpt-5.5. No public issue is linked.

Real behavior proof (required for external PRs)

Behavior or issue addressed: Cron-triggered embedded agent runs disabled the default LLM idle watchdog when no explicit timeout was configured, allowing an isolated main-agent cron Codex model call to remain active with queued work in its cron session and no model-idle recovery.
Real environment tested: Maintainer-provided installed OpenClaw 2026.6.8 runtime for the before evidence; local source checkout for the after-patch resolver/stream proof.
Exact steps or command run after this patch:
- Inspected sanitized installed-runtime diagnostics from the reported node.
- Confirmed the reported config had no agents.defaults.timeoutSeconds, no main-agent timeout entry, and no timeoutSeconds on the active configured provider surface.
- Ran an after-patch production-code harness importing src/agents/embedded-agent-runner/run/llm-idle-timeout.ts to exercise resolveLlmIdleTimeoutMs and streamWithIdleTimeout for the incident-shaped cron Codex route.
- node scripts/run-vitest.mjs src/agents/embedded-agent-runner/run/llm-idle-timeout.test.ts
- Earlier validation on this branch also passed:
  - node scripts/run-vitest.mjs src/agents/embedded-agent-runner/run/llm-idle-timeout.test.ts src/agents/openai-transport-stream.test.ts src/llm/providers/openai-responses-shared.test.ts
  - node scripts/run-tsgo.mjs -p test/tsconfig/tsconfig.core.test.json --incremental --tsBuildInfoFile .artifacts/tsgo-cache/core-test.tsbuildinfo
  - git diff --check origin/main...HEAD
  - codex review --base origin/main
  - .agents/skills/autoreview/scripts/autoreview --mode local
Evidence before fix, sanitized from installed-runtime diagnostics:

2026-06-18T00:27:29-04:00 | long-running session: sessionId=<redacted> sessionKey=agent:main:cron:<redacted> state=processing age=136s queueDepth=1 reason=queued_behind_active_work classification=long_running activeWorkKind=model_call lastProgress=codex_app_server:notification:item/started lastProgressAge=0s cronJobId=<redacted> cronRunId=<redacted> cronJob=<redacted> recovery=none

Relevant installed-runtime config precondition, sanitized:

{
  "agents.defaults.timeoutSeconds": null,
  "agents.defaults.model": {
    "primary": "openai/gpt-5.5",
    "fallbacks": ["openai/gpt-5.4"]
  },
  "agents.list.timeoutSeconds": [],
  "models.providers.*.timeoutSeconds": null
}

Evidence after fix from the source harness:

resolver cron codex-gpt-5.5 no-timeout => 120000ms
resolver cron local-provider no-timeout => 0ms
idle callback => LLM idle timeout (120s): no response from model
stream result => LLM idle timeout (120s): no response from model
elapsedMs => 120003

Observed result after fix: The incident-shaped cron Codex gpt-5.5 no-explicit-timeout route now receives the default 120s stream-idle fuse; the local-provider route remains opted out; a stalled stream is aborted by streamWithIdleTimeout after the restored default fuse.
What was not tested: A forced live Codex app-server stall was not recreated on the maintainer node because the original failure mode is not safely deterministic and forcing a live production Codex stall would disturb the running agent. The after-patch proof exercises the exact timeout decision and stream wrapper that were missing in the reported condition.
Proof limitations or environment constraints: This is incident-log plus focused runtime fuse proof, not a synthetic live-provider outage replay.
Before evidence from source: The previous test expectation documented the old cron-wide opt-out by expecting 0 for trigger: "cron" with no timeout configuration.

Tests and validation

node scripts/run-vitest.mjs src/agents/embedded-agent-runner/run/llm-idle-timeout.test.ts
node scripts/run-vitest.mjs src/agents/embedded-agent-runner/run/llm-idle-timeout.test.ts src/agents/openai-transport-stream.test.ts src/llm/providers/openai-responses-shared.test.ts
node scripts/run-tsgo.mjs -p test/tsconfig/tsconfig.core.test.json --incremental --tsBuildInfoFile .artifacts/tsgo-cache/core-test.tsbuildinfo
git diff --check origin/main...HEAD
codex review --base origin/main
.agents/skills/autoreview/scripts/autoreview --mode local

Regression coverage was updated in src/agents/embedded-agent-runner/run/llm-idle-timeout.test.ts to lock down the new cron default and the retained cron local-provider opt-out.

CI note: the previous failing run was on a stale merge from older main; the branch was rebased onto current origin/main and force-pushed with only the intended PR commit retained.

Risk checklist

Did user-visible behavior change? Yes

Did config, environment, or migration behavior change? No

Did security, auth, secrets, network, or tool execution behavior change? No

Highest-risk area: very slow remote/Codex cron responses with no stream progress may now abort after the default idle window unless an explicit timeout is configured.

Mitigation: existing explicit run/provider/agent timeout precedence remains unchanged, and local-provider calls continue to opt out of the implicit watchdog.

Current review state

Next action: ClawSweeper re-review and maintainer review.

Still waiting on: maintainer acceptance that restoring the default idle fuse for no-explicit-timeout cron Codex/remote model calls is the desired compatibility behavior.

Bot/reviewer comments addressed: local Codex review and autoreview reported no accepted/actionable findings; PR body now includes sanitized real incident evidence plus after-patch runtime fuse proof.

AI assistance disclosure

This PR was prepared with AI assistance. The change was reviewed against the touched resolver, its production caller path, focused regression tests, the reported installed-runtime incident shape, and the relevant embedded-run timeout behavior.

clawsweeper · 2026-06-18T20:35:09Z

Codex review: needs maintainer review before merge. Reviewed June 21, 2026, 11:18 PM ET / 03:18 UTC.

Summary
The branch removes the cron-wide idle-watchdog opt-out for embedded cloud model runs, updates resolver tests for cloud cron and local cron behavior, and aligns the agent-loop docs.

PR surface: Source -4, Tests +6, Docs 0. Total +2 across 3 files.

Reproducibility: yes. at source level: current main returns 0 for resolveLlmIdleTimeoutMs({ trigger: "cron" }), and the cron executor passes trigger: "cron" into the embedded run path that wraps streams only for positive idle timeouts.

Review metrics: 1 noteworthy metric.

Cron idle-watch default: 1 default behavior changed. No-explicit-timeout cloud cron model calls now receive the 120s stream-idle watchdog instead of relying only on the cron outer timeout.

Merge readiness
Overall: 🐚 platinum hermit
Proof: 🦞 diamond lobster
Patch quality: 🐚 platinum hermit
Result: ready for maintainer review.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

none.

Risk before merge

[P2] Existing cloud cron model calls with no explicit provider, agent, or run timeout can now abort after 120s of stream silence unless operators configure an explicit timeout.
[P1] This is an availability tradeoff: it restores recovery for wedged cron model calls, but maintainers should accept that very slow silent remote providers no longer rely only on the outer cron deadline.
[P2] The latest shipped release still has the cron no-timeout opt-out, so this is an upgrade-visible default behavior change even though it adds no new config key.

Maintainer options:

Accept the targeted idle fuse (recommended)
If maintainers agree the reported cron wedge is worse than preserving long silent cloud cron calls by default, this narrow tested change is ready for ordinary maintainer merge review.
Require default-preserving behavior
Maintainers can ask for the existing cron-timeout-only behavior to remain the default and add an explicit strict watchdog mode if upgrade compatibility is more important.
Pause for broader cron-stall policy
If this belongs with broader cron fallback and stream-stall policy work, keep this PR open until a maintainer-owned design replaces the narrow fix.

Next step before merge

[P2] Human maintainer review is required because the PR has the protected maintainer label and intentionally changes upgrade-visible cron timeout behavior.

Security
Cleared: The diff only changes timeout resolver behavior, focused tests, and docs; it does not touch dependencies, CI, secrets, auth, or supply-chain execution surfaces.

Review details

Best possible solution:

Merge only after a maintainer explicitly accepts restoring the default cloud cron idle fuse while preserving explicit timeout precedence and the local-provider opt-out.

Do we have a high-confidence way to reproduce the issue?

Yes at source level: current main returns 0 for resolveLlmIdleTimeoutMs({ trigger: "cron" }), and the cron executor passes trigger: "cron" into the embedded run path that wraps streams only for positive idle timeouts.

Is this the best way to solve the issue?

Yes, if maintainers accept the compatibility tradeoff; removing the cron-wide opt-out is narrower than adding a new config surface and keeps explicit run/provider/agent timeout precedence plus the local-provider exception.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against a1828110704f.

Label changes

Label justifications:

P1: The PR addresses a maintainer-reported cron agent model-call wedge that can block queued scheduled work and leave recovery disabled.
merge-risk: 🚨 compatibility: No-explicit-timeout cloud cron calls change behavior on upgrade and may abort after 120s of stream silence unless operators configure an explicit timeout.
merge-risk: 🚨 availability: The touched timeout resolver controls whether cron embedded model calls recover from silence or remain active behind queued work.
rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (logs): Sufficient logs: the PR body includes sanitized installed-runtime before evidence plus after-patch production-code resolver and stream-wrapper output for the incident-shaped cron Codex no-timeout path.
proof: sufficient: Contributor real behavior proof is sufficient. Sufficient logs: the PR body includes sanitized installed-runtime before evidence plus after-patch production-code resolver and stream-wrapper output for the incident-shaped cron Codex no-timeout path.

Evidence reviewed

PR surface:

Source -4, Tests +6, Docs 0. Total +2 across 3 files.

View PR surface stats

Area	Files	Added	Removed	Net
Source	1	0	4	-4
Tests	1	13	7	+6
Docs	1	1	1	0
Config	0	0	0	0
Generated	0	0	0	0
Other	0	0	0	0
Total	3	14	12	+2

What I checked:

Current main disables the no-timeout cron idle watchdog: On current main, resolveLlmIdleTimeoutMs({ trigger: "cron" }) returns 0 before local/cloud provider classification when no explicit run, provider, or agent timeout is present. (src/agents/embedded-agent-runner/run/llm-idle-timeout.ts:185, a1828110704f)
PR diff restores the cloud default path: The PR deletes only the cron-wide return 0 branch, so cron calls with no explicit timeout fall through to the existing cloud/local classification and 120s default watchdog. (src/agents/embedded-agent-runner/run/llm-idle-timeout.ts:185, c2002497f429)
Embedded runner uses the resolver result as the stream fuse: The embedded attempt passes config, trigger, run timeout override, provider request timeout, and model into resolveLlmIdleTimeoutMs, then wraps the stream only when the resolved timeout is positive. (src/agents/embedded-agent-runner/run/attempt.ts:3120, a1828110704f)
Cron reaches the touched embedded-runner path: The isolated cron executor calls runEmbeddedAgent with trigger: "cron" and forwards timeoutMs plus runTimeoutOverrideMs, so scheduled embedded runs exercise the changed resolver branch. (src/cron/isolated-agent/run-executor.ts:361, a1828110704f)
Regression coverage pins both sides: The PR updates the no-timeout cron expectation to the 120000ms default and adds a cron local-provider case that remains opted out at 0ms. (src/agents/embedded-agent-runner/run/llm-idle-timeout.test.ts:142, c2002497f429)
Docs are aligned with the changed behavior: The PR updates the public agent-loop timeout guidance to say cloud cron model runs use the default watchdog while local/self-hosted cron model runs keep the implicit-watchdog opt-out unless explicitly configured. Public docs: docs/concepts/agent-loop.md. (docs/concepts/agent-loop.md:170, c2002497f429)

Likely related people:

vincentkoc: Current blame and recent log history for the cron embedded runner call site and adjacent idle-timeout refactors point to this person as the strongest routing candidate for the agents/cron boundary. (role: recent area contributor; confidence: high; commits: 2732f5821561, 14e448e0e13d, bfc5e4929145; files: src/cron/isolated-agent/run-executor.ts, src/agents/embedded-agent-runner/run/llm-idle-timeout.ts)
Eduardo Buitrago: Authored the merged explicit per-run cron timeout forwarding change that shares this cron-to-idle-watchdog boundary. (role: recent cron timeout contributor; confidence: medium; commits: 336ba2a2b3f1; files: src/cron/isolated-agent/run-executor.ts, src/agents/embedded-agent-runner/run/attempt.ts)
steipete: Recent docs and timeout-policy history touches the same provider timeout and agent-loop behavior surface. (role: adjacent timeout policy contributor; confidence: medium; commits: b08e0da25b8a; files: docs/concepts/agent-loop.md)

What the crustacean ranks mean

🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works

ClawSweeper keeps one durable marker-backed review comment per issue or PR.
Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
Maintainers can also comment @clawsweeper review to request a fresh review only.
Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

bek91 · 2026-06-18T21:58:28Z

@clawsweeper re-review

I updated the PR body with sanitized maintainer-reported incident evidence for the actual agent:main:cron:* Codex harness gpt-5.5 no-explicit-timeout wedge, plus after-patch resolver/stream proof showing:

cron Codex/no-timeout resolves to 120000ms
cron local-provider/no-timeout remains 0
the restored stream watchdog aborts a stalled stream with LLM idle timeout (120s): no response from model

I did not force a live Codex app-server stall because the incident is not safely deterministic and would disturb a running agent.

clawsweeper · 2026-06-18T21:58:32Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

bek91 · 2026-06-18T22:20:34Z

Addressed the updated ClawSweeper finding in c200249.

What changed:

docs/concepts/agent-loop.md now matches this PR's behavior: no-explicit-timeout cloud cron model calls use the default 120s idle watchdog, while cron local/self-hosted model calls remain opted out of the implicit watchdog unless an explicit timeout is configured.
Validation: pnpm docs:check-mdx; git diff --check.

Compatibility note: this PR intentionally accepts the no-explicit-timeout cloud cron behavior change because the reported incident was a agent:main:cron:* Codex harness run with no explicit model/provider/agent timeout and no model-idle fuse. Operators that need longer silent cloud cron model calls should set an explicit provider or agent timeout.

@clawsweeper re-review

clawsweeper · 2026-06-18T22:20:37Z

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

* fix(agents): keep cron cloud idle watchdog enabled * docs: align cron idle timeout guidance

openclaw-barnacle Bot added agents Agent runtime and tooling size: XS maintainer Maintainer-authored PR labels Jun 18, 2026

bek91 marked this pull request as ready for review June 18, 2026 06:18

fix(agents): keep cron cloud idle watchdog enabled

0d3d0fb

bek91 force-pushed the fix/cron-main-agent-blocker branch from cd510a1 to 0d3d0fb Compare June 18, 2026 20:34

docs: align cron idle timeout guidance

c200249

openclaw-barnacle Bot added the docs Improvements or additions to documentation label Jun 18, 2026

vincentkoc self-assigned this Jun 22, 2026

vincentkoc merged commit 5e915e1 into main Jun 22, 2026
228 of 240 checks passed

vincentkoc deleted the fix/cron-main-agent-blocker branch June 22, 2026 22:47

github-actions Bot mentioned this pull request Jun 22, 2026

📡 Upstream Digest — 2026-06-22 23:14 UTC curtismercier/openclaw-mods#1133

Open

github-actions Bot pushed a commit to Desicool/openclaw that referenced this pull request Jun 23, 2026

fix(agents): keep cron cloud idle watchdog enabled (openclaw#94445)

4b6d05c

* fix(agents): keep cron cloud idle watchdog enabled * docs: align cron idle timeout guidance

velvet-shark mentioned this pull request Jun 25, 2026

fix: cron stream stalls fail over before job timeout #96096

Merged

This was referenced Jul 1, 2026

chore: bump openclaw/openclaw version from v2026.6.10 to v2026.6.11 Docker-Hub-sirmark/docker-openclaw#35

Merged

chore(openclaw): bump pinned version v2026.5.28 → v2026.6.11 Evans-Software-Solutions-Limited/axel-saas#121

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(agents): keep cron cloud idle watchdog enabled#94445

fix(agents): keep cron cloud idle watchdog enabled#94445
vincentkoc merged 2 commits into
mainfrom
fix/cron-main-agent-blocker

bek91 commented Jun 18, 2026 •

edited

Loading

Uh oh!

clawsweeper Bot commented Jun 18, 2026 •

edited

Loading

Uh oh!

bek91 commented Jun 18, 2026

Uh oh!

clawsweeper Bot commented Jun 18, 2026 •

edited

Loading

Uh oh!

bek91 commented Jun 18, 2026

Uh oh!

clawsweeper Bot commented Jun 18, 2026 •

edited

Loading

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Uh oh!

Conversation

bek91 commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Linked context

Real behavior proof (required for external PRs)

Tests and validation

Risk checklist

Current review state

AI assistance disclosure

Uh oh!

clawsweeper Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bek91 commented Jun 18, 2026

Uh oh!

clawsweeper Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bek91 commented Jun 18, 2026

Uh oh!

clawsweeper Bot commented Jun 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bek91 commented Jun 18, 2026 •

edited

Loading

clawsweeper Bot commented Jun 18, 2026 •

edited

Loading

clawsweeper Bot commented Jun 18, 2026 •

edited

Loading

clawsweeper Bot commented Jun 18, 2026 •

edited

Loading