fix(cron): detect ghost runs on main-session systemEvent jobs by liaoandi · Pull Request #63111 · openclaw/openclaw

liaoandi · 2026-04-08T10:51:19Z

Summary

When the OpenClaw gateway is unhealthy, cron jobs using sessionTarget: "main" + payload.kind: "systemEvent" complete in < 50 ms and are recorded as status: ok even though no agent turn was executed
Adds a GHOST_RUN_THRESHOLD_MS = 50 constant and a structured warn log in the onEvent handler in src/gateway/server-cron.ts when a run completes suspiciously fast
The warning includes jobId, durationMs, sessionTarget, payloadKind, and ghostRunThresholdMs so operators can identify silent failures

Root Cause

executeMainSessionCronJob in src/cron/service/timer.ts calls enqueueSystemEvent() and returns { status: "ok" } immediately without waiting for the agent to process the heartbeat. When the gateway is down or the agent session is unhealthy, the event is silently dropped but the run is still logged as ok.

Changes

src/gateway/server-cron.ts: Added GHOST_RUN_THRESHOLD_MS constant and ghost-run detection warning in the onEvent / appendCronRunLog path

Test plan

With a healthy gateway: main-session cron job runs produce no warning
With a downed gateway (e.g. invalid model config): runs that complete in < 50 ms emit cron: possible ghost run detected in the log with the structured fields
Existing cron service tests pass unchanged

🤖 Generated with Claude Code

When the gateway is unhealthy, main-session cron jobs configured with payload.kind=systemEvent complete in under 50ms and are recorded as ok even though no agent turn was executed. Add a GHOST_RUN_THRESHOLD_MS constant and emit a structured warn log when a run completes suspiciously fast, so operators can identify silent failures. Fixes openclaw#63106

greptile-apps · 2026-04-08T10:53:19Z

Greptile Summary

This PR adds ghost-run detection to the onEvent handler in src/gateway/server-cron.ts by warning when a main-session systemEvent cron job completes in under 50 ms, indicating the gateway may have been unhealthy and the agent turn was silently dropped.

The guard condition uses job.sessionTarget !== \"none\" but \"none\" is not a member of CronSessionTarget (\"main\" | \"isolated\" | \"current\" | \\session:${string}\``), so the comparison is always true. The condition should be `job.sessionTarget === "main"` to match the described root cause and avoid confusing TypeScript strict-mode warnings.

Confidence Score: 4/5

Safe to merge after fixing the !== "none" condition to === "main".

One P1 finding: the guard condition compares job.sessionTarget against "none", a value that is not part of the CronSessionTarget type, making the check always-true and semantically incorrect. The fix is a one-line change. Everything else in the diff is well-structured.

src/gateway/server-cron.ts — line 519, the sessionTarget guard condition.

Vulnerabilities

No security concerns identified. The change only adds a structured log warning — no new data is written to storage, no user input is reflected, and no auth/authz paths are touched.

Prompt To Fix All With AI

This is a comment left during a code review.
Path: src/gateway/server-cron.ts
Line: 519

Comment:
**`!== "none"` compares against an invalid `CronSessionTarget` value**

`CronSessionTarget` is `"main" | "isolated" | "current" | \`session:${string}\`` — `"none"` is never a valid member, so this condition is always `true`. TypeScript strict comparison checks will flag this as an impossible narrowing. The root cause described in the PR is specifically `sessionTarget === "main"`, which is the only target that can legally carry `payload.kind === "systemEvent"` (enforced by `assertSupportedJobSpec`). Use the positive check to match the documented intent:

```suggestion
          job.sessionTarget === "main" &&
```

How can I resolve this? If you propose a fix, please make it concise.

_{Reviews (1): Last reviewed commit: "fix(cron): detect ghost runs on main-ses..." | Re-trigger Greptile}

greptile-apps · 2026-04-08T10:53:23Z

+          typeof evt.durationMs === "number" &&
+          evt.durationMs < GHOST_RUN_THRESHOLD_MS &&
+          job &&
+          job.sessionTarget !== "none" &&


!== "none" compares against an invalid CronSessionTarget value

CronSessionTarget is "main" | "isolated" | "current" | \session:${string}`—"none"is never a valid member, so this condition is alwaystrue. TypeScript strict comparison checks will flag this as an impossible narrowing. The root cause described in the PR is specifically sessionTarget === "main", which is the only target that can legally carry payload.kind === "systemEvent"(enforced byassertSupportedJobSpec`). Use the positive check to match the documented intent:

Suggested change

job.sessionTarget !== "none" &&

job.sessionTarget === "main" &&

Prompt To Fix With AI

This is a comment left during a code review. Path: src/gateway/server-cron.ts Line: 519 Comment: **`!== "none"` compares against an invalid `CronSessionTarget` value** `CronSessionTarget` is `"main" | "isolated" | "current" | \`session:${string}\`` — `"none"` is never a valid member, so this condition is always `true`. TypeScript strict comparison checks will flag this as an impossible narrowing. The root cause described in the PR is specifically `sessionTarget === "main"`, which is the only target that can legally carry `payload.kind === "systemEvent"` (enforced by `assertSupportedJobSpec`). Use the positive check to match the documented intent: ```suggestion job.sessionTarget === "main" && ``` How can I resolve this? If you propose a fix, please make it concise.

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a47179a47a

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-04-08T10:54:26Z

+          job.sessionTarget !== "none" &&
+          job.payload.kind === "systemEvent"


Exclude busy main-lane fast path from ghost-run warning

This warning path will fire for healthy runs that intentionally return quickly: executeMainSessionCronJob returns status: "ok" immediately for recurring main-session jobs when runHeartbeatOnce reports requests-in-flight (see src/cron/service/timer.ts around the recurring-job branch), so sub-50ms durations are expected under normal load. Because this check keys only on durationMs < 50 plus systemEvent, it produces false “gateway may be unhealthy” alerts during routine contention, which can drown out real outage signals.

Useful? React with 👍 / 👎.

…next-heartbeat - Replace `!== "none"` (invalid type) with `=== "main"` (P1) - Add `wakeMode === "next-heartbeat"` to exclude wakeMode "now" fast path - Add wakeMode to structured warn fields for triage - Document requests-in-flight false-positive tradeoff (P2) Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 971215ed8c

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

chatgpt-codex-connector · 2026-04-09T08:58:18Z

+          evt.durationMs < GHOST_RUN_THRESHOLD_MS &&
+          job &&
+          job.sessionTarget === "main" &&
+          job.payload.kind === "systemEvent" &&
+          job.wakeMode === "next-heartbeat"


Avoid warning on healthy next-heartbeat jobs

This condition treats durationMs < 50 as evidence of a ghost run, but for sessionTarget: "main" + payload.kind: "systemEvent" + wakeMode: "next-heartbeat" that fast return is the normal path: executeMainSessionCronJob enqueues the event, calls requestHeartbeatNow, and returns without waiting for processing (src/cron/service/timer.ts). As a result, healthy jobs in this mode will repeatedly emit cron: possible ghost run detected, creating noisy false alarms that make real gateway-health warnings harder to trust.

Useful? React with 👍 / 👎.

next-heartbeat mode is fire-and-forget — every healthy invocation completes in <50 ms, so a duration check produces 100% false positives. Recurring now-mode jobs are also excluded because the busy-lane early return (openclaw#58833) is a legitimate fast path, not a ghost run. Addresses review feedback: - P2: exclude busy main-lane fast path - P2: avoid warning on healthy next-heartbeat jobs Co-Authored-By: Claude Opus 4.6 (1M context) <[email protected]>

steipete · 2026-04-25T14:54:27Z

Closing this as not reproducible on current main after Codex review.

Close as cannot_reproduce: the PR’s narrowed root cause no longer matches current main. One-shot sessionTarget: "main" + payload.kind: "systemEvent" + wakeMode: "now" jobs now wait on runHeartbeatOnce() and propagate skipped/error outcomes instead of immediately returning status: "ok".

What I checked:

Current main waits for wake-now main-session jobs: executeMainSessionCronJob() enqueues the system event, then for job.wakeMode === "now" calls runHeartbeatOnce() and only returns ok on a real ran result; skipped and error reasons are propagated instead of being recorded as success. (src/cron/service/timer.ts:1211, 245451b6a9c5)
Tests pin the non-ghost behavior for one-shot wake-now jobs: The current test suite explicitly asserts that wakeMode: "now" one-shot main-session jobs wait for heartbeat completion before finishing and only then record lastStatus = "ok". (src/cron/service.runs-one-shot-main-job-disables-it.test.ts:463, 245451b6a9c5)
Fast ok returns are now a deliberate busy-lane fallback, not the PR’s unhealthy-gateway root cause: Current code/tests show the intentional fast ok path is the busy-main-lane fallback for requests-in-flight, especially for recurring jobs, matching the review comments that warned about false positives. That is different from the PR body’s claim that the main-session wakeMode: "now" path always returns immediate ok. (src/cron/service/timer.regression.test.ts:762, 245451b6a9c5)
Current gateway cron logging does not implement the proposed warning: server-cron.ts still appends finished run-log entries directly; there is no ghost-run warning on main. That means this PR is not already merged, but also that the proposed heuristic is no longer justified by the current execution path it was meant to cover. (src/gateway/server-cron.ts:553, 245451b6a9c5)
PR discussion usefully narrowed the scope before it became obsolete: The review thread correctly called out the invalid !== "none" check and false positives for busy and next-heartbeat cases; the follow-up commits narrowed the PR to one-shot wakeMode: "now" jobs. After checking current main, that narrowed path already waits for heartbeat completion, so the PR’s stated ghost-run condition is no longer reproducible from code inspection. (64697d21f11b)

Review notes: reviewed against 245451b6a9c5.

openclaw-barnacle Bot added gateway Gateway runtime size: XS labels Apr 8, 2026

greptile-apps Bot reviewed Apr 8, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Apr 8, 2026

View reviewed changes

chatgpt-codex-connector Bot reviewed Apr 9, 2026

View reviewed changes

steipete closed this Apr 25, 2026

clawsweeper Bot mentioned this pull request Apr 27, 2026

cron: ghost runs recorded as ok when gateway is down (durationMs < 50ms) #63106

Open

4 tasks

clawsweeper Bot mentioned this pull request Jun 18, 2026

fix(cron): warn on main heartbeat handoff ghost runs #72677

Open

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

fix(cron): detect ghost runs on main-session systemEvent jobs#63111

fix(cron): detect ghost runs on main-session systemEvent jobs#63111
liaoandi wants to merge 3 commits into
openclaw:mainfrom
liaoandi:fix/63106-cron-ghost-run-detection

liaoandi commented Apr 8, 2026

Uh oh!

greptile-apps Bot commented Apr 8, 2026

Vulnerabilities

Uh oh!

greptile-apps Bot Apr 8, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Apr 8, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Apr 9, 2026

Uh oh!

steipete commented Apr 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	job.sessionTarget !== "none" &&
	job.sessionTarget === "main" &&

		job.sessionTarget !== "none" &&
		job.payload.kind === "systemEvent"

Uh oh!

Uh oh!

Conversation

liaoandi commented Apr 8, 2026

Summary

Root Cause

Changes

Test plan

Uh oh!

greptile-apps Bot commented Apr 8, 2026

Greptile Summary

Confidence Score: 4/5

Vulnerabilities

Uh oh!

greptile-apps Bot Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Apr 8, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Apr 9, 2026

Choose a reason for hiding this comment

Uh oh!

steipete commented Apr 25, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants