Skip to content

fix(cron): warn on main heartbeat handoff ghost runs#72677

Open
liaoandi wants to merge 2 commits into
openclaw:mainfrom
liaoandi:fix/63106-cron-next-heartbeat-handoff-warning
Open

fix(cron): warn on main heartbeat handoff ghost runs#72677
liaoandi wants to merge 2 commits into
openclaw:mainfrom
liaoandi:fix/63106-cron-next-heartbeat-handoff-warning

Conversation

@liaoandi

@liaoandi liaoandi commented Apr 27, 2026

Copy link
Copy Markdown
Contributor

Fixes #63106.

Summary

  • Add a scoped warning for fast successful main-session systemEvent cron jobs using wakeMode="next-heartbeat".
  • Persist the possible-main-next-heartbeat-ghost-run annotation in cron run logs.
  • Carry and render cron run-log warnings in the Control UI run history.
  • Add cron.ghostRunWarningThresholdMs config with schema/help metadata and public docs.
  • Keep CronRunLogEntry.warnings source-compatible in generated Swift by defaulting the optional initializer parameter to nil.
  • Keep release-note context in the PR body only; this branch does not edit CHANGELOG.md.

Real behavior proof

Behavior or issue addressed: Verified that a fast successful main-session systemEvent cron handoff using wakeMode: "next-heartbeat" writes the possible-main-next-heartbeat-ghost-run warning into the cron run log, that Control UI run history can render that warning from CronRunLogEntry.warnings, and that existing Swift call sites can construct CronRunLogEntry without passing the new optional warnings label.

Real environment tested: Local macOS checkout on current PR head 5238f67159, rebased on current upstream/main 4a206db106, using the real buildGatewayCronService, real cron run-log reader, Control UI cron run-history renderer, protocol generators, and Swift compiler typecheck. The cron proof used an isolated temporary OPENCLAW_HOME and did not touch the real user cron store.

Exact steps or command run after this patch: Rebased onto current upstream/main, regenerated gateway protocol artifacts, ran focused cron/gateway and Control UI tests, checked formatting/whitespace, verified generated protocol output stability, and typechecked a Swift compatibility snippet that constructs CronRunLogEntry without warnings.

Evidence after fix: Current-head focused test output:

$ node scripts/test-projects.mjs src/gateway/server-cron.test.ts src/cron/cron-protocol-conformance.test.ts ui/src/ui/views/cron.test.ts --reporter verbose

 RUN  v4.1.7 /Users/antonio/projects/openclaw_worktrees/pr72677_clean

 Test Files  3 passed (3)
      Tests  42 passed (42)
   Start at  18:42:27
   Duration  16.44s
$ ./node_modules/.bin/oxfmt --check scripts/protocol-gen-swift.ts src/gateway/server-cron.ts src/gateway/server-cron.test.ts src/cron/run-log.ts src/cron/cron-protocol-conformance.test.ts ui/src/ui/views/cron.ts ui/src/ui/views/cron.test.ts
Checking formatting...

All matched files use the correct format.
Finished in 250ms on 7 files using 12 threads.

Current-head isolated run-log proof shape, redacted from the real gateway cron service/run-log path:

{
  "home": "/tmp/openclaw-ghost-proof.redacted",
  "jobId": "redacted-job-id",
  "logPath": "/tmp/openclaw-ghost-proof.redacted/runs/redacted-job-id.jsonl",
  "latest": {
    "action": "finished",
    "status": "ok",
    "summary": "proof handoff",
    "runId": "cron:redacted-job-id:redacted-start-ms",
    "durationMs": 7,
    "deliveryStatus": "not-requested",
    "warnings": [
      "possible-main-next-heartbeat-ghost-run"
    ]
  }
}

Swift compatibility proof:

$ swiftc -typecheck \
  apps/shared/OpenClawKit/Sources/OpenClawProtocol/AnyCodable.swift \
  apps/shared/OpenClawKit/Sources/OpenClawProtocol/GatewayModels.swift \
  /private/tmp/openclaw-pr72677-swift-compat.swift
(no output)

The compatibility snippet constructs CronRunLogEntry(...) without a warnings: argument, matching existing Swift call sites.

Protocol generation stability:

$ node --import tsx scripts/protocol-gen.ts
wrote /Users/antonio/projects/openclaw_worktrees/pr72677_clean/dist/protocol.schema.json

$ node --import tsx scripts/protocol-gen-swift.ts
wrote /Users/antonio/projects/openclaw_worktrees/pr72677_clean/apps/shared/OpenClawKit/Sources/OpenClawProtocol/GatewayModels.swift

$ git diff --exit-code -- dist/protocol.schema.json apps/shared/OpenClawKit/Sources/OpenClawProtocol/GatewayModels.swift
(no output)

Format and whitespace checks:

$ git diff --check upstream/main...HEAD
(no output)

Observed result after fix: The forced main-session next-heartbeat handoff completed successfully, the persisted run-log entry contains warnings: ["possible-main-next-heartbeat-ghost-run"], Control UI run history renders that warning as a warning chip, regenerated Swift now emits warnings: [String]? = nil, and an old-style Swift initializer call without warnings: typechecks.

What was not tested: I did not dispatch a real scheduled agent turn from the user's production cron store; the warning proof used an isolated temporary OpenClaw home with the real gateway cron service and run-log modules so it would not affect real scheduled jobs. Full swift test --package-path apps/shared/OpenClawKit could not run on this machine because the active developer directory is Command Line Tools only and SwiftPM rejects the package platform declarations .iOS(.v18) / .macOS(.v15) before compiling sources; the targeted Swift compiler proof above covers the specific initializer compatibility blocker.

Tests

  • node scripts/test-projects.mjs src/gateway/server-cron.test.ts src/cron/cron-protocol-conformance.test.ts ui/src/ui/views/cron.test.ts --reporter verbose
  • ./node_modules/.bin/oxfmt --check scripts/protocol-gen-swift.ts src/gateway/server-cron.ts src/gateway/server-cron.test.ts src/cron/run-log.ts src/cron/cron-protocol-conformance.test.ts ui/src/ui/views/cron.ts ui/src/ui/views/cron.test.ts
  • git diff --check upstream/main...HEAD
  • node --import tsx scripts/protocol-gen.ts
  • node --import tsx scripts/protocol-gen-swift.ts
  • git diff --exit-code -- dist/protocol.schema.json apps/shared/OpenClawKit/Sources/OpenClawProtocol/GatewayModels.swift
  • swiftc -typecheck apps/shared/OpenClawKit/Sources/OpenClawProtocol/AnyCodable.swift apps/shared/OpenClawKit/Sources/OpenClawProtocol/GatewayModels.swift /private/tmp/openclaw-pr72677-swift-compat.swift

@openclaw-barnacle openclaw-barnacle Bot added gateway Gateway runtime size: M labels Apr 27, 2026
@greptile-apps

greptile-apps Bot commented Apr 27, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR adds a scoped ghost-run warning for fast ok main-session systemEvent cron jobs running with wakeMode="next-heartbeat", where a quick finish typically indicates the cron scheduler handed work to the next heartbeat rather than confirming actual agent processing. The change is consistent across all relevant layers: type definitions, Zod schema, generated config schema, help/label metadata, run-log parsing, and the gateway cron service itself.

Confidence Score: 5/5

Safe to merge — additive warning path with no changes to existing control flow.

All changes are additive: new config field with a safe default (50ms), new warning emission that does not alter the run outcome, and proper de-serialization of the new warnings field in run-log parsing. Scoping of ghostRunWarning within the evt.action === "finished" block is correct and consistent with the appendCronRunLog call that references it. Edge cases (threshold = 0 to disable, undefined durationMs) are explicitly handled. Tests cover both the positive and negative trigger paths.

No files require special attention.

Reviews (1): Last reviewed commit: "fix(cron): warn on main heartbeat handof..." | Re-trigger Greptile

@clawsweeper

clawsweeper Bot commented Apr 29, 2026

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge. Reviewed July 3, 2026, 5:59 AM ET / 09:59 UTC.

Summary
The PR adds a cron ghost-run warning annotation for fast main-session next-heartbeat systemEvent handoffs, persists and renders it through run history/protocol/UI/docs, and also includes a small unrelated Matrix listener refactor.

PR surface: Source +91, Tests +157, Docs +12, Other +4. Total +264 across 20 files.

Reproducibility: yes. from source: current main still requests a next-heartbeat cron wake and records ok without awaiting agent processing or persisting any warning marker. I did not run a live gateway-down repro because this was a read-only review.

Review metrics: 2 noteworthy metrics.

  • Config/default surface: 1 optional cron config key added with default 50ms. New defaulted config affects operator-visible behavior and needs upgrade/product review before merge.
  • Protocol surface: 1 optional CronRunLogEntry field added. Generated clients and run-history consumers will observe the new field, so compatibility proof matters.

Root-cause cluster
Relationship: fixed_by_candidate
Canonical: #63106
Summary: This PR is the open implementation candidate for the canonical main-session next-heartbeat cron ghost-run warning issue; adjacent reports cover other cron ghost-run or systemEvent failure modes.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Merge readiness
Overall: 🦪 silver shellfish
Proof: 🦪 silver shellfish
Patch quality: 🦐 gold shrimp
Result: blocked until stronger real behavior proof is added.

Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch.

Rank-up moves:

  • [P1] Resolve the conflict against current main and remove or split the unrelated Matrix listener refactor if it is not part of the cron fix.
  • Update the PR body with exact-head redacted proof; terminal output, logs, copied live output, screenshots, or recordings are fine when they show the changed behavior, and private data should be redacted.
  • [P1] Fix the stale JSONL docs wording and make the config/protocol diagnostic shape explicit for maintainer product review.

Proof guidance:

  • [P1] Needs stronger real behavior proof before merge: The PR body has useful redacted terminal/log proof for the cron, UI, protocol, and Swift paths, but it predates the live head and does not cover the extra Matrix changes; add exact-head redacted proof in the PR body for re-review. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.

Risk before merge

  • [P1] The PR adds a new optional cron config key with a default value plus an optional run-log/protocol field, so existing operators and generated clients will see a new upgrade surface that green CI alone does not settle.
  • [P1] The live PR is conflict-marked and the supplied real behavior proof predates head 3928f86 and does not cover the added Matrix listener changes.
  • [P1] The new cron docs point users at a job run-log JSONL path even though current run history is SQLite-backed and JSONL is legacy migration input.

Maintainer options:

  1. Refresh and settle the diagnostic surface (recommended)
    Resolve conflicts, remove or split unrelated Matrix churn, fix the stale docs wording, and get maintainer acceptance for the default-on config/protocol warning before merge.
  2. Accept the new warning surface intentionally
    Maintainers can accept the config and protocol additions as an intentional operator diagnostic after reviewing upgrade impact and generated-client compatibility.
  3. Replace with a narrower implementation
    If the config/protocol surface is not wanted, pause this branch and land a smaller warning-only or separately designed follow-up for the canonical issue.

Next step before merge

  • [P1] This needs maintainer/contributor handling for conflict refresh, exact-head proof, and product acceptance of the config/protocol diagnostic shape rather than an automated repair lane.

Security
Cleared: Cleared: the diff does not add dependencies, workflow execution, downloads, secret handling, or broader permissions; the remaining concern is compatibility, not security.

Review findings

  • [P3] Remove stale JSONL run-log guidance — docs/cli/cron.md:235
Review details

Best possible solution:

The best end state is a refreshed, narrowed PR that implements the warning in the accepted config/protocol shape, has exact-head proof, and keeps docs aligned to SQLite run history.

Do we have a high-confidence way to reproduce the issue?

Yes, from source: current main still requests a next-heartbeat cron wake and records ok without awaiting agent processing or persisting any warning marker. I did not run a live gateway-down repro because this was a read-only review.

Is this the best way to solve the issue?

Unclear as submitted. The gateway/run-log warning is a plausible mitigation, but the defaulted config key, protocol field, conflict state, and stale docs/proof need maintainer acceptance and refresh before this is the best fix.

Full review comments:

  • [P3] Remove stale JSONL run-log guidance — docs/cli/cron.md:235
    The same page says cron run history is retained in SQLite and that runLog.maxBytes only remains for older file-backed logs. Pointing users at the job's run-log JSONL for this new warning sends them to a legacy path instead of the current openclaw cron runs/SQLite-backed history.
    Confidence: 0.86

Overall correctness: patch is correct
Overall confidence: 0.78

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against 36dd9ee3c3c1.

Label changes

Label justifications:

  • P2: This is a normal-priority cron reliability fix with limited blast radius but real operator impact when scheduled main-session work silently looks successful.
  • merge-risk: 🚨 compatibility: The PR adds a defaulted config key and a generated protocol/run-log field that can affect upgrades, operators, and generated clients.
  • rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦪 silver shellfish and patch quality is 🦐 gold shrimp.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The PR body has useful redacted terminal/log proof for the cron, UI, protocol, and Swift paths, but it predates the live head and does not cover the extra Matrix changes; add exact-head redacted proof in the PR body for re-review. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed

PR surface:

Source +91, Tests +157, Docs +12, Other +4. Total +264 across 20 files.

View PR surface stats
Area Files Added Removed Net
Source 13 97 6 +91
Tests 3 157 0 +157
Docs 2 12 0 +12
Config 0 0 0 0
Generated 0 0 0 0
Other 2 5 1 +4
Total 20 271 7 +264

What I checked:

  • Root and scoped policy read: Root AGENTS.md plus src/gateway, ui, docs, and scripts scoped AGENTS.md files were read; the applicable guidance treats config/default and protocol changes as compatibility-sensitive and requires whole-surface PR review. (AGENTS.md:1, 36dd9ee3c3c1)
  • Current main still has the next-heartbeat fast handoff: Current main requests a cron heartbeat for main-session next-heartbeat system events and then returns ok without awaiting agent processing, matching the linked issue's source-level reproduction. (src/cron/service/timer.ts:2089, 36dd9ee3c3c1)
  • Current main run-log append has no warning annotation: The gateway cron finished-event path appends status, summary, delivery, timing, model, provider, and usage data, but no ghost-run warning marker. (src/gateway/server-cron.ts:713, 36dd9ee3c3c1)
  • Current main protocol lacks run-log warnings: CronRunLogEntrySchema on current main exposes timing and telemetry fields but does not include a warnings array for the requested marker. (packages/gateway-protocol/src/schema/cron.ts:565, 36dd9ee3c3c1)
  • Docs conflict with new JSONL wording: Current docs already say retained cron run history is SQLite-backed and maxBytes only remains for older file-backed run logs, so the PR's new JSONL investigation wording is stale for the new warning. Public docs: docs/cli/cron.md. (docs/cli/cron.md:229, 36dd9ee3c3c1)
  • Live PR state and proof context: GitHub reports head 3928f86 as CONFLICTING/DIRTY; the PR body proof is useful but names older heads and does not cover the extra Matrix commit on the live head. (3928f865da48)

Likely related people:

What the crustacean ranks mean
  • 🦀 challenger crab: rare, exceptional readiness with strong proof, clean implementation, and convincing validation.
  • 🦞 diamond lobster: very strong readiness with only minor maintainer review expected.
  • 🐚 platinum hermit: good normal PR, likely mergeable with ordinary maintainer review.
  • 🦐 gold shrimp: useful signal, but proof or patch confidence is still limited.
  • 🦪 silver shellfish: thin signal; proof, validation, or implementation needs work.
  • 🧂 unranked krab: not merge-ready because proof is missing/unusable or there are serious correctness or safety concerns.
  • 🌊 off-meta tidepool: rating does not apply to this item.

Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

How this review workflow works
  • ClawSweeper keeps one durable marker-backed review comment per issue or PR.
  • Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
  • A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
  • PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
  • Maintainers can also comment @clawsweeper review to request a fresh review only.
  • Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
  • Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
  • Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

@openclaw-barnacle openclaw-barnacle Bot added scripts Repository scripts docker Docker and sandbox tooling channel: line Channel integration: line app: web-ui App: web-ui channel: discord Channel integration: discord and removed scripts Repository scripts docker Docker and sandbox tooling app: web-ui App: web-ui labels Apr 29, 2026
@openclaw-barnacle openclaw-barnacle Bot added extensions: memory-core Extension: memory-core commands Command implementations agents Agent runtime and tooling size: XL triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. proof: supplied External PR includes structured after-fix real behavior proof. app: web-ui App: web-ui and removed channel: discord Channel integration: discord channel: line Channel integration: line size: M triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. extensions: memory-core Extension: memory-core commands Command implementations agents Agent runtime and tooling labels May 8, 2026
@clawsweeper clawsweeper Bot added rating: 🐚 platinum hermit Good normal PR readiness with ordinary maintainer review expected. P2 Normal backlog priority with limited blast radius. and removed rating: 🦞 diamond lobster Very strong PR readiness with only minor maintainer review expected. labels May 21, 2026
@clawsweeper

clawsweeper Bot commented May 21, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@openclaw-barnacle openclaw-barnacle Bot removed the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 21, 2026
@clawsweeper clawsweeper Bot added the proof: sufficient ClawSweeper judged the real behavior proof convincing. label May 21, 2026
@openclaw-barnacle openclaw-barnacle Bot added channel: nostr Channel integration: nostr extensions: diagnostics-otel Extension: diagnostics-otel docker Docker and sandbox tooling extensions: acpx extensions: codex extensions: diffs and removed size: L labels May 24, 2026
@liaoandi

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented May 26, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@liaoandi

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Current head 55b75e7 carries CronRunLogEntry.warnings through the Control UI run-history type/renderer/test, removes the release-owned CHANGELOG.md entry, and updates the PR body Real behavior proof with focused UI + cron/gateway test output.

@clawsweeper

clawsweeper Bot commented May 26, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@liaoandi

Copy link
Copy Markdown
Contributor Author

@clawsweeper re-review

Current head acd4a5f preserves Swift source compatibility for CronRunLogEntry.warnings by defaulting the generated optional initializer parameter to nil, regenerates GatewayModels.swift, updates Real behavior proof with focused cron/UI tests, protocol generation stability, and a Swift typecheck proving old call sites compile without warnings:.

@clawsweeper

clawsweeper Bot commented May 26, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper

clawsweeper Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

@liaoandi thanks for the PR. ClawSweeper is still waiting on real behavior proof before this can move forward.

Useful proof can be a screenshot, short video, terminal output, copied live output, linked artifact, or redacted logs that show the changed behavior after the fix. Please redact private tokens, phone numbers, private endpoints, customer data, and anything else sensitive.

Once proof is added to the PR body or a comment, ClawSweeper or a maintainer can re-check it.

@clawsweeper

clawsweeper Bot commented Jul 19, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper status: review started.

I am starting a fresh review of this pull request: fix(cron): warn on main heartbeat handoff ghost runs This is item 1/1 in the current shard. Shard 7/22.

This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking.

Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

app: web-ui App: web-ui channel: matrix Channel integration: matrix docs Improvements or additions to documentation gateway Gateway runtime merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. proof: supplied External PR includes structured after-fix real behavior proof. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. scripts Repository scripts size: M status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

cron: ghost runs recorded as ok when gateway is down (durationMs < 50ms)

1 participant