Skip to content

fix(cron): record designed heartbeat skips as successful no-ops - #145242

Closed
Lubaoshuai wants to merge 6 commits into
openclaw:mainfrom
Lubaoshuai:fix/heartbeat-designed-skip-accounting
Closed

Lubaoshuai wants to merge 6 commits into
openclaw:mainfrom
Lubaoshuai:fix/heartbeat-designed-skip-accounting

Conversation

@Lubaoshuai

@Lubaoshuai Lubaoshuai commented Sep 11, 2026 •

Copy link
Copy Markdown

What Problem This Solves

An agent whose heartbeat file is intentionally empty (or comments-only) skips every interval by design, but each skip was recorded as a skipped cron run, which resolveCronCompletionStatus maps to failed. openclaw tasks list --status failed accrued one row per interval (~48/day) for the heartbeat cron and failure alerts fired, even though consecutiveErrors stayed 0 (#144959).

Designed no-op skips now settle as ok runs whose summary carries the skip reason (heartbeat skipped: empty-heartbeat-file), matching the busy-deferral handoff path that already returned ok. The heartbeat runner's own state (system heartbeat last, consecutiveSkipped) keeps reporting the precise skipped status, so no signal is lost.

Evidence

  • Real-behavior proof (updated in f8c3fa9 and 6fb6ec6 per review): src/cron/service.heartbeat-real-preflight.test.ts wires the real heartbeat runner (real preflight, no injected heartbeat results), a real comments-only scratch row in the sqlite state store (stock "keep this file empty" content), and the real cron service. The two review findings are addressed:
    • Same-partition fixture: the sandbox is now selected as the cron partition the preflight actually reads — OPENCLAW_STATE_DIR points the default partition (<state>/cron/jobs.json) at the seeded cronStorePath, so resolveHeartbeatPreflight genuinely consumes the seeded comments-only scratch instead of a different store.
    • Required dependency supplied: the CronServiceDeps.requestHeartbeat member is provided (non-waiting) alongside requestHeartbeatAndWait, matching the production contract.
    • Safeguard restoration: commit ffb0026 accidentally dropped the fixture safeguards; 6fb6ec6 restores them (partition stub, requestHeartbeat dependency, skip-summary + spy assertions), so the test again cannot pass through ordinary heartbeat execution.
    • Skip assertions, not just success: the finished event must carry summary: "heartbeat skipped: empty-heartbeat-file", and the reply/telegram spies must stay unused — so an ordinary successful agent turn can no longer satisfy the test. Paused-reminder recovery stays covered: the operator disabled skip keeps its skipped/failed outcome in the unit matrix.

Runtime output (local vitest run at 6fb6ec6, cron config, redacted):

✓ src/cron/service.heartbeat-real-preflight.test.ts > real heartbeat preflight task accounting > settles a real empty-scratch preflight as a successful no-op task 1155ms

 Test Files  1 passed (1)
      Tests  1 passed (1)

Full local cron-suite run at 6fb6ec6 (service.heartbeat-payload.test.ts + service.heartbeat-real-preflight.test.ts + service.test.ts):

 Test Files  3 passed (3)
      Tests  10 passed (10)
  • The test also asserts the ledger that feeds tasks list --status failed contains zero failed cron rows after the designed skip.
  • Unit-level matrix in service.heartbeat-payload.test.ts (10/10): designed skips settle ok/succeeded with consecutiveErrors: 0; the operator disabled skip keeps its skipped outcome for visibility.
  • Matches the existing intentional-silence precedent in resolveCronCompletionStatus, and the busy-deferral handoff path in timer-execution.ts that already returned ok. The runner's own state (system heartbeat last, consecutiveSkipped) continues to report the precise skipped status.

A heartbeat that intentionally skips (empty heartbeat file, no pending
event, disabled, ...) was recorded as a skipped cron run, which
resolveCronCompletionStatus maps to failed: tasks list --status failed
accrued a row per interval and failure alerts fired even though
consecutiveErrors stayed 0 (openclaw#144959).

Designed no-op skips now settle as ok runs whose summary carries the
skip reason, matching the busy-deferral handoff path that already
returns ok. The heartbeat runner's own state (system heartbeat last,
consecutiveSkipped) keeps reporting the precise skipped status.
@clawsweeper

clawsweeper Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

@openclaw-barnacle openclaw-barnacle Bot added size: XS triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. labels Sep 11, 2026
@clawsweeper clawsweeper Bot added P2 Normal backlog priority with limited blast radius. merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. labels Sep 11, 2026
@clawsweeper

clawsweeper Bot commented Sep 11, 2026 •

Copy link
Copy Markdown
Contributor

Codex review: needs real behavior proof before merge.

What this changes

Records empty-heartbeat-file and no-pending-event skips as successful Cron runs while retaining other skipped outcomes.

Example: A heartbeat monitor runs every 30 minutes with comments-only scratch.

  • Before: Cron records skipped with failed completion and increments consecutiveSkipped.
  • After: Cron records ok with successful completion and a heartbeat skipped summary, resetting consecutiveSkipped.

Review scores

Measure Result What it means
Overall readiness 🧂 unranked krab (1/6) Focused implementation and useful harness coverage, with unresolved product semantics and no qualifying shipped-path proof.
Proof confidence 🧂 unranked krab (1/6) Real behavior proof is necessary before merge. See Before merge.
Patch quality 🦐 gold shrimp (3/6) No actionable review findings were identified.

Product

Kind: Bug fix · Worth it: Needs a maintainer decision · Fix scope: Complete
User problem: Intentional heartbeat no-ops produce failed accounting and misleading operational failure counts.
Reason: The original Task surface is retired, while this PR still changes Cron history, counters, and alert semantics established by prior owner work.

Merge readiness

⛔ Blocked before merge - 3 items remain

Keep open: this PR contains a distinct Cron accounting change that current main does not implement, and its operator semantics need an owner decision.

Priority: P2
Reviewed head: 6fb6ec666ba14baedd59f576abcc23b530cf5b84
Owner decision: Required. See Decision needed.

Decision needed

  • Question: Should intentional heartbeat no-ops become successful Cron runs, resetting consecutiveSkipped and suppressing includeSkipped alerts, now that Tasks is retired?
  • Recommendation: Preserve skipped run status: Retain skip history and counters, and consider a narrow Cron completion-accounting repair for intentional no-ops.
  • Why: Existing history deliberately preserves skipped outcomes, and the author acknowledges the broader reporting change; no recorded owner approval selects the new semantics.

Before merge

  • Add real behavior proof - Supplied Vitest output exercises real CronService, heartbeat preflight, and SQLite with injected reply/delivery dependencies. Provide shipped Gateway/CLI output for intentional and unsuccessful skips, plus existing-state restart evidence; redact IPs, API keys, phone numbers, and non-public endpoints. The inspected screenshots demonstrate another PR's provider-preflight behavior. After adding proof, update the PR body, then request a new review with @clawsweeper re-review. The PR author or someone with repository write access can request it.
  • Resolve maintainer decision - Resolve the maintainer decision shown above before merge.
  • Add data-model compatibility proof - The review found that existing stored data may not work after upgrade. Show that existing data still loads and works with this change.

Findings

None.

Tests

  • Missing end-to-end proof: No shipped Gateway/CLI observation covers intentional and disabled skips, and no pre-fix failing execution is supplied.
Agent review details

How this fits together

Cron dispatches heartbeat wakes, waits for the heartbeat runner's terminal result, then persists run history, counters, completion status, and alert decisions.

flowchart LR
  A[Cron schedule] --> B[Heartbeat wake queue]
  B --> C[Heartbeat preflight]
  C --> D[Cron outcome mapping]
  D --> E[Persisted run history]
  D --> F[Counters and alerts]
Loading

Technical review

Best possible solution:

Resolve intentional no-op accounting in the existing Cron completion owner while preserving skip diagnostics and unsuccessful-skip visibility.

Do we have a high-confidence way to reproduce the issue?

Current-main source shows empty-scratch preflight returning skipped, Cron retaining that status, and completion accounting returning failed; the retired tasks list command cannot reproduce the original presentation on current main.

Is this the best way to solve the issue?

The patch uses the existing execution owner, but changing run status has a wider footprint than fixing completion accounting and needs an explicit Cron policy choice.

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against d4babe8e8ba4.

Provenance checked

  • Heartbeat child settlement changes intended behavior with a stated reason (1b69866: Wait for the heartbeat child's terminal outcome so accepted wake dispatch cannot falsely report success.)
  • Main-session heartbeat result mapping changes intended behavior with a stated reason (6d1c1c9: Unify heartbeat scheduling under Cron while retaining guarded wake execution and terminal outcome reporting.)

Testing

Proof path: in-process harness.

Security

None.

Evidence

What I checked:

PR surface

Source +29, Tests +158. Total +187 across 3 files. Added test files: 1.

View PR surface stats
Area Files Added Removed Net
Source 1 31 2 +29
Tests 2 158 0 +158
Docs 0 0 0 0
Config 0 0 0 0
Generated 0 0 0 0
Other 0 0 0 0
Total 3 189 2 +187

Root-cause cluster

Relationship: fixed_by_candidate
Canonical: #144959
Summary: The linked report and two candidate PRs concern intentional heartbeat skips reaching failed accounting through different owners; Tasks retirement removes one presentation surface but leaves Cron semantics.

Members:

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Labels

Label changes:

No label changes.

Label justifications:

  • P2: Intentional heartbeat no-ops affect operational reporting, with no demonstrated execution outage or data loss.
  • merge-risk: 🚨 compatibility: Changing skipped runs to ok changes persisted run statuses, consecutiveSkipped, and existing includeSkipped alert behavior.
  • rating: 🧂 unranked krab: Overall readiness is 🧂 unranked krab; proof is 🧂 unranked krab and patch quality is 🦐 gold shrimp.
  • status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

  • Record the selected Cron accounting policy and align the implementation and counter claims with it.
  • If sponsored, adapt the retired Task-ledger assertions to current Cron history and completion surfaces.

Rating scale

6/6 🦀 challenger crab · 5/6 🦞 diamond lobster · 4/6 🐚 platinum hermit · 3/6 🦐 gold shrimp · 2/6 🦪 silver shellfish · 1/6 🧂 unranked krab. Overall follows the weaker of proof and patch quality; ✨ marks media proof (a screenshot, video, or linked artifact) that directly shows the changed behavior.

Workflow

ClawSweeper edits this one comment on every review. Comment @clawsweeper re-review for a fresh review only. Repair and merge execution lanes are retired.

History

Review history (8 earlier review cycles)
  • reviewed 2026-09-11T20:28:32.559Z sha 859dcde :: needs real behavior proof before merge. :: [P1] Preserve skipped outcomes for paused one-shot reminders
  • reviewed 2026-09-11T20:41:20.798Z sha 859dcde :: needs real behavior proof before merge. :: [P1] Preserve skipped outcomes for paused one-shot reminders
  • reviewed 2026-09-11T21:09:25.158Z sha c76f1da :: needs real behavior proof before merge. :: none
  • reviewed 2026-09-13T16:31:22.586Z sha 0243ca4 :: needs real behavior proof before merge. :: [P2] [P2] Seed the scratch partition that the heartbeat actually reads | [P2] [P2] Supply the required heartbeat request dependency
  • reviewed 2026-09-13T16:57:50.577Z sha f8c3fa9 :: needs real behavior proof before merge. :: none
  • reviewed 2026-09-14T01:21:20.016Z sha ffb0026 :: needs real behavior proof before merge. :: [P2] [P2] Restore the preflight partition and skip-specific assertions | [P2] [P2] Restore the required heartbeat request dependency
  • reviewed 2026-09-22T17:57:06.810Z sha 6fb6ec6 :: needs real behavior proof before merge. :: none
  • reviewed 2026-10-05T09:10:53.388Z sha 6fb6ec6 :: needs real behavior proof before merge. :: none

Reviewed October 16, 2026, 3:01 AM ET / 07:01 UTC (Revision 9).

@openclaw-barnacle openclaw-barnacle Bot removed the triage: needs-pr-context Candidate: external PR body lacks required problem context or evidence. label Sep 11, 2026
CI flagged that blanket-settling every heartbeat skip as ok erased the
operator-visible skipped state for globally paused/disabled heartbeats.
Restrict the ok settlement to the designed no-op reasons
(empty-heartbeat-file, no-pending-event); every other skip reason,
including disabled, keeps the previous skipped outcome.
@clawsweeper clawsweeper Bot removed the merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. label Sep 11, 2026
…l task

End-to-end case wiring the real heartbeat runner, a real comments-only
scratch row in the sqlite state store, and the real cron service: the
preflight skips the turn with empty-heartbeat-file and the run settles
as ok/succeeded with no failed cron task rows in the ledger.

Signed-off-by: Lubaoshuai <[email protected]>
@Lubaoshuai

Copy link
Copy Markdown
Author

Added the real-behavior proof requested by the review: service.heartbeat-real-preflight.test.ts drives the real heartbeat runner preflight over a real comments-only scratch row in the sqlite state store and asserts the run settles as a successful no-op with zero failed cron task rows in the ledger (the tasks list --status failed source). PR body Evidence section updated accordingly. @clawsweeper re-review

Address the ClawSweeper review findings on the proof fixture:
- seed and read the same cron partition (OPENCLAW_STATE_DIR points the
  default partition used by resolveHeartbeatPreflight at the sandbox)
- supply the required requestHeartbeat dependency
- assert the empty-heartbeat-file outcome and that the agent turn and
  delivery never run, so a plain successful heartbeat cannot pass

Signed-off-by: Lubaoshuai <[email protected]>
…tions

The previous commit accidentally removed the fixture safeguards that tie
this regression test to the intended path: the OPENCLAW_STATE_DIR stub
that points preflight at the sandbox partition, the required
requestHeartbeat dependency from CronServiceDeps, the empty-heartbeat
skip-summary assertion, and the reply/delivery spy checks that prove the
agent turn never ran. Restore all of them (plus the state-db teardown and
env unstubs they require), so the test cannot pass through ordinary
heartbeat execution.

Local vitest (cron config): 10/10 pass across service.heartbeat-payload,
service.heartbeat-real-preflight and service.test.
@Lubaoshuai

Copy link
Copy Markdown
Author

6fb6ec6 restores the fixture safeguards that ffb0026 accidentally removed, addressing both P2 findings:

  • Preflight partition restored: OPENCLAW_STATE_DIR again points the default cron partition at the sandbox, so resolveHeartbeatPreflight genuinely consumes the seeded comments-only scratch instead of falling through to an ordinary agent turn.
  • Required dependency restored: the CronServiceDeps.requestHeartbeat member is supplied again alongside requestHeartbeatAndWait (type contract at src/cron/service/state.ts:170).
  • Skip-specific assertions restored: the finished event must carry summary: "heartbeat skipped: empty-heartbeat-file" and the reply/telegram spies must stay unused, plus the state-db teardown and vi.unstubAllEnvs() the fixture needs.

Local vitest run at 6fb6ec6 (cron config, redacted terminal output):

✓ src/cron/service.heartbeat-real-preflight.test.ts > real heartbeat preflight task accounting > settles a real empty-scratch preflight as a successful no-op task 1155ms

 Test Files  1 passed (1)
      Tests  1 passed (1)

Full local cron-suite run at 6fb6ec6 (service.heartbeat-payload.test.ts + service.heartbeat-real-preflight.test.ts + service.test.ts): 3 files / 10 tests, all passed. oxfmt --check clean on all touched files. PR body Evidence section updated.

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@clawsweeper clawsweeper Bot added the merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. label Sep 22, 2026
@Lubaoshuai

Copy link
Copy Markdown
Author

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented Sep 22, 2026 •

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event exact_review_queue).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

@Lubaoshuai

Copy link
Copy Markdown
Author

Thank you for the detailed revision-7 review — both fixture findings are confirmed resolved, and the remaining items are now about product semantics rather than the patch itself.

Two honest acknowledgments:

  1. Operator-contract footprint. The review is right that converting designed skips to ok runs makes them invisible to existing consumers of skipped cron history, consecutiveSkipped, and includeSkipped alerts. That is a real reporting change, and fix(cron): record skipped cron runs as cancelled, not failed #145025's projection approach (keep cron outcomes skipped, change the task projection so they stop surfacing as failures) has a smaller operator-contract footprint while still fixing the alert-noise problem in tasks list --status failed counts intentional heartbeat "empty-heartbeat-file" skips as failed #144959.

  2. Gateway-level demo. A production dist build is not available in this environment (the prepack pipeline requires the full workspace toolchain), so the vitest harness — which drives the real CronService, real heartbeat preflight, and the real SQLite state store, with only the reply/telegram delivery dependencies fixture-injected and asserted unused — is the strongest evidence I can attach for now. It covers the designed no-op path, the unsuccessful-skip control (operator disabled stays skipped/failed), and the ledger assertion; I've marked it as such in the body rather than overstating it.

Given the reviewer recommendation and the required owner decision, I'll defer to the maintainers: if the projection approach in #145025 is preferred, I'm happy to close this PR in its favor (or rework this one onto the projection semantics if that's useful). Until then I'll leave the PR open so the decision has both options on the table.

@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale due to inactivity.
Please add updates or it will be closed.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Oct 15, 2026
@Lubaoshuai

Copy link
Copy Markdown
Author

Status update (the PR is awaiting a maintainer decision, not contributor action):

Happy to provide anything else that would help the decision.

@clawsweeper clawsweeper Bot added clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. and removed rating: 🦪 silver shellfish Thin PR readiness signal; proof, validation, or implementation needs work. clawsweeper:intake:bug Preliminary submission classification; full source review pending. clawsweeper:intake:needs-environment Preliminary submission classification; full source review pending. clawsweeper:intake:needs-repro Preliminary submission classification; full source review pending. labels Oct 16, 2026
@openclaw-barnacle openclaw-barnacle Bot removed the stale Marked as stale due to inactivity label Oct 16, 2026
@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale after 7 days without qualifying human activity.
Without new human activity, Barnacle may close it as not planned 7 days after this message was posted. This is administrative inactivity cleanup, not a claim that the problem is fixed.
A human update restarts the inactivity clock; a new PR head also restarts it. A maintainer can add no-stale for a durable hold, or bad-barnacle to suppress Barnacle. Remove the hold when work can resume.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Oct 24, 2026
@Lubaoshuai

Copy link
Copy Markdown
Author

Acknowledging the stale mark — this is expected, since the PR has been sitting on the owner decision recorded in the ClawSweeper review (intentional no-op accounting here vs. the projection approach in #145025) for about a month, with findings=0 and an explicit "Keep open" from the reviewer.

To be explicit about the endpoint rather than bumping indefinitely: if no owner decision lands by the time this stale window expires, I will close this PR myself — the reviewer's recommendation leans toward #145025's projection approach, and I don't want to keep an operator-semantics fork alive on life support. The issue #144959 it set out to fix remains open, and whichever direction is chosen, the failure-noise problem still deserves a fix.

If any maintainer wants this kept alive past that, a single reply (or the no-stale label) will do.

@openclaw-barnacle openclaw-barnacle Bot removed the stale Marked as stale due to inactivity label Oct 25, 2026
@openclaw-barnacle

Copy link
Copy Markdown

This pull request has been automatically marked as stale after 7 days without qualifying human activity.
Without new human activity, Barnacle may close it as not planned 7 days after this message was posted. This is administrative inactivity cleanup, not a claim that the problem is fixed.
A human update restarts the inactivity clock; a new PR head also restarts it. A maintainer can add no-stale for a durable hold, or bad-barnacle to suppress Barnacle. Remove the hold when work can resume.

@openclaw-barnacle openclaw-barnacle Bot added the stale Marked as stale due to inactivity label Nov 2, 2026
@openclaw-barnacle

Copy link
Copy Markdown

Closing due to inactivity.
If you believe this PR should be revived, post in #clawtributors on Discord to talk to a maintainer.
That channel is the escape hatch for high-quality PRs that get auto-closed.

@openclaw-barnacle openclaw-barnacle Bot closed this Nov 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 compatibility 🚨 May break existing users, config, migrations, defaults, or upgrade paths. P2 Normal backlog priority with limited blast radius. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: S stale Marked as stale due to inactivity status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant