Skip to content

fix: let target:none heartbeats recover from stale pending delivery#84679

Closed
extrasmall0 wants to merge 1 commit into
openclaw:mainfrom
extrasmall0:fix/heartbeat-target-none-stale-pending
Closed

fix: let target:none heartbeats recover from stale pending delivery#84679
extrasmall0 wants to merge 1 commit into
openclaw:mainfrom
extrasmall0:fix/heartbeat-target-none-stale-pending

Conversation

@extrasmall0

@extrasmall0 extrasmall0 commented May 20, 2026

Copy link
Copy Markdown
Contributor

Fixes #84672.

This keeps internal-only heartbeats from replaying stale pending-final-delivery text forever.

What changed

  • add a narrow heartbeat reply option to skip stale pending replay
  • use that option only when the heartbeat delivery target resolves to none
  • clear the stale pending-final-delivery fields before continuing with a fresh heartbeat run
  • add regression coverage for the skip path

Real behavior proof

Reproduced the stale replay flow with a heartbeat run whose resolved delivery target is none while pendingFinalDelivery* fields are still populated from an earlier pass.

Before this patch:

  1. a prior run leaves pendingFinalDeliveryText / metadata populated
  2. a later target:none heartbeat re-enters the fast path
  3. the runner reuses the stale pending-final-delivery payload instead of starting clean
  4. the same internal-only heartbeat text can be replayed again even though it no longer matches the current run

With this patch:

  1. the target:none path explicitly skips stale pending replay
  2. the old pending-final-delivery fields are cleared before the new heartbeat continues
  3. the heartbeat proceeds with a fresh reply computation instead of re-emitting the stale text

That keeps target:none heartbeats recoverable after a stale pending-delivery state while leaving non-none delivery targets on the existing path.

Validation

  • pnpm vitest src/auto-reply/reply/get-reply.fast-path.test.ts src/infra/heartbeat-runner.returns-default-unset.test.ts

@openclaw-barnacle openclaw-barnacle Bot added size: S triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup. labels May 20, 2026
@clawsweeper

clawsweeper Bot commented May 20, 2026

Copy link
Copy Markdown
Contributor

Thanks for the context here. I did a careful shell check against current main, and this is already implemented.

Current main already fixes the central heartbeat failure by no longer replaying non-ack pending-final-delivery text on heartbeat runs, with regression coverage; the remaining branch-specific clear path is stale against main and carries session/message-delivery risk.

So I’m closing this as already implemented rather than keeping a duplicate issue open.

Review details

Best possible solution:

Close this branch, keep the current-main no-replay implementation, and let broader bounded cleanup continue through #85770 if maintainers want additional stale-state clearing.

Do we have a high-confidence way to reproduce the issue?

Yes. The v2026.5.27 source still returned stale pending heartbeat replay text, the discussion includes a field repro of the shipped target:none stuck state, and current-main tests now cover the no-replay behavior.

Is this the best way to solve the issue?

Yes for current main, not for this PR branch. Current main solves the recovery path by suppressing heartbeat replay while preserving non-ack pending state; any additional cleanup should be scoped to heartbeat-owned stale state or handled by the bounded replay follow-up.

Security review:

Security review cleared: No dependency, workflow, secret, permission, or external code-execution surface changes were found; the remaining concern is functional session/message delivery risk.

AGENTS.md: found and applied where relevant.

What I checked:

Likely related people:

  • Lior Balmas: Git blame attributes the current-main heartbeat pending-final-delivery no-replay implementation and adjacent regression coverage to commit 492105d. (role: recent area contributor; confidence: high; commits: 492105db5a36; files: src/auto-reply/reply/get-reply.ts, src/infra/heartbeat-runner.skips-busy-session-lane.test.ts)
  • agocs: The related send-success cleanup PR covers the same pendingFinalDelivery* heartbeat state and has detailed real-environment proof for the broader stuck heartbeat cluster. (role: adjacent bugfix author; confidence: medium; commits: c67031be4b05; files: src/infra/heartbeat-runner.ts)
  • NianJiuZst: The open bounded-replay PR is the current related path for generic stale pending-final-delivery cleanup beyond the target-none recovery case. (role: adjacent follow-up owner; confidence: medium; commits: 887aa2c3bb0d; files: src/auto-reply/reply/get-reply.ts, src/auto-reply/reply/get-reply.fast-path.test.ts)

Codex review notes: model gpt-5.5, reasoning high; reviewed against 76130fd98873; fix evidence: commit 492105db5a36, main fix timestamp 2026-05-28T10:57:03-07:00.

@clawsweeper clawsweeper Bot added rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. P1 High-priority user-facing bug, regression, or broken workflow. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. labels May 20, 2026
@clawsweeper

clawsweeper Bot commented May 20, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper PR egg

🎁 Pass real behavior proof to wake the egg and unlock a hatchable treat.

Where did the egg go?
  • The egg game starts only after the PR passes the real-behavior proof check.
  • Before that, no creature or rarity is rolled. The treat waits for real proof.
  • This is still just collectible flavor: proof affects review readiness, not creature quality.

@clawsweeper clawsweeper Bot added the merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. label May 22, 2026
@extrasmall0

Copy link
Copy Markdown
Contributor Author

The overlap with #83187 is real, but I don't think they're duplicates.

#83187 clears pendingFinalDelivery* after a successful heartbeat send. This PR covers an earlier failure mode: when a later heartbeat resolves to target:none, stale pendingFinalDelivery* state can still short-circuit reply selection before any send path runs.

That is why this patch touches both:

  • src/auto-reply/reply/get-reply.ts to let the target:none recovery path skip stale pending replay
  • src/infra/heartbeat-runner.ts to clear the stale pending fields before the fresh heartbeat continues

So the split is:

  • #83187: post-send cleanup after a successful delivery
  • this PR: pre-send recovery when there is no delivery target and stale pending state is blocking a fresh reply

I added the reproduction and before/after behavior notes to the PR body for this exact branch because the bug shows up before the broader send-success cleanup in #83187 would help.

@dertbv

dertbv commented May 27, 2026

Copy link
Copy Markdown

Field repro of the target:none stuck-pending state on 2026.5.22

Posting a wild-environment reproduction of the exact branch this PR fixes. This is unpatched-build evidence — I can't supply patched-build recovery proof without applying the diff in production — but it strengthens both #83184 and the merge case for this PR by confirming the target:none skip path is what's actively biting users on shipped builds.

Setup

  • OpenClaw 2026.5.22 (a374c3a), npm-global install
  • macOS 26 arm64 (Mac Mini M4 Pro), launchd-managed gateway, 24/7
  • agents.defaults.heartbeat enabled, every: 30m, target: none, model openai/gpt-5.4-mini
  • Non-isolated heartbeat session keyed agent:main:heartbeat — the exact shared-session shape the PR targets

What happened

  1. Substantive heartbeat reply at 2026-05-22T14:58:52Z. Model-backed heartbeat_respond call set notify: true with a real notification payload. Captured in ~/.openclaw/clawlens/audit.jsonl (last successful heartbeat_respond entry on this session).
  2. pendingFinalDelivery: true was written and never cleared. From that moment on every 30-min tick short-circuited.
  3. 96 hours of silent skip behavior. openclaw system heartbeat last --json returned status: skipped, reason: target-none with the stale 2026-05-22 text as the latest preview. [heartbeat] started log lines kept firing on restart, but no heartbeat_respond tool call ran on the heartbeat session for the entire 4-day window. This is the exact target:none fast-path / pending-replay collision the PR's "what changed" bullets describe.
  4. Recovered by clearing the pending-final-delivery fields on disk and bouncing the gateway. Backed up ~/.openclaw/agents/main/sessions/sessions.json, removed the 8 pendingFinalDelivery* keys from the agent:main:heartbeat entry, restarted. Runner immediately resumed normal 30-min ticks; 31h later, zero drift toward stuck.

sessions.json state at recovery time

agent:main:heartbeat:
  pendingFinalDelivery: true
  pendingFinalDeliveryCreatedAt:        1779461937666  # 2026-05-22T14:58:57Z
  pendingFinalDeliveryLastAttemptAt:    1779796784893  # 2026-05-26T11:59:44Z
  pendingFinalDeliveryAttemptCount:     186
  pendingFinalDeliveryLastError:        null
  pendingFinalDeliveryText:             <2026-05-22 notification text, replayed>

That's 186 silent skip attempts on the target:none path against a single stale delivery payload — the failure mode the patch's skipHeartbeatPendingFinalDeliveryReplay flag exists to break.

Notes for the maintainer concern

ClawSweeper's merge-risk: 🚨 message-delivery flag asks for the clear to be scoped to heartbeat-owned stale state. In this repro that scoping question is moot: agent:main:heartbeat is the dedicated heartbeat session key (separate from agent:main:main), there is no user-visible pending reply living on this session, and the stale text was provably heartbeat-origin (originated from heartbeat_respond). If the PR's clear is gated to the heartbeat session key as the patch does today, it cannot erase a non-heartbeat pending delivery on this code path — at least for the target:none + non-isolated shared-heartbeat-session configuration this PR targets.

What this does not prove

I have not applied or run the patched build, so I can't supply post-patch recovery logs. What I can confirm: the unpatched target:none path is the active failure mode on shipped 2026.5.22, the stuck state is durable (96h, 186 skip attempts), and on-disk clear of the same fields the PR clears in-code is what restored service. If a maintainer wants the longer-form session-store diff or the redacted gateway.log window covering the stop, happy to attach.

Cross-ref: #83184 (root issue), #83187 (post-send cleanup, complementary not duplicate per @extrasmall0's comment above), #85770 (alternative bound-replay approach).

@extrasmall0

Copy link
Copy Markdown
Contributor Author

The new field repro above changes the proof picture enough that this should get another pass.

It confirms the exact shipped target:none failure mode on the dedicated agent:main:heartbeat session key, shows the stuck state persisting for 186 skip attempts over 96 hours, and shows that clearing the same stale pendingFinalDelivery* fields restored normal heartbeats. That directly addresses the "is this happening in the wild" part of the proof gap.

On the scope concern: for this path, the pending state lives on the heartbeat session key rather than the user chat session, so the clear is not crossing into unrelated user-visible final-delivery state in the reproduced setup.

@clawsweeper re-review

@clawsweeper

clawsweeper Bot commented May 28, 2026

Copy link
Copy Markdown
Contributor

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: the existing ClawSweeper review comment will be edited in place when the review finishes.

Re-review progress:

@clawsweeper

clawsweeper Bot commented May 28, 2026

Copy link
Copy Markdown
Contributor

ClawSweeper applied the proposed close for this PR.

@clawsweeper clawsweeper Bot closed this May 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

merge-risk: 🚨 message-delivery 🚨 May drop, duplicate, misroute, suppress, or wrongly target messages. merge-risk: 🚨 session-state 🚨 May lose, corrupt, stale, or mis-associate session, agent, or context state. P1 High-priority user-facing bug, regression, or broken workflow. rating: 🧂 unranked krab Not merge-ready due to missing proof or serious correctness/safety concerns. size: S status: 📣 needs proof The PR needs real behavior proof before ClawSweeper can clear the contributor ask. triage: needs-real-behavior-proof Candidate: external PR needs after-fix proof from a real setup.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: HEARTBEAT gets stuck

2 participants