fix #85871: [Bug]: Heartbeat scheduler silently fails to fire on 5.20 and all 5.x versions (regression from 4.23)#88970
Conversation
|
Codex review: needs maintainer review before merge. Reviewed June 15, 2026, 6:09 AM ET / 10:09 UTC. Summary PR surface: Source +29, Tests +64. Total +93 across 3 files. Reproducibility: yes. at source level: current main and Review metrics: none identified. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Risk before merge
Maintainer options:
Next step before merge
Security Review detailsBest possible solution: Land the focused scheduler-boundary fix after maintainer review and current-head CI, preserving the existing Do we have a high-confidence way to reproduce the issue? Yes, at source level: current main and Is this the best way to solve the issue? Yes: the heartbeat scheduler is the narrow owner for deciding whether background heartbeat work dispatches or retries, and the PR uses the existing retryable busy contract rather than adding provider-specific retries, new config, or a downstream fallback. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 0314819f918a. Label changesLabel justifications:
Evidence reviewedPR surface: Source +29, Tests +64. Total +93 across 3 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
|
5b170c6 to
ef2952c
Compare
|
@vincentkoc, PR's been with you since last week. ClawSweeper says ready. Mind taking a peek? Thanks! |
|
@joshavant when you have time, could you sanity-check #88970? It is a small heartbeat scheduler fix for #85871. The PR makes active embedded-run / reply-admission races return retryable Current state: |
e331da8 to
224b0c8
Compare
|
Maintainer verification complete for
Land-ready. The reporter's exact Ubuntu/Ollama installation was not reproduced; the repaired invariant is provider-independent heartbeat admission state. |
… on 5.20 and all 5.x versions (regression from 4.23) (openclaw#88970) * fix heartbeat deferral during active embedded runs * fix heartbeat admission busy retry * fix(heartbeat): bind retry to local admission --------- Co-authored-by: Vincent Koc <[email protected]>
… on 5.20 and all 5.x versions (regression from 4.23) (openclaw#88970) * fix heartbeat deferral during active embedded runs * fix heartbeat admission busy retry * fix(heartbeat): bind retry to local admission --------- Co-authored-by: Vincent Koc <[email protected]>
26194 drift commits, 222 flagged (security/regression keywords). Notable: XSS fix (openclaw#83104), PATH injection (openclaw#73264), npm_execpath injection (openclaw#73262), implicit tool grant fix (openclaw#75055), payment credential redaction (openclaw#75230), heartbeat regression (openclaw#88970).
Summary
runHeartbeatOncechecked queued lanes / active reply work before callinggetReplyFromConfig, but downstream reply admission intentionally rejects heartbeat turns when another reply run wins the same session slot. Because that admission rejection surfaced back to the heartbeat runner as an empty reply, the scheduler emittedok-empty/ returnedraninstead of the retryablerequests-in-flightresult. The existing active embedded-run path had the same root contract failure: background heartbeat work could enter a reply path that intentionally drops heartbeat turns rather than queueing them.getReplyFromConfigreturns an empty result so a reply-admission race is converted intorequests-in-flightinstead of a phantomran. That fixes the state-owner contract rather than masking the symptom with a provider-specific retry, timeout, or fallback.requests-in-flightstatus when heartbeat dispatch collides with active embedded/reply work, including the race where activity appears after preflight but before reply admission.requests-in-flightresult is the existing wake-layer retry contract, and the test-onlyreturn undefinedsimulates the actual reply-admission skip that previously became phantomran; it is not a production fallback, catch-all, or symptom-masking default.Linked context
Real behavior proof
Behavior addressed: the live gateway heartbeat scheduler starts and reaches agent/model dispatch, and scheduled heartbeats defer instead of being silently consumed while same-agent work is active.
Real environment tested: local OpenClaw gateway started through
pnpm gateway:devwith isolated runtime state, a shortevery: 2sheartbeat,target: none, and a real configured provider/model. The busy/retry boundary was exercised through the actual scheduler wake path with active embedded work registered foragent:main:main.Exact steps or command run after this patch:
daily-fix run-validation --name live-heartbeat-gateway-current -- bash heartbeat-live-gateway-proof.shdaily-fix run-validation --name heartbeat-busy-retry-current -- pnpm exec tsx heartbeat-scheduled-embedded-proof.tsEvidence after fix: live gateway log excerpt shows the gateway became ready, the scheduler emitted
[heartbeat] started, and a heartbeat turn reached agent dispatch:The busy/retry proof then showed the first scheduled wake deferred before dispatch while active embedded work existed, and the retry dispatched after the busy run cleared:
Observed result after fix: a real gateway heartbeat cycle no longer stayed silent after startup; it logged scheduler startup and reached agent/model dispatch on the configured interval. When same-agent embedded work was active, the scheduled wake returned the retryable
requests-in-flightresult before model dispatch, then retried and dispatched after the active run cleared.What was not tested: the reporter's exact Ubuntu/Ollama install and a full 5-minute interval / 10+ minute wall-clock wait. The local proof shortened the interval to exercise the same scheduler path without waiting multiple production cycles.
Tests and validation
src/infra/heartbeat-runner.skips-busy-session-lane.test.tscovers the reply-admission race where activity appears after heartbeat preflight andgetReplyFromConfigreturns an empty result;src/infra/heartbeat-runner.returns-default-unset.test.tscovers active embedded-run pre-dispatch deferral for the resolved heartbeat session and another same-agent session.requests-in-flightskip instead of phantomran, then the wake retry can dispatch after the active run clears.daily-fix run-validation --name heartbeat-busy-regression-current-head -- node scripts/run-vitest.mjs run src/infra/heartbeat-runner.skips-busy-session-lane.test.ts src/infra/heartbeat-runner.returns-default-unset.test.ts src/infra/heartbeat-wake.test.ts— passed: 3 files, 75 tests.daily-fix run-validation --name heartbeat-lint-current-head -- pnpm exec oxlint src/infra/heartbeat-runner.ts src/infra/heartbeat-runner.skips-busy-session-lane.test.ts— passed.daily-fix run-validation --name live-heartbeat-gateway-current -- bash heartbeat-live-gateway-proof.sh— passed; realpnpm gateway:devstartup loggedREADY=ok,[heartbeat] started, andHEARTBEAT_DISPATCH_AFTER_STARTED=1.daily-fix run-validation --name heartbeat-busy-retry-current -- pnpm exec tsx heartbeat-scheduled-embedded-proof.ts— passed; scheduledsource=interval/intent=scheduledheartbeat returnedrequests-in-flightwhileagent:main:mainhad active embedded work, then the wake retry reached model dispatch and returnedranafter the embedded run cleared.Risk checklist
Current review state
[P1]: addressed with current-head live gateway proof plus scheduler busy/retry proof.