-
-
Notifications
You must be signed in to change notification settings - Fork 80.7k
[6.1] Cron run "failed" notifications fire during hot reload and during retries, causing alert fatigue #90595
Copy link
Copy link
Open
Labels
P2Normal backlog priority with limited blast radius.Normal backlog priority with limited blast radius.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Description
Metadata
Metadata
Assignees
Labels
P2Normal backlog priority with limited blast radius.Normal backlog priority with limited blast radius.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.ClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.ClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.ClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.ClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.Channel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.Very strong issue quality with high-confidence source-level or clear reproduction.
Type
Fields
Priority
None yet
1. Environment
latest)minimaxprovider alias)2. Summary
After upgrading from OpenClaw 5.12 to 6.1, cron run failure notifications became noisy. Investigation shows that most "failed" notifications are false positives — the actual isolated session is still running and completes minutes later, posting a second "completed" notification in the same scheduling window.
The pattern is consistent with three 5.26-introduced mechanisms that did not exist in 5.12:
cron.retry— defaultmaxAttempts: 3,backoffMs: [30000, 60000, 300000],retryOn: [rate_limit, overloaded, network, timeout, server_error]. In 5.12 there was no retry, a single fail was final.auth.cooldowns— defaultauthPermanentBackoffMinutes: 10after a transient failure. 5.12 had no cooldown concept.openclaw.jsonmtime and applies a hot reload within seconds. If a cron run is in flight when mtime changes, the run is reported asfailedvia Feishu, then the Gateway restarts, then a new isolated session resumes the same work and posts acompletednotification minutes later. In 5.12 there was no hot reload; mtime changes only mattered on next manual restart.The triple combination is the regression: 5.12 cron failures were terminal and unambiguous. 6.1 cron failures are now ambiguous, with a transient fail → retry → cooldown → possible hot-reload-interrupt all triggering their own notifications.
3. Reproduction (minimal)
openclaw.json(any whitespace edit — it just has to bump mtime).Cron job X failed: cron: job interrupted by gateway restartnotification.completednotification for the same job in the same scheduling window.User experience: two contradicting notifications for the same job. A user filtering "failed" alerts cannot tell whether the work was lost or whether the notification is a false positive.
4. Sample Gateway log (sanitized)
5. Expected behavior
For an interrupted-by-reload run, we would expect the cron delivery notification to describe the current state of the work (e.g.
interrupted,resuming, or suppressed while a new run is in flight) rather thanfailed. The user should not have to grep the Gateway log to determine whether a real failure occurred.6. Suggested fixes
a. Distinguish
interruptedfromfailedin cron delivery notifications. Surface a new outcome type so Feishu (and other channels) can render it differently from a real fail.b. Optionally resume in-place on hot reload rather than restart in a new isolated session, to avoid duplicate-notification noise and lost partial work.
c. Document the new
cron.retryandauth.cooldownsdefaults prominently in the 5.26 → 6.x upgrade notes so users upgrading from 5.12 are not surprised by the new retry traffic and cooldown behavior.d. Add a setting to opt out of config hot reload (e.g.
gateway.hotReload: false) for users who prefer the 5.12 "manual restart only" model.7. Workarounds in the meantime
openclaw.jsonwhile cron runs are in flight.openclaw cron set <job> --retry-disabledor similar to opt out of the new retry behavior.cron run listCLI.