Skip to content

[6.1] Cron run "failed" notifications fire during hot reload and during retries, causing alert fatigue #90595

Description

@cx306806112

1. Environment

  • OpenClaw version: 2026.6.1 (latest stable on npm dist-tag latest)
  • Upgraded from: 2026.5.12 (5.12) where this issue did not exist
  • Provider: an anthropic-messages-compatible LLM endpoint (M2.7 + M3 model endpoints, both via minimax provider alias)
  • Channel: Feishu (1 bot account)
  • Cron workload: ~30 cron jobs (5-minute to daily schedules), 2 isolated lane worker pool

2. Summary

After upgrading from OpenClaw 5.12 to 6.1, cron run failure notifications became noisy. Investigation shows that most "failed" notifications are false positives — the actual isolated session is still running and completes minutes later, posting a second "completed" notification in the same scheduling window.

The pattern is consistent with three 5.26-introduced mechanisms that did not exist in 5.12:

  1. cron.retry — default maxAttempts: 3, backoffMs: [30000, 60000, 300000], retryOn: [rate_limit, overloaded, network, timeout, server_error]. In 5.12 there was no retry, a single fail was final.
  2. auth.cooldowns — default authPermanentBackoffMinutes: 10 after a transient failure. 5.12 had no cooldown concept.
  3. Config hot reload on mtime change — 5.26 watches openclaw.json mtime and applies a hot reload within seconds. If a cron run is in flight when mtime changes, the run is reported as failed via Feishu, then the Gateway restarts, then a new isolated session resumes the same work and posts a completed notification minutes later. In 5.12 there was no hot reload; mtime changes only mattered on next manual restart.

The triple combination is the regression: 5.12 cron failures were terminal and unambiguous. 6.1 cron failures are now ambiguous, with a transient fail → retry → cooldown → possible hot-reload-interrupt all triggering their own notifications.

3. Reproduction (minimal)

  1. Configure a cron job calling the provider on a 3-5 minute schedule.
  2. Wait until a run is in flight.
  3. Touch openclaw.json (any whitespace edit — it just has to bump mtime).
  4. Watch Feishu within ~1s — you will receive a Cron job X failed: cron: job interrupted by gateway restart notification.
  5. Wait — the same cron job will complete successfully a few minutes later, posting a second completed notification for the same job in the same scheduling window.

User experience: two contradicting notifications for the same job. A user filtering "failed" alerts cannot tell whether the work was lost or whether the notification is a false positive.

4. Sample Gateway log (sanitized)

[diagnostic] lane task error: lane=cron-nested durationMs=134613
  error="FailoverError: The AI service is temporarily overloaded. Please try again in a moment."
[reload] config change detected; evaluating reload (cron)
[reload] config hot reload applied (cron)
[diagnostic] message processed: channel=cron chatId=unknown outcome=error duration=134666ms
  error="FallbackSummaryError: All models failed (2): minimax/MiniMax-M2.7: overloaded |
         minimax/MiniMax-M3: minimax is overloaded right now. ... (overloaded)"

5. Expected behavior

For an interrupted-by-reload run, we would expect the cron delivery notification to describe the current state of the work (e.g. interrupted, resuming, or suppressed while a new run is in flight) rather than failed. The user should not have to grep the Gateway log to determine whether a real failure occurred.

6. Suggested fixes

a. Distinguish interrupted from failed in cron delivery notifications. Surface a new outcome type so Feishu (and other channels) can render it differently from a real fail.
b. Optionally resume in-place on hot reload rather than restart in a new isolated session, to avoid duplicate-notification noise and lost partial work.
c. Document the new cron.retry and auth.cooldowns defaults prominently in the 5.26 → 6.x upgrade notes so users upgrading from 5.12 are not surprised by the new retry traffic and cooldown behavior.
d. Add a setting to opt out of config hot reload (e.g. gateway.hotReload: false) for users who prefer the 5.12 "manual restart only" model.

7. Workarounds in the meantime

  • Avoid touching openclaw.json while cron runs are in flight.
  • Consider openclaw cron set <job> --retry-disabled or similar to opt out of the new retry behavior.
  • Live with the noisy false-positive notifications and verify the actual work state via the cron run list CLI.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions