Skip to content

cron: failure alerts enabled with defaults (--failure-alert) silently stop firing after gateway restart #96589

Description

@yetval

Summary

Enabling cron failure alerts with defaults only (openclaw cron edit <job> --failure-alert with no other --failure-alert-* flags) works until the next gateway restart, then silently stops firing forever. The store codec persists the enabled-with-defaults state but cannot read it back, so on reload the job's failureAlert becomes undefined (indistinguishable from "no alert config"), and a failing job then fails invisibly.

Environment

  • Commit: 3ab8d6a (origin/main at time of filing)
  • Surface: cron SQLite store codec (src/cron/store/failure-alert-codec.ts), reached by the documented openclaw cron edit --failure-alert CLI path and the failure-alert resolver.

Steps to reproduce

  1. Enable per-job failure alerts with defaults only: openclaw cron edit <job> --failure-alert (no other --failure-alert-* flags). This sets patch.failureAlert = {} (an enabled-with-defaults config), confirmed at src/cli/cron-cli/register.cron-edit.ts:541 where failureAlertFlag === true with no fields leaves failureAlert as {}.
  2. Let a run fail enough times to trip the default threshold. The alert fires (in-memory failureAlert is {}, which the resolver treats as enabled).
  3. Restart the gateway (or otherwise reload the cron store from SQLite).
  4. Let the job fail again. No alert is ever delivered.

Expected

An enabled-with-defaults failure alert (failureAlert = {}) must survive a store round trip and keep firing after restart, exactly as it did in-memory.

Actual

After restart the job's failureAlert reloads as undefined. With no global cronConfig.failureAlert.enabled === true, the resolver returns null and no alert is ever requested. The job keeps failing silently.

Root cause

bindFailureAlertColumns writes the enabled-with-defaults state as failure_alert_disabled = 0 with every other column null:

// src/cron/store/failure-alert-codec.ts
return {
  failure_alert_disabled: failureAlert ? 0 : null,
  failure_alert_after: failureAlert?.after ?? null,
  ...
};

But failureAlertFromRow never reads that 0 sentinel. Its empty-check only inspects the other columns, so when they are all null it returns undefined regardless of failure_alert_disabled === 0:

// src/cron/store/failure-alert-codec.ts
export function failureAlertFromRow(row: CronJobRow): CronFailureAlert | false | undefined {
  if (row.failure_alert_disabled === 1) {
    return false;
  }
  if (
    row.failure_alert_after == null &&
    !row.failure_alert_channel &&
    !row.failure_alert_to &&
    row.failure_alert_cooldown_ms == null &&
    row.failure_alert_include_skipped == null &&
    !row.failure_alert_mode &&
    !row.failure_alert_account_id
  ) {
    return undefined;
  }
  ...
}

So three states collapse to two on read: disabled -> false (correct), populated -> populated (correct), but enabled-with-defaults ({}, written as failure_alert_disabled = 0) -> undefined (wrong, should be {}).

The runtime store served to the scheduler comes from this column path: loadedCronStoreFromRows builds store.jobs via rows.map(rowToCronJob), and rowToCronJob calls failureAlertFromRow (src/cron/store/row-codec.ts:218). The job_json sidecar that preserves {} is only used for configJobs (config write-back), not for the runtime alert decision. The behavioral gate is resolveFailureAlert (src/cron/service/failure-alerts.ts:73): if (!jobConfig && globalConfig?.enabled !== true) return null;. With jobConfig = {} it is truthy (alert active); with jobConfig = undefined it returns null (alert disabled).

A minimal fix is to treat failure_alert_disabled === 0 as the enabled sentinel and not fold it into the empty-check, so an empty enabled config reads back as {}.

Sibling surfaces

Real behavior proof

Behavior addressed: cron failure alert enabled with defaults only must survive a store round trip and keep firing after restart.
Real environment tested: real SQLite-backed cron store driven via saveCronJobsStore then loadCronJobsStore, and the real resolveFailureAlert policy resolver, at commit 3ab8d6a.
Exact steps or command run after this patch: round-trip three jobs (failureAlert: false, a populated config, and {}) through the real store, then call resolveFailureAlert on the in-memory {} job and on the reloaded job, with no global config set.
Evidence after fix:

# OBSERVED (buggy, origin/main 3ab8d6aa609ab7f1a6bee14500694bd7ca243489)
=== STORE ROUND TRIP (real SQLite-backed cron store) ===
failureAlert=false        -> reloaded: false
failureAlert={populated}  -> reloaded: {"after":3,"channel":"telegram","to":"ops"}
failureAlert={} (enabled, all defaults) -> reloaded: undefined

=== ALERT POLICY (real resolveFailureAlert, no global config) ===
in-memory job (failureAlert={})         -> alert active? true (after=2, channel=last)
reloaded job after restart (failureAlert=undefined) -> alert active? false

# EXPECTED (after treating failure_alert_disabled === 0 as the enabled sentinel, identical inputs)
=== STORE ROUND TRIP (real SQLite-backed cron store) ===
failureAlert=false        -> reloaded: false
failureAlert={populated}  -> reloaded: {"after":3,"channel":"telegram","to":"ops"}
failureAlert={} (enabled, all defaults) -> reloaded: {}

=== ALERT POLICY (real resolveFailureAlert, no global config) ===
in-memory job (failureAlert={})         -> alert active? true (after=2, channel=last)
reloaded job after restart (failureAlert={}) -> alert active? true

Observed result after fix: the enabled-with-defaults alert config collapses to undefined across a store round trip on current main, which silently disables the alert after restart; with the sentinel honored it round-trips to {} and the alert stays active.
What was not tested: the end-to-end gateway restart and the actual openclaw cron edit --failure-alert CLI invocation were not exercised in a live gateway; the proof drives the real store persistence path and the real resolver directly. Webhook-mode and global-config-enabled combinations were not exercised, though the same column collapse applies to any defaults-only enabled config.

Metadata

Metadata

Assignees

No one assigned

    Labels

    P1High-priority user-facing bug, regression, or broken workflow.clawsweeper:fix-shape-clearClawSweeper found a clear likely implementation shape for this issue.clawsweeper:queueable-fixClawSweeper marked this issue as an existing queue_fix_pr work candidate.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.no-staleExclude from stale automation

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions