Skip to content

[Bug]: failureAlert silently drops when cron stalls in runtime-plugins phase — lastFailureNotificationDeliveryStatus="unknown", operator never sees the failure #89847

Description

@BryceMurray

Environment

TL;DR

When a cron stalls in the gateway's runtime-plugins phase (the failure pattern in #87327), the failureAlert envelope attached to that cron also fails to deliver — lastFailureNotificationDeliveryStatus: "unknown" on every observed instance. The operator never sees an alert in the configured Telegram topic. Two real failures (a weekday morning digest, a weekly cohort refresh) went 5 days and 8 hours respectively without surfacing to the alert channel — discovered only by manually inspecting openclaw cron list and seeing error status.

The failureAlert config on the cron is correct (after: 1, mode: announce, channel: telegram, group id correct). The wiring works on other failure modes; it does not work for runtime-plugins stalls.

Symptom

Cron config (verified via openclaw cron get <id>):

{
  "name": "<redacted-cron-name>",
  "enabled": true,
  "payload": { "kind": "agentTurn", "lightContext": true, "timeoutSeconds": 300, ... },
  "delivery": {
    "mode": "announce", "channel": "telegram",
    "to": "<group-id>", "threadId": <topic-id>, "bestEffort": true
  },
  "failureAlert": {
    "after": 1, "mode": "announce",
    "channel": "telegram", "to": "<group-id>"
  },
  "state": {
    "lastRunStatus": "error",
    "lastDeliveryStatus": "unknown",
    "lastFailureNotificationDeliveryStatus": "unknown",
    "lastError": "cron: isolated agent run stalled before execution start (last phase: runtime-plugins)",
    "consecutiveErrors": 1
  }
}

Run-log entry shows the same on the cron-runs JSONL:

{
  "status": "error",
  "error": "cron: isolated agent run stalled before execution start (last phase: runtime-plugins)",
  "durationMs": 60511,
  "deliveryStatus": "unknown",
  "failureNotificationDelivery": { "status": "unknown" }
}

Both deliveryStatus (main result) and failureNotificationDelivery.status (alert) are "unknown" — the gateway never attempted, or attempted-but-failed, to deliver either.

Evidence (two crons, same shape)

Cron Schedule Last error Days silent Operator visibility before manual audit
weekday morning digest (sonnet-4-6) cron 0 6 * * 1-5 2026-06-03 06:03 CT 8h None
weekly cohort refresh (haiku-4-5) cron 0 22 * * 4 2026-05-29 22:00 CT 5 days None

Both have identical failureAlert shape; both produced identical lastFailureNotificationDeliveryStatus: "unknown"; neither surfaced to the alert topic.

The same workspace has a cron-health-monitor cron that does detect these failures the next day and posts an aggregate digest to a different topic. Operators watching only the per-cron alert channel see nothing. Operators watching only the per-cron target topic see nothing. The failure is detectable only by reading the aggregate digest in a third place — or by running openclaw cron list manually.

Why this is distinct from #87327

#87327 is the root cause (the runtime-plugins phase stalls). This issue is about the delivery layer's behavior when that root cause fires:

Hypothesis (low-confidence)

The failure-alert envelope construction may happen inside the agent runtime that's stalling. If true: the alert path can never fire from this kind of pre-execution failure because the code path that would fire the alert hasn't been initialized either. Fix would be to make failure-alert wiring a gateway-level concern that lives outside the per-cron agent runtime — when the cron-setup phase fails, gateway sends the alert directly using the cron's failureAlert config, no agent runtime involvement.

What's wrong

1. Alert path is dependent on the same runtime stack that's failing

If runtime-plugins fails, the alert can't fire. This means the alert mechanism is effectively useless for the highest-priority class of failures — those that prevent the worker from running at all.

2. "unknown" is a confusing delivery-status value

lastFailureNotificationDeliveryStatus: "unknown" doesn't distinguish between (a) tried-and-failed, (b) was-not-attempted, (c) was-acknowledged-but-receipt-unknown. Compare to the run-result's deliveryStatus: "delivered" on success — same field with a clean enum value when it works.

Suggest: explicit values "not_attempted", "attempted_failed", "sent_unconfirmed", "delivered". Operators can then tell why an alert didn't reach them.

3. failureAlert doesn't accept threadId (Telegram-specific observability gap)

Separately surfaced: the cron's delivery block accepts threadId (for routing to a specific Telegram topic), but the failureAlert block does not. Result: even when the alert path does deliver (e.g. on a non-runtime-plugins failure), it lands in the chat's default thread, not the topic the operator is watching. This may already be tracked elsewhere; if not, worth a separate issue.

Asks

  1. Decouple failure-alert delivery from the agent runtime. The cron scheduler / gateway knows the cron's failureAlert config and knows the run failed. It should be able to send a failure alert without re-initializing the agent runtime stack that just failed. This is the primary ask.

  2. Replace "unknown" with explicit delivery-attempt states. not_attempted vs attempted_failed vs delivered_unconfirmed — so operators can tell which path is broken.

  3. Add threadId support to failureAlert config. Mirror what delivery already accepts so per-cron alerts can route to the same topic the operator is watching, not the chat's default thread.

  4. (Operational, low-effort) Surface consecutiveErrors > 0 more prominently in openclaw cron list — bold or color the row. Today an error status takes a glance to spot in a list of 30+ crons.

Related


Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Normal backlog priority with limited blast radius.clawsweeper:needs-maintainer-reviewClawSweeper marked this issue as needing maintainer review before automation.clawsweeper:needs-product-decisionClawSweeper marked this issue as needing a product or behavior decision.clawsweeper:no-new-fix-prClawSweeper does not recommend queueing a new automated fix PR for this issue.clawsweeper:source-reproClawSweeper found a high-confidence source-level issue reproduction.impact:message-lossChannel message delivery can be lost, duplicated, or misrouted.issue-rating: 🦞 diamond lobsterVery strong issue quality with high-confidence source-level or clear reproduction.

    Type

    No type

    Fields

    Priority

    None yet

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions