feat(reminders): operator visibility into reminder failures and skips (#1494)#1503
Merged
Aaronontheweb merged 4 commits intoJun 26, 2026
Conversation
…netclaw-dev#1494) Reminder failures were invisible: a stuck/failing reminder only emitted a daemon log warning, so the gotowebinar incident skipped 49 fires in a day unnoticed. This surfaces failures where operators look. - Failure → destination channel: when a Channel-delivery reminder execution fails for any reason (including the netclaw-dev#1500 stall backstop, or auto-disable), a plain-language notice is posted to the reminder's own destination channel via a new IReminderChannelNotifier seam — interface in Netclaw.Actors (transport- agnostic, hands over a resolved ChannelDeliveryTargetInfo), implemented in Netclaw.Daemon over IChannelRegistry. Bounded by the auto-disable threshold, so no channel spam. Skips are NOT posted (too noisy) — only counted. - Skipped-duplicate visibility: the two skip sites (scheduled + deferred queue) now increment an in-memory per-reminder counter (since daemon start) instead of only logging. - `netclaw reminder status <id>`: new CLI subcommand + GET /api/reminders/{id}/ status REST endpoint, backed by a new GetReminderStatusQuery the manager answers from live state — enabled, in-flight, next fire, consecutive failures, skipped fires, and recent run history. The actor stays transport-agnostic: ReminderManagerActor resolves the channel target (same assembly) and the daemon notifier owns the channel post; a NullReminderChannelNotifier Null Object serves tests / no-channel daemons (a required dependency, not a nullable one). Tests: ReminderChannelFailureNotifier posts the right ChannelSendRequest (destination + DM kinds); status query returns per-reminder health and not-found. Full Netclaw.Actors.Tests (2482) and Netclaw.Daemon.Tests (754) green. Updates the netclaw-operations skill (scheduling reference, 2.19.0 → 2.20.0). Note: the behavioral eval suite (skill-content edit) needs a live model endpoint not available here — run in CI / before merge.
Verified findings from the workflow code review: - LlmSessionTestBase: register IReminderChannelNotifier (the new required reminder-manager dependency) so DI-resolved reminder managers in session integration tests initialize instead of dying with ActorInitializationException. - HandleGetStatusAsync: a transient read failure (schedule fetch / history IO) no longer impersonates "not found" — it faults the Ask (Status.Failure → the endpoint maps to 5xx) so the operator sees a real error instead of being told a wedged reminder was deleted. That was a silent fallback on the very feature meant to expose them. - Status query: read and display the same RecentHistoryCount (5) records, newest first — the CLI previously showed the OLDEST 5 of a last-10 window, hiding the most recent (failing) runs. - Channel notice: post the per-failure notice only below the auto-disable threshold; on the threshold-hitting failure the disabled notice already carries the last error, so both was double noise on the most important event. - NextFire: pass null through (FormatTimestamp renders null as "unknown"); the CLI shows "not scheduled" for disabled/unscheduled reminders. - Status handler runs its two independent backend reads concurrently (Task.WhenAll). Full Netclaw.Actors.Tests (2482) green; reminder + Daemon notifier tests green. Not changed (verified lower-severity / pre-existing patterns): whole-list scheduler fetch (matches existing GetReminder), notifier↔ChannelSendTools and CLI-subcommand scaffolding duplication.
Aaronontheweb
commented
Jun 26, 2026
| /// Fire-and-forget (<c>void</c>) — the manager must never block on channel | ||
| /// delivery, mirroring <see cref="Configuration.IOperationalNotificationSink"/>. | ||
| /// </summary> | ||
| public interface IReminderChannelNotifier |
| public void NotifyFailure(ChannelDeliveryTargetInfo target, string text) | ||
| => _ = PostAsync(target, text); | ||
|
|
||
| private async Task PostAsync(ChannelDeliveryTargetInfo target, string text) |
| // unbounded skip stream that #1494 makes visible via status instead. | ||
| if (count < FailurePauseThreshold) | ||
| { | ||
| PostFailureNoticeToChannel( |
Aaronontheweb
enabled auto-merge (squash)
June 26, 2026 17:05
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes #1494. Reminder failures were invisible to operators — only a buried daemon log line — so the gotowebinar reminder silently skipped 49 fires in a day before anyone noticed. This surfaces failures where operators actually look, and adds a status command.
Changes
1. Failure → destination channel (your primary ask)
When a
channel-delivery reminder execution fails for any reason — including the #1500 stall backstop recovering a wedged run, or hitting the auto-disable threshold — a plain-language notice is posted to the reminder's own destination channel, where the operator expects that reminder's output.IReminderChannelNotifierseam: interface inNetclaw.Actors(transport-agnostic — the manager hands over an already-resolvedChannelDeliveryTargetInfo), implemented inNetclaw.DaemonoverIChannelRegistry. Keeps the actor boundary transport-agnostic per the constitution.NullReminderChannelNotifierNull Object for tests / no-channel daemons — a required dependency (per the new "no optional/nullable deps" rule from fix(tools): allow autonomous sessions to write the workspace directory (#1493) #1498), not a nullable one.2. Skipped-duplicate visibility
The two skip sites (scheduled fire + deferred-queue dequeue) now increment an in-memory per-reminder counter instead of only logging — so the silent-skip pattern is finally countable.
3.
netclaw reminder status <id>New CLI subcommand +
GET /api/reminders/{id}/statusREST endpoint, backed by a newGetReminderStatusQuerythe manager answers from live state: enabled, in-flight right now, next fire, consecutive failures, skipped-fire count (since daemon start), and recent run history. The at-a-glance "is this reminder healthy or silently failing?" view the incident lacked.How it fits with #1500
#1500 makes a wedged reminder recover (fail after ~20 min instead of forever). #1494 makes that recovery visible: the failure posts to the channel, and the skips that piled up in between are countable via
status. Together they close the incident.Layering
ReminderManagerActor(inNetclaw.Actors) resolves the channel target viaReminderExecutionActor.ResolveChannelDeliveryTarget(same assembly) and hands it to the notifier; the daemon-side notifier owns the actual channel post. The actor never references channel/transport types.Tests
ReminderChannelFailureNotifierTests: posts the correctChannelSendRequestfor bothdestinationanddirect_messagekinds (real fire-and-forget awaited via a capture signal, no sleeps).ReminderManagerActorTests: status query returns per-reminder health; unknown reminder → not-found.Netclaw.Actors.Tests(2482) andNetclaw.Daemon.Tests(754) green. Slopwatch clean (3 pre-existing warnings, unrelated files); headers verified.Docs
Updates the
netclaw-operationsskill (scheduling reference) with the failure-to-channel behavior and thereminder statuscommand; bumpsmetadata.version2.19.0 → 2.20.0.Gate not run locally
The behavioral eval suite (skill-content edit) needs a live model endpoint not configured here — should run in CI / before merge.