fix(reminders): release the duplicate-execution guard when a Mode A reminder session wedges (#1492)#1500
Merged
Aaronontheweb merged 1 commit intoJun 26, 2026
Conversation
…eminder session wedges (netclaw-dev#1492) A Channel-delivery (Mode A) reminder runs its own session pipeline and concludes only when that pipeline emits TurnCompleted/Error. ReminderExecutionActor had no execution-level ceiling on this path (unlike WebhookExecutionActor, which has a ReceiveTimeout), so when a session wedged — degraded, stopped calling tools, and went silent without ever reaching a terminal signal (e.g. a hung sub-agent) — the actor waited forever. Because ReminderManagerActor only clears _activeExecutions when the child reports ReminderExecutionCompleted, the duplicate-execution guard was held indefinitely and every subsequent fire was silently skipped (reminder_skipped_duplicate_execution) — 49 skips in one day in the reported incident, never recovering. Fix: arm an inactivity ReceiveTimeout (ExecutionStallTimeout, default 20m — a static field mirroring DeliveryObservedTimeout so tests can shrink it) on the Mode A path once the pipeline is initialized. Every session output resets it; a terminal TurnCompleted/Error stops the actor first and disarms it. If the session goes silent for the window, the actor concludes the run as failed, releasing the guard so the next fire runs. Mode B (CurrentSession) is unchanged — its wait is already bounded by DeliveryObservedTimeout — so it does not arm this backstop. Regression test drives a Mode A reminder whose pipeline emits one non-terminal output then goes silent, and asserts ReminderExecutionCompleted(success: false) arrives via the (shrunk) stall backstop. Full Netclaw.Actors.Tests green (2480).
Aaronontheweb
commented
Jun 26, 2026
| // terminal TurnCompleted/Error stops us first. If the session wedges | ||
| // and goes silent without a terminal signal, this fires and releases | ||
| // the duplicate-execution guard instead of hanging forever (#1492). | ||
| Context.SetReceiveTimeout(ExecutionStallTimeout); |
Collaborator
Author
There was a problem hiding this comment.
LGTM, simple fix.
This was referenced Jun 26, 2026
Merged
Aaronontheweb
added a commit
that referenced
this pull request
Jun 26, 2026
…#1494) (#1503) * feat(reminders): operator visibility into reminder failures and skips (#1494) Reminder failures were invisible: a stuck/failing reminder only emitted a daemon log warning, so the gotowebinar incident skipped 49 fires in a day unnoticed. This surfaces failures where operators look. - Failure → destination channel: when a Channel-delivery reminder execution fails for any reason (including the #1500 stall backstop, or auto-disable), a plain-language notice is posted to the reminder's own destination channel via a new IReminderChannelNotifier seam — interface in Netclaw.Actors (transport- agnostic, hands over a resolved ChannelDeliveryTargetInfo), implemented in Netclaw.Daemon over IChannelRegistry. Bounded by the auto-disable threshold, so no channel spam. Skips are NOT posted (too noisy) — only counted. - Skipped-duplicate visibility: the two skip sites (scheduled + deferred queue) now increment an in-memory per-reminder counter (since daemon start) instead of only logging. - `netclaw reminder status <id>`: new CLI subcommand + GET /api/reminders/{id}/ status REST endpoint, backed by a new GetReminderStatusQuery the manager answers from live state — enabled, in-flight, next fire, consecutive failures, skipped fires, and recent run history. The actor stays transport-agnostic: ReminderManagerActor resolves the channel target (same assembly) and the daemon notifier owns the channel post; a NullReminderChannelNotifier Null Object serves tests / no-channel daemons (a required dependency, not a nullable one). Tests: ReminderChannelFailureNotifier posts the right ChannelSendRequest (destination + DM kinds); status query returns per-reminder health and not-found. Full Netclaw.Actors.Tests (2482) and Netclaw.Daemon.Tests (754) green. Updates the netclaw-operations skill (scheduling reference, 2.19.0 → 2.20.0). Note: the behavioral eval suite (skill-content edit) needs a live model endpoint not available here — run in CI / before merge. * fix(reminders): address code-review findings on #1494 Verified findings from the workflow code review: - LlmSessionTestBase: register IReminderChannelNotifier (the new required reminder-manager dependency) so DI-resolved reminder managers in session integration tests initialize instead of dying with ActorInitializationException. - HandleGetStatusAsync: a transient read failure (schedule fetch / history IO) no longer impersonates "not found" — it faults the Ask (Status.Failure → the endpoint maps to 5xx) so the operator sees a real error instead of being told a wedged reminder was deleted. That was a silent fallback on the very feature meant to expose them. - Status query: read and display the same RecentHistoryCount (5) records, newest first — the CLI previously showed the OLDEST 5 of a last-10 window, hiding the most recent (failing) runs. - Channel notice: post the per-failure notice only below the auto-disable threshold; on the threshold-hitting failure the disabled notice already carries the last error, so both was double noise on the most important event. - NextFire: pass null through (FormatTimestamp renders null as "unknown"); the CLI shows "not scheduled" for disabled/unscheduled reminders. - Status handler runs its two independent backend reads concurrently (Task.WhenAll). Full Netclaw.Actors.Tests (2482) green; reminder + Daemon notifier tests green. Not changed (verified lower-severity / pre-existing patterns): whole-list scheduler fetch (matches existing GetReminder), notifier↔ChannelSendTools and CLI-subcommand scaffolding duplication.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes #1492. A single stuck reminder execution silently blocked all future fires of that reminder — 49
reminder_skipped_duplicate_executionevents in one day in the reported incident, never recovering.Root cause (confirmed Mode A)
The
gotowebinarreminder used Channel delivery (send_channel_message) — "Mode A". On that pathReminderExecutionActorruns its own session pipeline and concludes only when the pipeline emitsTurnCompleted/Error. It had no execution-level ceiling (unlikeWebhookExecutionActor, which sets aReceiveTimeoutat line 50). When the session wedged — degraded, stopped calling tools, went silent without ever reaching a terminal signal (the issue notes it was spawning code-analyst sub-agents) — the actor waited forever.ReminderManagerActorclears_activeExecutionsonly onReminderExecutionCompleted, so the duplicate-execution guard was held indefinitely and every later fire was skipped.Existing tests already bracket this: Mode A completes promptly on a no-notification turn (
Execution_fails_when_required_policy_and_no_notification_sent), and Mode B is bounded by the 1-hourDeliveryObservedTimeout(CurrentSession_delivery_required_fails_when_delivery_is_not_observed). The gap was a Mode A turn that never reaches a terminal signal at all.Fix
Arm an inactivity
ReceiveTimeout(ExecutionStallTimeout, default 20m — a static field mirroringDeliveryObservedTimeoutso tests can shrink it; no new config/schema knob) on the Mode A path once the pipeline is initialized:TurnCompleted/Errorstops the actor first and disarms it.Mode B (CurrentSession) is unchanged — it does not arm this backstop (already bounded by
DeliveryObservedTimeout).WebhookExecutionActoralready had the equivalent ceiling, so this just brings reminders to parity.Test
New regression
Mode_A_wedged_session_is_failed_by_stall_backstop_releasing_the_guard: drives a Mode A reminder whose pipeline emits one non-terminal output then goes silent, shrinks the timeout, and assertsReminderExecutionCompleted(success: false)arrives via the backstop. FullNetclaw.Actors.Testsgreen (2480); slopwatch + headers pass.Follow-up
#1494 makes this failure operator-visible (posts a notice to the destination channel) — without that, a stalled reminder now recovers but the operator still isn't told it failed. A manager-side
Terminatedwatch (defense-in-depth for a child that crashes without reporting) is a possible separate hardening; not needed for this fix since the actor concludes viaReportAndStop.