Skip to content

fix(reminders): release the duplicate-execution guard when a Mode A reminder session wedges (#1492)#1500

Merged
Aaronontheweb merged 1 commit into
netclaw-dev:devfrom
Aaronontheweb:fix/1492-reminder-hang
Jun 26, 2026
Merged

fix(reminders): release the duplicate-execution guard when a Mode A reminder session wedges (#1492)#1500
Aaronontheweb merged 1 commit into
netclaw-dev:devfrom
Aaronontheweb:fix/1492-reminder-hang

Conversation

@Aaronontheweb

Copy link
Copy Markdown
Collaborator

What

Fixes #1492. A single stuck reminder execution silently blocked all future fires of that reminder — 49 reminder_skipped_duplicate_execution events in one day in the reported incident, never recovering.

Root cause (confirmed Mode A)

The gotowebinar reminder used Channel delivery (send_channel_message) — "Mode A". On that path ReminderExecutionActor runs its own session pipeline and concludes only when the pipeline emits TurnCompleted/Error. It had no execution-level ceiling (unlike WebhookExecutionActor, which sets a ReceiveTimeout at line 50). When the session wedged — degraded, stopped calling tools, went silent without ever reaching a terminal signal (the issue notes it was spawning code-analyst sub-agents) — the actor waited forever.

ReminderManagerActor clears _activeExecutions only on ReminderExecutionCompleted, so the duplicate-execution guard was held indefinitely and every later fire was skipped.

Existing tests already bracket this: Mode A completes promptly on a no-notification turn (Execution_fails_when_required_policy_and_no_notification_sent), and Mode B is bounded by the 1-hour DeliveryObservedTimeout (CurrentSession_delivery_required_fails_when_delivery_is_not_observed). The gap was a Mode A turn that never reaches a terminal signal at all.

Fix

Arm an inactivity ReceiveTimeout (ExecutionStallTimeout, default 20m — a static field mirroring DeliveryObservedTimeout so tests can shrink it; no new config/schema knob) on the Mode A path once the pipeline is initialized:

  • Every session output resets it, so it never preempts a live turn.
  • A terminal TurnCompleted/Error stops the actor first and disarms it.
  • If the session goes silent for the window, the actor concludes the run as failed, releasing the guard so the next fire runs.

Mode B (CurrentSession) is unchanged — it does not arm this backstop (already bounded by DeliveryObservedTimeout). WebhookExecutionActor already had the equivalent ceiling, so this just brings reminders to parity.

Test

New regression Mode_A_wedged_session_is_failed_by_stall_backstop_releasing_the_guard: drives a Mode A reminder whose pipeline emits one non-terminal output then goes silent, shrinks the timeout, and asserts ReminderExecutionCompleted(success: false) arrives via the backstop. Full Netclaw.Actors.Tests green (2480); slopwatch + headers pass.

Follow-up

#1494 makes this failure operator-visible (posts a notice to the destination channel) — without that, a stalled reminder now recovers but the operator still isn't told it failed. A manager-side Terminated watch (defense-in-depth for a child that crashes without reporting) is a possible separate hardening; not needed for this fix since the actor concludes via ReportAndStop.

…eminder session wedges (netclaw-dev#1492)

A Channel-delivery (Mode A) reminder runs its own session pipeline and concludes
only when that pipeline emits TurnCompleted/Error. ReminderExecutionActor had no
execution-level ceiling on this path (unlike WebhookExecutionActor, which has a
ReceiveTimeout), so when a session wedged — degraded, stopped calling tools, and
went silent without ever reaching a terminal signal (e.g. a hung sub-agent) —
the actor waited forever. Because ReminderManagerActor only clears
_activeExecutions when the child reports ReminderExecutionCompleted, the
duplicate-execution guard was held indefinitely and every subsequent fire was
silently skipped (reminder_skipped_duplicate_execution) — 49 skips in one day in
the reported incident, never recovering.

Fix: arm an inactivity ReceiveTimeout (ExecutionStallTimeout, default 20m — a
static field mirroring DeliveryObservedTimeout so tests can shrink it) on the
Mode A path once the pipeline is initialized. Every session output resets it; a
terminal TurnCompleted/Error stops the actor first and disarms it. If the
session goes silent for the window, the actor concludes the run as failed,
releasing the guard so the next fire runs. Mode B (CurrentSession) is unchanged
— its wait is already bounded by DeliveryObservedTimeout — so it does not arm
this backstop.

Regression test drives a Mode A reminder whose pipeline emits one non-terminal
output then goes silent, and asserts ReminderExecutionCompleted(success: false)
arrives via the (shrunk) stall backstop. Full Netclaw.Actors.Tests green (2480).
@Aaronontheweb Aaronontheweb added reminders Reminder scheduling, execution, and history reliability Retries, resilience, graceful degradation labels Jun 26, 2026

@Aaronontheweb Aaronontheweb left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

// terminal TurnCompleted/Error stops us first. If the session wedges
// and goes silent without a terminal signal, this fires and releases
// the duplicate-execution guard instead of hanging forever (#1492).
Context.SetReceiveTimeout(ExecutionStallTimeout);

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, simple fix.

@Aaronontheweb
Aaronontheweb merged commit 0d5298e into netclaw-dev:dev Jun 26, 2026
15 checks passed
@Aaronontheweb
Aaronontheweb deleted the fix/1492-reminder-hang branch June 26, 2026 03:15
Aaronontheweb added a commit that referenced this pull request Jun 26, 2026
…#1494) (#1503)

* feat(reminders): operator visibility into reminder failures and skips (#1494)

Reminder failures were invisible: a stuck/failing reminder only emitted a daemon
log warning, so the gotowebinar incident skipped 49 fires in a day unnoticed.
This surfaces failures where operators look.

- Failure → destination channel: when a Channel-delivery reminder execution
  fails for any reason (including the #1500 stall backstop, or auto-disable), a
  plain-language notice is posted to the reminder's own destination channel via a
  new IReminderChannelNotifier seam — interface in Netclaw.Actors (transport-
  agnostic, hands over a resolved ChannelDeliveryTargetInfo), implemented in
  Netclaw.Daemon over IChannelRegistry. Bounded by the auto-disable threshold, so
  no channel spam. Skips are NOT posted (too noisy) — only counted.

- Skipped-duplicate visibility: the two skip sites (scheduled + deferred queue)
  now increment an in-memory per-reminder counter (since daemon start) instead of
  only logging.

- `netclaw reminder status <id>`: new CLI subcommand + GET /api/reminders/{id}/
  status REST endpoint, backed by a new GetReminderStatusQuery the manager
  answers from live state — enabled, in-flight, next fire, consecutive failures,
  skipped fires, and recent run history.

The actor stays transport-agnostic: ReminderManagerActor resolves the channel
target (same assembly) and the daemon notifier owns the channel post; a
NullReminderChannelNotifier Null Object serves tests / no-channel daemons (a
required dependency, not a nullable one).

Tests: ReminderChannelFailureNotifier posts the right ChannelSendRequest
(destination + DM kinds); status query returns per-reminder health and not-found.
Full Netclaw.Actors.Tests (2482) and Netclaw.Daemon.Tests (754) green. Updates the
netclaw-operations skill (scheduling reference, 2.19.0 → 2.20.0).

Note: the behavioral eval suite (skill-content edit) needs a live model endpoint
not available here — run in CI / before merge.

* fix(reminders): address code-review findings on #1494

Verified findings from the workflow code review:

- LlmSessionTestBase: register IReminderChannelNotifier (the new required
  reminder-manager dependency) so DI-resolved reminder managers in session
  integration tests initialize instead of dying with ActorInitializationException.
- HandleGetStatusAsync: a transient read failure (schedule fetch / history IO)
  no longer impersonates "not found" — it faults the Ask (Status.Failure → the
  endpoint maps to 5xx) so the operator sees a real error instead of being told a
  wedged reminder was deleted. That was a silent fallback on the very feature
  meant to expose them.
- Status query: read and display the same RecentHistoryCount (5) records, newest
  first — the CLI previously showed the OLDEST 5 of a last-10 window, hiding the
  most recent (failing) runs.
- Channel notice: post the per-failure notice only below the auto-disable
  threshold; on the threshold-hitting failure the disabled notice already carries
  the last error, so both was double noise on the most important event.
- NextFire: pass null through (FormatTimestamp renders null as "unknown"); the CLI
  shows "not scheduled" for disabled/unscheduled reminders.
- Status handler runs its two independent backend reads concurrently (Task.WhenAll).

Full Netclaw.Actors.Tests (2482) green; reminder + Daemon notifier tests green.

Not changed (verified lower-severity / pre-existing patterns): whole-list
scheduler fetch (matches existing GetReminder), notifier↔ChannelSendTools and
CLI-subcommand scaffolding duplication.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

reliability Retries, resilience, graceful degradation reminders Reminder scheduling, execution, and history

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: autonomous reminder sessions permanently block duplicate guard due to missing session timeout

1 participant