feat(kernel/heartbeat): deliver HealthCheckFailed to notification.alert_channels#3218
Merged
houko merged 2 commits intoApr 26, 2026
Merged
Conversation
…rt_channels The heartbeat monitor already publishes `SystemEvent::HealthCheckFailed` to the internal event bus, but nothing routes those events to the operator notification layer. As a result, an unresponsive agent shows up only as a WARN log line — Telegram/Slack/Discord/etc. operators never see it, unlike `tool_failure` and `task_failed`, which are pushed to `notification.alert_channels` (and matching `notification.agent_rules`) via `push_notification`. This patch makes heartbeat alerts symmetric with the other failure-class notifications: 1. In `start_heartbeat_monitor`, on the *transition* to unresponsive (same gate that already de-dupes the WARN log and the event-bus publish), call `push_notification(agent_id, "health_check_failed", msg)` so configured channels receive a human-readable alert. 2. Extend the global-fallback match arm in `push_notification` to include `"health_check_failed"` alongside `task_completed` / `task_failed` / `tool_failure`, so a deployment with only a global `[notification.alert_channels]` block (no per-agent rules) still gets these alerts. `agent_rules` already filters by event-type string, so operators can route heartbeat alerts to a dedicated topic with: [[notification.agent_rules]] agent_pattern = "*" events = ["health_check_failed"] [[notification.agent_rules.channels]] channel_type = "telegram" recipient = "..." thread_id = "..." Recovery is left intentionally silent (no "agent recovered" notification) to avoid the noise that an N-agent deployment would generate during normal restart churn — the WARN/INFO log pair already covers it for log readers.
github-actions
Bot
added
size/S
houko
approved these changes
Apr 26, 2026
houko
left a comment
houko
left a comment
Contributor
There was a problem hiding this comment.
LGTM. Direction and implementation both correct — minimal diff, gated on the same transition boundary as the WARN log and event-bus publish, quiet-hours continue is upstream so silence still applies.
Two follow-ups worth tracking (not blocking this merge):
- Add a regression test pinning the transition behavior — one tick unresponsive → one push, second tick still unresponsive → no further push, recovery → no push. Prevents parallel work from silently disabling the wiring (per CLAUDE.md "integration tests at the injection site").
- The live test box in the PR body is still unchecked. Recommend running it post-merge against a real Telegram alert_channels topic before relying on this in prod.
Separate cleanup (out of scope here): docs/configuration/core/page.mdx is drifted from the actual NotificationConfig schema (kind/target/agent/targets vs real channel_type/recipient/agent_pattern/channels/events), and "health_check_failed" event_type isn't enumerated. Worth a separate docs PR.
github-actions
Bot
removed
the
ready-for-review
2 tasks
houko
added a commit
that referenced
this pull request
Apr 26, 2026
…heck_failed (#3222) Follow-up for #3218 — locks the global-fallback match arm so a future refactor cannot silently disable heartbeat alerts. Adds four pure routing tests against LibreFangKernel::push_notification: - health_check_failed falls back to alert_channels when no agent_rule matches - a matching agent_rule overrides alert_channels - no configured targets produce zero sends (silent, not panicking) - unknown event_type yields zero sends (typo guard — never page on unrecognized event_type) Reuses the RecordingChannelAdapter pattern already used by test_notify_escalated_approval_prefers_request_route_to. No production code changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
SystemEvent::HealthCheckFailed, but operators onnotification.alert_channelsnever see unresponsive-agent alerts — onlytool_failureandtask_failedare routed there. This patch makes heartbeat symmetric.kernel/mod.rs:start_heartbeat_monitor, on the unresponsive-transition gate that already de-dupes the WARN log and the event-bus publish, additionally callpush_notification(agent_id, "health_check_failed", msg).push_notificationto include"health_check_failed"alongsidetask_completed/task_failed/tool_failure, so a deployment with only a bare[notification.alert_channels]block (noagent_rules) still receives heartbeat alerts.Routing example
Operators who want a dedicated topic/channel for heartbeats can now use:
Or they can rely on the global fallback by leaving
agent_rulesempty and configuring a single[notification.alert_channels]block.Design notes
"Agent recovered from unresponsive state"is sufficient for log-based observability.known_unresponsive.insert()gate that protects the WARN log and the event-bus publish, so we get one notification per transition (not one per heartbeat tick while the agent stays down).is_quiet_hours) still applies because the new push is downstream of that early-continue.Test plan
cargo build -p librefang-kernel --libcargo test -p librefang-kernel --lib heartbeat— 10 passed (existing tests unchanged).cargo clippy -p librefang-kernel --lib --all-targets -- -D warnings— clean.[notification.alert_channels]to point at a Telegram topic, force an autonomous agent into an unresponsive state, confirm a single message arrives in the topic and a single message arrives on recovery (well — none, by design).Follow-up worth tracking separately
AutonomousConfig.heartbeat_channel: Option<String>is declared but unused. With this PR, per-agent routing is available vianotification.agent_rulesinstead. The vestigial field could be either wired up or removed; not in scope for this PR. Tracked in #3217.