Skip to content

feat(runtime): emit agent-loop exit-reason metric (#6227)#6232

Merged
houko merged 2 commits into
mainfrom
feat/6227-agent-loop-exit-metric
Jun 19, 2026
Merged

feat(runtime): emit agent-loop exit-reason metric (#6227)#6232
houko merged 2 commits into
mainfrom
feat/6227-agent-loop-exit-metric

Conversation

@houko

@houko houko commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

Summary

Emits a counter at every agent-loop termination so operators can finally alert on non-success exits.
Before this, the agent loop recorded no metric when it aborted on repeated tool failures, max iterations, a loop-guard circuit break, or a provider content-filter — the only signal was reading transcripts.
A cron / trigger fire that aborted on repeated tool failures still recorded librefang_cron_fires_total{outcome="ok"} because the loop returned (did not throw).

New metric:

librefang_agent_loop_exits_total{agent, reason}

Acceptance alert (per #6227): rate(librefang_agent_loop_exits_total{reason!="completed"}) > 0 per agent.

Reason set

Matched to the real termination branches in run_streaming.rs, mod.rs, and loop_guard.rs:

  • completed — any Ok(_) return: a finalized EndTurn reply, a silent completion (NO_REPLY / cascade-leak / progress-leak / mid-stream cascade-leak abort), a MaxTokens partial return, an interrupt-cancel early return, or the provider-not-configured early return.
  • max_iterationsErr(MaxIterationsExceeded): the for loop ran out.
  • repeated_tool_failuresErr(RepeatedToolFailures): consecutive_all_failed >= MAX_CONSECUTIVE_ALL_FAILED.
  • circuit_break — loop-guard global circuit breaker tripped.
  • content_filteredErr(ContentFiltered): provider safety / content filter blocked the reply.
  • error — any other propagated Err (LLM call failure, memory error, etc.).

Deviations from the issue's suggested set (documented per the task)

  • No empty_response reason. The issue listed it, but there is no distinct empty-response termination branch: an empty LLM response is a one-shot in-loop retry (EndTurnRetry::EmptyResponsecontinue); the turn then either retries to a real reply or finalizes via finalize_end_turn_text, both of which land on completed.
  • Added content_filtered. ContentFiltered is a real, distinct terminal Err branch operators will want to separate from generic error.

How exactly-one increment is guaranteed

Structural, not by discipline at each site:

  • run_agent_loop and run_agent_loop_streaming become thin instrumented wrappers (the #[instrument] span name and fields are unchanged, so the instrument_span_fields / CLI span-name tests stay green).
  • Each wrapper calls its *_inner body once and feeds the single returned Result to record_agent_loop_exit(agent, &result).
  • classify_exit_reason maps that one Result to one reason; the counter is incremented once.
  • No terminal site inside the loop touches the metric, so there is no path where one branch falls through into another and double-counts. ?-propagated exits (circuit-break, LLM/memory errors) are caught at the wrapper too — they would have been missed by per-return-site increments.

Circuit-break surfaces as Err(Internal(msg)) propagated up through the ? on execute_single_tool_call. It is classified by matching the shared loop_guard::CIRCUIT_BREAKER_MSG_PREFIX constant — a single source of truth now referenced by both the producer (LoopGuard::check) and the classifier, so the match is compiler-anchored rather than a loose duplicated string literal.

Label cardinality

The agent label is sanitized (control chars stripped, length capped at 64) by sanitize_agent_label, mirroring sanitize_tool_label in tool_call.rs and the bounded-cardinality contract on record_cron_fire_metric in the kernel's cron.rs.

Changes

  • crates/librefang-runtime/src/agent_loop/mod.rssanitize_agent_label, classify_exit_reason, record_agent_loop_exit; run_agent_loop split into instrumented wrapper + run_agent_loop_inner.
  • crates/librefang-runtime/src/agent_loop/run_streaming.rsrun_agent_loop_streaming split into instrumented wrapper + run_agent_loop_streaming_inner, calling super::record_agent_loop_exit.
  • crates/librefang-runtime/src/loop_guard.rs — extract CIRCUIT_BREAKER_MSG_PREFIX const; use it in the circuit-break message.
  • crates/librefang-runtime/src/agent_loop/tests/utilities.rs — three injection-site tests (below).
  • CHANGELOG.md[Unreleased] / Added entry.

Tests

Added at the injection site (agent_loop::tests::utilities):

  • test_classify_exit_reason_covers_every_branch — pins each reason mapping (completed, max_iterations, repeated_tool_failures, content_filtered, circuit_break, error) so a future error-variant or branch change can't silently re-bucket a termination.
  • test_record_agent_loop_exit_increments_once_with_labelsDebuggingRecorder (same pattern as the existing record_tool_call_metric tests) asserts the counter increments exactly once with the right agent and reason labels.
  • test_sanitize_agent_label_strips_control_and_caps_length — cardinality guard.

Verification

All run in the sanctioned librefang-rust-dev Docker image (no native cargo on this host), scoped to librefang-runtime:

  • cargo check -p librefang-runtime --lib — Finished, clean.
  • cargo clippy -p librefang-runtime --all-targets -- -D warnings — Finished, zero warnings.
  • mold -run cargo test -p librefang-runtime --lib -- test_classify_exit_reason test_record_agent_loop_exit test_sanitize_agent_label3 passed; 0 failed.
  • mold -run cargo test -p librefang-runtime --lib agent_loop::tests::utilities70 passed; 0 failed.
  • cargo fmt -p librefang-runtime applied.

Live LLM verification is not required: this change does not alter any LLM call path or prompt/metering wiring; it only adds an observability counter classified from the loop's existing return value, fully covered by the unit tests above.

Closes #6227

Evan and others added 2 commits June 19, 2026 14:23
Add `librefang_agent_loop_exits_total{agent,reason}`, incremented exactly
once per agent-loop termination so operators can alert on non-success exits
with `rate(librefang_agent_loop_exits_total{reason!="completed"}) > 0` per
agent — previously the only signal for repeated-tool-failure, max-iteration,
circuit-break, and content-filter aborts was reading transcripts (a cron /
trigger fire aborting on repeated tool failures still recorded
`librefang_cron_fires_total{outcome="ok"}` because the loop returned).

Single-increment is structural: `run_agent_loop` and
`run_agent_loop_streaming` become thin instrumented wrappers around
`*_inner` bodies; the single returned `Result` is classified by
`classify_exit_reason` and emitted via `record_agent_loop_exit` exactly once.
No terminal site inside the loop touches the metric, so no branch fall-through
can double-count.

Reason set matches the real termination branches:
`completed` (any Ok — finalized reply, silent completion, MaxTokens partial,
interrupt cancel, provider-not-configured), `max_iterations`,
`repeated_tool_failures`, `circuit_break`, `content_filtered`, `error`.
There is no distinct `empty_response` reason: an empty LLM response is a
one-shot in-loop retry that then finalizes as `completed`.

Circuit-break (propagated up as `Internal(msg)` via `?`) is matched against
the shared `loop_guard::CIRCUIT_BREAKER_MSG_PREFIX` constant — single source
of truth referenced by both producer and classifier. The agent label is
sanitized (control chars stripped, length capped at 64) like the existing
tool / cron agent labels.

Tests (injection site, `agent_loop::tests::utilities`):
`test_classify_exit_reason_covers_every_branch` pins each reason mapping,
`test_record_agent_loop_exit_increments_once_with_labels` asserts a single
increment with the right agent/reason labels,
`test_sanitize_agent_label_strips_control_and_caps_length` guards cardinality.
@github-actions github-actions Bot added area/docs Documentation and guides area/runtime Agent loop, LLM drivers, WASM sandbox size/L 250-999 lines changed labels Jun 19, 2026
@houko
houko merged commit db4a70b into main Jun 19, 2026
31 checks passed
@houko
houko deleted the feat/6227-agent-loop-exit-metric branch June 19, 2026 07:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/docs Documentation and guides area/runtime Agent loop, LLM drivers, WASM sandbox size/L 250-999 lines changed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

No metric for agent-loop exit reason (repeated tool failures / max iterations / empty response)

2 participants