feat(runtime): emit agent-loop exit-reason metric (#6227)#6232
Merged
Conversation
Add `librefang_agent_loop_exits_total{agent,reason}`, incremented exactly
once per agent-loop termination so operators can alert on non-success exits
with `rate(librefang_agent_loop_exits_total{reason!="completed"}) > 0` per
agent — previously the only signal for repeated-tool-failure, max-iteration,
circuit-break, and content-filter aborts was reading transcripts (a cron /
trigger fire aborting on repeated tool failures still recorded
`librefang_cron_fires_total{outcome="ok"}` because the loop returned).
Single-increment is structural: `run_agent_loop` and
`run_agent_loop_streaming` become thin instrumented wrappers around
`*_inner` bodies; the single returned `Result` is classified by
`classify_exit_reason` and emitted via `record_agent_loop_exit` exactly once.
No terminal site inside the loop touches the metric, so no branch fall-through
can double-count.
Reason set matches the real termination branches:
`completed` (any Ok — finalized reply, silent completion, MaxTokens partial,
interrupt cancel, provider-not-configured), `max_iterations`,
`repeated_tool_failures`, `circuit_break`, `content_filtered`, `error`.
There is no distinct `empty_response` reason: an empty LLM response is a
one-shot in-loop retry that then finalizes as `completed`.
Circuit-break (propagated up as `Internal(msg)` via `?`) is matched against
the shared `loop_guard::CIRCUIT_BREAKER_MSG_PREFIX` constant — single source
of truth referenced by both producer and classifier. The agent label is
sanitized (control chars stripped, length capped at 64) like the existing
tool / cron agent labels.
Tests (injection site, `agent_loop::tests::utilities`):
`test_classify_exit_reason_covers_every_branch` pins each reason mapping,
`test_record_agent_loop_exit_increments_once_with_labels` asserts a single
increment with the right agent/reason labels,
`test_sanitize_agent_label_strips_control_and_caps_length` guards cardinality.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Emits a counter at every agent-loop termination so operators can finally alert on non-success exits.
Before this, the agent loop recorded no metric when it aborted on repeated tool failures, max iterations, a loop-guard circuit break, or a provider content-filter — the only signal was reading transcripts.
A cron / trigger fire that aborted on repeated tool failures still recorded
librefang_cron_fires_total{outcome="ok"}because the loop returned (did not throw).New metric:
Acceptance alert (per #6227):
rate(librefang_agent_loop_exits_total{reason!="completed"}) > 0per agent.Reason set
Matched to the real termination branches in
run_streaming.rs,mod.rs, andloop_guard.rs:completed— anyOk(_)return: a finalized EndTurn reply, a silent completion (NO_REPLY / cascade-leak / progress-leak / mid-stream cascade-leak abort), a MaxTokens partial return, an interrupt-cancel early return, or the provider-not-configured early return.max_iterations—Err(MaxIterationsExceeded): theforloop ran out.repeated_tool_failures—Err(RepeatedToolFailures):consecutive_all_failed >= MAX_CONSECUTIVE_ALL_FAILED.circuit_break— loop-guard global circuit breaker tripped.content_filtered—Err(ContentFiltered): provider safety / content filter blocked the reply.error— any other propagatedErr(LLM call failure, memory error, etc.).Deviations from the issue's suggested set (documented per the task)
empty_responsereason. The issue listed it, but there is no distinct empty-response termination branch: an empty LLM response is a one-shot in-loop retry (EndTurnRetry::EmptyResponse→continue); the turn then either retries to a real reply or finalizes viafinalize_end_turn_text, both of which land oncompleted.content_filtered.ContentFilteredis a real, distinct terminalErrbranch operators will want to separate from genericerror.How exactly-one increment is guaranteed
Structural, not by discipline at each site:
run_agent_loopandrun_agent_loop_streamingbecome thin instrumented wrappers (the#[instrument]span name and fields are unchanged, so theinstrument_span_fields/ CLI span-name tests stay green).*_innerbody once and feeds the single returnedResulttorecord_agent_loop_exit(agent, &result).classify_exit_reasonmaps that oneResultto one reason; the counter is incremented once.?-propagated exits (circuit-break, LLM/memory errors) are caught at the wrapper too — they would have been missed by per-return-site increments.Circuit-break surfaces as
Err(Internal(msg))propagated up through the?onexecute_single_tool_call. It is classified by matching the sharedloop_guard::CIRCUIT_BREAKER_MSG_PREFIXconstant — a single source of truth now referenced by both the producer (LoopGuard::check) and the classifier, so the match is compiler-anchored rather than a loose duplicated string literal.Label cardinality
The
agentlabel is sanitized (control chars stripped, length capped at 64) bysanitize_agent_label, mirroringsanitize_tool_labelintool_call.rsand the bounded-cardinality contract onrecord_cron_fire_metricin the kernel'scron.rs.Changes
crates/librefang-runtime/src/agent_loop/mod.rs—sanitize_agent_label,classify_exit_reason,record_agent_loop_exit;run_agent_loopsplit into instrumented wrapper +run_agent_loop_inner.crates/librefang-runtime/src/agent_loop/run_streaming.rs—run_agent_loop_streamingsplit into instrumented wrapper +run_agent_loop_streaming_inner, callingsuper::record_agent_loop_exit.crates/librefang-runtime/src/loop_guard.rs— extractCIRCUIT_BREAKER_MSG_PREFIXconst; use it in the circuit-break message.crates/librefang-runtime/src/agent_loop/tests/utilities.rs— three injection-site tests (below).CHANGELOG.md—[Unreleased] / Addedentry.Tests
Added at the injection site (
agent_loop::tests::utilities):test_classify_exit_reason_covers_every_branch— pins each reason mapping (completed,max_iterations,repeated_tool_failures,content_filtered,circuit_break,error) so a future error-variant or branch change can't silently re-bucket a termination.test_record_agent_loop_exit_increments_once_with_labels—DebuggingRecorder(same pattern as the existingrecord_tool_call_metrictests) asserts the counter increments exactly once with the rightagentandreasonlabels.test_sanitize_agent_label_strips_control_and_caps_length— cardinality guard.Verification
All run in the sanctioned
librefang-rust-devDocker image (no native cargo on this host), scoped tolibrefang-runtime:cargo check -p librefang-runtime --lib— Finished, clean.cargo clippy -p librefang-runtime --all-targets -- -D warnings— Finished, zero warnings.mold -run cargo test -p librefang-runtime --lib -- test_classify_exit_reason test_record_agent_loop_exit test_sanitize_agent_label—3 passed; 0 failed.mold -run cargo test -p librefang-runtime --lib agent_loop::tests::utilities—70 passed; 0 failed.cargo fmt -p librefang-runtimeapplied.Live LLM verification is not required: this change does not alter any LLM call path or prompt/metering wiring; it only adds an observability counter classified from the loop's existing return value, fully covered by the unit tests above.
Closes #6227