Skip to content

Commit eed74bd

Browse files
wpfleger96Duncan
andauthored
feat(buzz-agent): surface stop reason and silent-turn WARN in telemetry (block#7038)
## What Two telemetry additions to make a known silent-death failure mode visible in harness logs. **1. `stop` field on `llm: call completed` INFO** (`crates/buzz-agent/src/llm.rs`) The `ProviderStop` value was already parsed and stored on `LlmResponse` but never emitted in the log line. Without it, "model chose `end_turn`" vs "gateway truncated/refused" is indistinguishable from telemetry alone. **2. WARN on silent-turn signature** (`crates/buzz-agent/src/agent.rs`) Emits a `WARN` when a turn produces no publish, no visible assistant text, and either near-zero or absent output tokens. The WARN logic is extracted into `warn_if_silent_turn` (pure synchronous function) so the seam is testable without the async run loop. Three independent gates before the WARN fires: 1. **`!buzz_reply_call_seen`** — no publish attempt in any round, tracked unconditionally via the existing `is_buzz_reply_call` matcher. Read-only tool calls do NOT suppress the WARN; a turn that ran tools but never published and died at 3 tokens is still a silent death. 2. **`text_is_empty`** — no visible assistant text in the final round. A terse reply like "OK" (≤12 tokens, non-empty) is not a silent death. 3. **Token check** — two distinct WARN messages: - `Some(t) where t <= 12`: near-zero token count, the observed failure signature (2–12 tokens) - `None` usage: provider omitted token counts entirely, separately diagnostic Tests use a scoped `tracing_subscriber` layer (same pattern as the existing stall-warn tests in `llm.rs`) to exercise the WARN seam directly: - Canonical signature (no publish, no text, 4 tokens) → 1 WARN - Non-empty assistant text → 0 WARNs - Publish seen → 0 WARNs - `None` usage (no publish, no text) → 1 WARN ## Why Recurring silent-death incident in a specific agent×channel combination: sessions die with 1 LLM call, 2–12 output tokens, no tool calls, no message, no error — recorded as a "successful" turn. The harness log shows the token count but not the `stop_reason`, leaving the root cause undiagnosable without request-level tracing. The observed shape also includes tool-step-then-3-token-death (one tool call, then silence) — the publish-aware gate catches both shapes. Context thread: buzz://message?channel=91fd9ca1-cf04-4ef7-b18f-aa2aee55692b&id=e3f1693f2e29f26a0c840f8054d592270c1504beacdc1d9c2063d8ab82960a06 ## Scope Logging and telemetry only. No behavior change, no retry-logic change, no stop-reason mapping change. --------- Signed-off-by: Will Pfleger <[email protected]> Co-authored-by: Duncan <dcfd242e557282d7a1e2cf2e6877522682f1e5c6156dc92ca7d90eaedd3b0f95@buzz.block.builderlab.xyz>
1 parent 00e61ea commit eed74bd

2 files changed

Lines changed: 238 additions & 1 deletion

File tree

‎crates/buzz-agent/src/agent.rs‎

Lines changed: 237 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,6 +68,12 @@ fn replace_unsupported_images(history: &mut [HistoryItem]) -> usize {
6868
/// [`Config::require_reply`](crate::config::Config::require_reply).
6969
const MAX_REPLY_NAGS: u32 = 2;
7070

71+
/// Output-token upper bound (inclusive) for the silent-death signature: the
72+
/// observed failure emits 2–12 tokens. Turns with `output_tokens <= 12`
73+
/// and no prior tool call are flagged. Legitimate one-sentence replies land
74+
/// well above this value even in the most terse case.
75+
const SILENT_TURN_TOKEN_THRESHOLD: u64 = 12;
76+
7177
/// Server label on the synthetic reply-guard objection.
7278
///
7379
/// Not a real MCP server. It rides the same tool-result path as `_Stop` hook
@@ -348,6 +354,11 @@ impl RunCtx<'_> {
348354
//
349355
// Named for what it proves: a *recognized attempt* to publish, not a
350356
// successful publish. See `is_buzz_reply_call`.
357+
// Tracks whether a publish-shaped tool call was seen this turn, updated
358+
// unconditionally (not gated on `require_reply`) so the silent-turn
359+
// diagnostic has a turn-level view regardless of config. A turn that
360+
// ran read-only tools and then died at 3 tokens IS a silent death;
361+
// only a genuine publish should suppress the WARN.
351362
let mut buzz_reply_call_seen = false;
352363
let mut reply_nags = 0u32;
353364
// Per-`run()` reactive context-recovery budget. Per-turn, not
@@ -696,12 +707,33 @@ impl RunCtx<'_> {
696707
"provider: stop=tool_use but zero tool_calls".into(),
697708
));
698709
}
710+
// Capture before response.text is moved into history.
711+
let text_is_empty = response.text.trim().is_empty();
699712
self.history.push(HistoryItem::Assistant {
700713
text: response.text,
701714
tool_calls: Vec::new(),
702715
reasoning_details: response.reasoning_details.clone(),
703716
});
704717
let stop = map_stop(response.stop);
718+
// Diagnostic: warn when no publish was seen across the whole
719+
// turn, the final response has no visible text, and the
720+
// token count looks silent. Two independent gates:
721+
// 1. `!buzz_reply_call_seen` — no publish attempt in any
722+
// round (read-only tool calls do NOT suppress: a turn
723+
// that ran tools but never published then died at 3
724+
// tokens is still a silent death).
725+
// 2. `text_is_empty` — model emitted no visible text
726+
// (a terse reply like "OK" is not silent).
727+
// 3. token count or usage-absent check.
728+
// Fires before the `_Stop` hook so the warning appears in
729+
// the log even if the hook rejects the stop and the loop
730+
// continues. Does not alter control flow.
731+
warn_if_silent_turn(
732+
buzz_reply_call_seen,
733+
text_is_empty,
734+
response.output_tokens,
735+
response.stop,
736+
);
705737
// Only gate genuine end_turn — don't override max_tokens/refusal.
706738
if stop == StopReason::EndTurn {
707739
if stop_rejections >= self.cfg.stop_max_rejections {
@@ -746,7 +778,10 @@ impl RunCtx<'_> {
746778
}
747779
// Deliberately after truncation: a publish-shaped call that was
748780
// discarded never runs, so it must not suppress the reminder.
749-
if self.cfg.require_reply && !buzz_reply_call_seen {
781+
// Updated unconditionally (not gated on `require_reply`) so the
782+
// silent-turn diagnostic has a publish-aware turn-level signal
783+
// regardless of config.
784+
if !buzz_reply_call_seen {
750785
buzz_reply_call_seen = calls.iter().any(|c| is_buzz_reply_call(c, self.mcp));
751786
}
752787
self.history.push(HistoryItem::Assistant {
@@ -1286,10 +1321,69 @@ fn map_stop(p: ProviderStop) -> StopReason {
12861321
}
12871322
}
12881323

1324+
/// Returns `true` when a reported output-token count is at or below the
1325+
/// silent-death threshold. The observed failure signature is 2–12 tokens.
1326+
///
1327+
/// Takes a bare `u64` — the caller handles `None` usage separately (a
1328+
/// provider that omits token counts is a distinct diagnostic case, not
1329+
/// automatically "near-zero").
1330+
///
1331+
/// Extracted as a pure function so it can be tested without standing up an
1332+
/// async agent loop.
1333+
fn is_silent_turn(output_tokens: u64) -> bool {
1334+
output_tokens <= SILENT_TURN_TOKEN_THRESHOLD
1335+
}
1336+
1337+
/// Emits the silent-turn diagnostic WARN when the turn produced no publish,
1338+
/// no visible text, and either near-zero or absent output tokens.
1339+
///
1340+
/// `buzz_reply_call_seen` is the publish-aware gate (from
1341+
/// `is_buzz_reply_call`), updated unconditionally regardless of
1342+
/// `require_reply`. Read-only tool calls do NOT suppress the WARN — a turn
1343+
/// that ran tools but never published and then died at 3 tokens is a silent
1344+
/// death.
1345+
///
1346+
/// Two distinct WARN shapes:
1347+
/// - Near-zero token count (`output_tokens <= 12`): canonical silent-death.
1348+
/// - Unknown usage (`None`) with no publish and no text: separately
1349+
/// diagnostic; does not assert near-zero since the count is unknown.
1350+
///
1351+
/// Extracted as a free function so the WARN seam can be exercised by a
1352+
/// scoped tracing subscriber without standing up the full async run loop.
1353+
fn warn_if_silent_turn(
1354+
buzz_reply_call_seen: bool,
1355+
text_is_empty: bool,
1356+
output_tokens: Option<u64>,
1357+
stop: ProviderStop,
1358+
) {
1359+
if buzz_reply_call_seen || !text_is_empty {
1360+
return;
1361+
}
1362+
match output_tokens {
1363+
Some(t) if is_silent_turn(t) => {
1364+
tracing::warn!(
1365+
stop = ?stop,
1366+
output_tokens = t,
1367+
"agent: turn ended with no publish attempt and near-zero output tokens — possible silent model/gateway early-stop"
1368+
);
1369+
}
1370+
None => {
1371+
tracing::warn!(
1372+
stop = ?stop,
1373+
"agent: turn ended with no publish attempt and no usage reported — cannot confirm output size"
1374+
);
1375+
}
1376+
_ => {}
1377+
}
1378+
}
1379+
12891380
#[cfg(test)]
12901381
mod tests {
12911382
use super::*;
12921383
use serde_json::json;
1384+
use std::sync::atomic::{AtomicUsize, Ordering};
1385+
use std::sync::Arc;
1386+
use tracing_subscriber::layer::SubscriberExt;
12931387

12941388
/// `truncate_history` cannot serve as the context-window fallback: it is
12951389
/// measured in BYTES (`max_history_bytes`, default 16 MiB, a request-body
@@ -1635,4 +1729,146 @@ mod tests {
16351729
"three identical rounds must remain consistently proven"
16361730
);
16371731
}
1732+
1733+
// ── is_silent_turn (pure predicate) ─────────────────────────────────────
1734+
1735+
/// Counts WARN events emitted by `warn_if_silent_turn` calls inside `f`.
1736+
///
1737+
/// Identifies silent-turn WARNs by target (`buzz_agent::agent`) + WARN
1738+
/// level + presence of the `stop` field, which is unique to these two
1739+
/// WARNs in this module. Using the target avoids parsing message strings,
1740+
/// which are routed through `record_debug` as `Display`-formatted values
1741+
/// and are not reliably interceptable via `record_str` across tracing
1742+
/// versions.
1743+
fn count_silent_turn_warnings(f: impl FnOnce()) -> usize {
1744+
struct Capture {
1745+
count: Arc<AtomicUsize>,
1746+
}
1747+
struct Visitor {
1748+
saw_stop: bool,
1749+
}
1750+
impl tracing::field::Visit for Visitor {
1751+
fn record_debug(&mut self, field: &tracing::field::Field, _: &dyn std::fmt::Debug) {
1752+
if field.name() == "stop" {
1753+
self.saw_stop = true;
1754+
}
1755+
}
1756+
}
1757+
impl<S: tracing::Subscriber> tracing_subscriber::Layer<S> for Capture {
1758+
fn on_event(
1759+
&self,
1760+
event: &tracing::Event<'_>,
1761+
_: tracing_subscriber::layer::Context<'_, S>,
1762+
) {
1763+
if *event.metadata().level() != tracing::Level::WARN {
1764+
return;
1765+
}
1766+
if event.metadata().target() != "buzz_agent::agent" {
1767+
return;
1768+
}
1769+
let mut v = Visitor { saw_stop: false };
1770+
event.record(&mut v);
1771+
if v.saw_stop {
1772+
self.count.fetch_add(1, Ordering::SeqCst);
1773+
}
1774+
}
1775+
}
1776+
let count = Arc::new(AtomicUsize::new(0));
1777+
let sub = tracing_subscriber::registry().with(Capture {
1778+
count: count.clone(),
1779+
});
1780+
tracing::subscriber::with_default(sub, f);
1781+
count.load(Ordering::SeqCst)
1782+
}
1783+
1784+
/// Predicate: values within the observed failure range (2–12) fire.
1785+
/// Pair (0, 12) catches an always-false mutation and an off-by-one at 12.
1786+
#[test]
1787+
fn is_silent_turn_fires_at_and_below_threshold() {
1788+
assert!(
1789+
is_silent_turn(0),
1790+
"zero output tokens must be a silent turn"
1791+
);
1792+
assert!(
1793+
is_silent_turn(SILENT_TURN_TOKEN_THRESHOLD),
1794+
"exactly at threshold (12) must be a silent turn — 12 is in the observed range"
1795+
);
1796+
}
1797+
1798+
/// One above the threshold must NOT fire, catching `<` vs `<=` and
1799+
/// always-true mutations.
1800+
#[test]
1801+
fn is_silent_turn_silent_above_threshold() {
1802+
assert!(
1803+
!is_silent_turn(SILENT_TURN_TOKEN_THRESHOLD + 1),
1804+
"one above threshold (13) must not be a silent turn"
1805+
);
1806+
}
1807+
1808+
// ── warn_if_silent_turn (WARN seam) ───────────────────────────────────
1809+
1810+
/// The canonical silent-death signature — no publish, no text, ≤12 tokens
1811+
/// — must emit exactly one WARN. Deleting the WARN call, weakening the
1812+
/// token check, or hardcoding `buzz_reply_call_seen = true` are all caught.
1813+
#[test]
1814+
fn warn_if_silent_turn_fires_for_canonical_signature() {
1815+
let n = count_silent_turn_warnings(|| {
1816+
warn_if_silent_turn(
1817+
false, // no publish seen
1818+
true, // no text
1819+
Some(4), // 4 tokens — in the 2–12 range
1820+
ProviderStop::EndTurn,
1821+
);
1822+
});
1823+
assert_eq!(n, 1, "canonical silent-death must emit exactly one WARN");
1824+
}
1825+
1826+
/// A turn that ends with non-empty assistant text is NOT a silent death
1827+
/// even if token count is low — a terse reply like "OK" is legitimate.
1828+
/// Deleting the `text_is_empty` gate would cause this to fail.
1829+
#[test]
1830+
fn warn_if_silent_turn_silent_for_nonempty_text() {
1831+
let n = count_silent_turn_warnings(|| {
1832+
warn_if_silent_turn(
1833+
false, // no publish
1834+
false, // text IS present
1835+
Some(3), // low tokens — would fire without the text gate
1836+
ProviderStop::EndTurn,
1837+
);
1838+
});
1839+
assert_eq!(n, 0, "a turn with non-empty assistant text must not WARN");
1840+
}
1841+
1842+
/// A turn that published (buzz_reply_call_seen = true) then ended with a
1843+
/// short final completion must not trigger the WARN. This is the normal
1844+
/// publish-then-wrap pattern. Deleting the `buzz_reply_call_seen` gate
1845+
/// would cause this to fail.
1846+
#[test]
1847+
fn warn_if_silent_turn_silent_after_publish() {
1848+
let n = count_silent_turn_warnings(|| {
1849+
warn_if_silent_turn(
1850+
true, // publish seen
1851+
true, // no text in final round
1852+
Some(0), // zero tokens — would fire without the publish gate
1853+
ProviderStop::EndTurn,
1854+
);
1855+
});
1856+
assert_eq!(n, 0, "a turn where a publish ran must not WARN");
1857+
}
1858+
1859+
/// Unknown usage (None) with no publish and no text emits the distinct
1860+
/// "no usage reported" WARN. Mutating the None arm to fall through to
1861+
/// `_ => {}` would cause this.
1862+
#[test]
1863+
fn warn_if_silent_turn_fires_distinct_warn_for_none_usage() {
1864+
let n = count_silent_turn_warnings(|| {
1865+
warn_if_silent_turn(
1866+
false, // no publish
1867+
true, // no text
1868+
None, // provider omitted usage
1869+
ProviderStop::EndTurn,
1870+
);
1871+
});
1872+
assert_eq!(n, 1, "unknown-usage silent turn must emit exactly one WARN");
1873+
}
16381874
}

‎crates/buzz-agent/src/llm.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -229,6 +229,7 @@ impl Llm {
229229
input_tokens = ?response.input_tokens,
230230
cached_input_tokens = ?response.cached_input_tokens,
231231
output_tokens = ?response.output_tokens,
232+
stop = ?response.stop,
232233
"llm: call completed"
233234
);
234235
}

0 commit comments

Comments
 (0)