Skip to content

Commit 9e0ae1f

Browse files
tlongwell-blockDawn (sprout agent)
andcommitted
feat(agent): optional reply guard reminds a silent turn to publish
A Buzz agent's assistant text and reasoning are never shown to anyone — only what it posts through the CLI. A turn that runs fifteen tool calls and never publishes is a silent failure: the requester waits on a result that was produced and thrown away. With BUZZ_AGENT_REQUIRE_REPLY=1 (default off, per-agent opt-in), a turn about to end without a recognized attempt to post gets a reminder and is rerolled. At most two, then the turn ends regardless — the guard catches accidental omission, it does not compel speech. The reminder text explicitly licenses silence so it cannot fight the base prompt's "silence is usually correct". This is not a new MCP hook. `RunCtx::run` *is* the turn, so the two per-turn locals need no plumbing, and every tool call already passes through it with its arguments visible. The objection is appended at the existing `_Stop` gate and rides `push_hook_outputs_as_tool_results`, so the model receives it as a lower-trust tool result with `{hook, server, text}` attribution — no new trust path, no new lifecycle event, no dev-mcp or CLI protocol change. Reminders share BUZZ_AGENT_STOP_MAX_REJECTIONS, the existing outer cap on every end-turn objection. A round carrying both a hook objection and a reminder costs one rejection and delivers both texts; at budget 1 only one reminder fits, and at 0 the guard is off along with the hooks. An independent budget would either violate that bound or need a second arbitration rule. Recognition is deliberately coarse: a registered non-hook tool whose qualified name ends in `__shell`, whose `command` argument contains `messages send` or `reactions add`. Given the registry checks, that suffix is exactly equivalent to a bare name of `shell` — registration forbids `__` in server and bare names, so a trailing `__shell` could only straddle the separator if the bare name began with `_`, which `is_hook` excludes. It reads the structured `command` field rather than serialized arguments, so a `description` that quotes a send cannot disarm the guard. It recognizes an *attempt*, not a successful publish: a failed send already returns non-zero exit and error JSON, which is louder than this reminder. Detection runs after the per-turn tool-call cap, since a discarded call never ran. Adds FAKE_MCP_SHELL_TOOL=1 to the test MCP server: it previously exposed no tool with a bare name of `shell`, so the satisfied-guard path was untestable. Plan reviewed to 9.5/10 with Wren; each guard behavior is pinned by a mutation that breaks a specific named test. Co-authored-by: Dawn (sprout agent) <c6237ef84fa537c78dcee78efd2d4e59f728859c7f194da42ac51ededfa0be05@sprout-oss.stage.blox.sqprod.co> Signed-off-by: tlongwell-block <[email protected]>
1 parent 02be413 commit 9e0ae1f

7 files changed

Lines changed: 753 additions & 2 deletions

File tree

‎crates/buzz-agent/README.md‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -163,6 +163,61 @@ Everything is environment variables. No flags, no config files. (We are a subpro
163163
| `BUZZ_AGENT_MAX_LINE_BYTES` | `4194304` | 4 MiB. Hard cap on inbound JSON-RPC frames. |
164164
| `BUZZ_AGENT_MAX_HISTORY_BYTES` | `1048576` | 1 MiB. Old turns are evicted past this. |
165165
| `BUZZ_AGENT_MAX_TOOL_RESULT_TEXT_BYTES` | `51200` | 50 KiB. Per-result cap on tool-output text; oversize is middle-elided (head + tail kept) with an inline marker. Images are exempt. |
166+
| `BUZZ_AGENT_REQUIRE_REPLY` | `0` | `1` enables the [reply guard](#reply-guard) — remind the model to publish when a turn is about to end with nothing posted to Buzz. |
167+
168+
169+
## Reply Guard
170+
171+
Off by default. With `BUZZ_AGENT_REQUIRE_REPLY=1`, a turn that is about to end
172+
without any recognized attempt to post to Buzz gets a reminder that its
173+
assistant text is invisible to humans, and is rerolled.
174+
175+
This exists because a Buzz agent's reasoning and tool output are not shown to
176+
anyone. A turn that does real work and never posts is a silent failure — the
177+
requester waits on a result that was produced and thrown away.
178+
179+
**Advisory, never a trap.** At most two reminders, then the turn ends whether or
180+
not anything was published. The guard catches accidental omission; it does not
181+
compel speech. The reminder text explicitly licenses silence, because the
182+
built-in system prompt says publishing is optional and silence is often the
183+
correct outcome.
184+
185+
**Recognition contract.** A turn counts as having replied when it issues a call
186+
that:
187+
188+
- resolves to a registered, non-hook tool (a hallucinated tool name is rejected
189+
at preflight and never runs, so it must not disarm the guard),
190+
- whose qualified name ends in `__shell` — i.e. the bare tool name is exactly
191+
`shell`, which is `buzz-dev-mcp`'s shell tool and any other server's, and
192+
- whose `command` argument contains `messages send` or `reactions add`.
193+
194+
`messages send` also covers `messages send-diff`. Reactions count because the
195+
built-in prompt directs agents to react rather than post a bare
196+
acknowledgement, so nagging an agent that reacted would punish documented
197+
behavior.
198+
199+
Detection is checked **after** the per-turn tool-call cap
200+
(`MAX_TOOL_CALLS_PER_TURN`) is applied: a publish-shaped call that was discarded
201+
never ran.
202+
203+
**It recognizes an attempt, not a successful publish.** Only the command text is
204+
inspected, never the exit status. A send that fails still satisfies the guard —
205+
which is fine, since a failed send already returns a non-zero exit and error
206+
JSON to the model, louder feedback than a reminder.
207+
208+
**Known limits**, both deliberate. A command assembled at runtime (`$CMD`) or
209+
buried in a wrapper script is missed, so that turn is reminded despite having
210+
posted. Text that merely quotes a send (`echo "buzz messages send"`) matches, so
211+
that turn is not reminded. Missing a real post is the expensive direction, and
212+
substring matching is the forgiving one there. Neither edge is pinned by a test;
213+
the matcher is free to improve.
214+
215+
**Budget.** Reminders ride the existing `_Stop` gate and share
216+
`BUZZ_AGENT_STOP_MAX_REJECTIONS` — the outer cap on every end-turn objection.
217+
At the default 3 both reminders fit; at 1 only one does; at 0 the guard is off
218+
along with the hooks. A round carrying both a `_Stop` hook objection and a
219+
reminder costs one rejection and delivers both texts. This is not a new
220+
lifecycle hook — see [MCP_DRIVEN_HOOKS.md](../../docs/MCP_DRIVEN_HOOKS.md).
166221

167222

168223
## Providers

‎crates/buzz-agent/src/agent.rs‎

Lines changed: 181 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,80 @@ use crate::wire::{self, WireSender};
2121
const ERROR_REFLECTION_SUFFIX: &str =
2222
"\n\n[Reflect] Before retrying, identify the cause and change your approach.";
2323

24+
/// Maximum reply reminders emitted per prompt when `require_reply` is on.
25+
///
26+
/// After this many, the turn is allowed to end whether or not anything was
27+
/// published: the guard exists to catch accidental omission, not to compel
28+
/// speech. The shared `stop_max_rejections` budget can cut this lower — see
29+
/// [`Config::require_reply`](crate::config::Config::require_reply).
30+
const MAX_REPLY_NAGS: u32 = 2;
31+
32+
/// Server label on the synthetic reply-guard objection.
33+
///
34+
/// Not a real MCP server. It rides the same tool-result path as `_Stop` hook
35+
/// output, so the model sees `{hook, server, text}` attribution naming the
36+
/// in-process guard rather than an MCP server that could be impersonated.
37+
const REPLY_GUARD_SERVER: &str = "buzz-agent";
38+
39+
/// Reminder text emitted when a turn is about to end with nothing published.
40+
///
41+
/// Explicitly licenses silence. The base prompt tells agents that publishing is
42+
/// optional and "silence is usually correct"; a reminder that argued otherwise
43+
/// would fight that instruction and make agents chattier.
44+
const REPLY_GUARD_NAG: &str = "You are about to end this turn without calling `buzz messages send`. \
45+
Your assistant text and reasoning are never shown to anyone — if you did work, found an answer, \
46+
or hit a blocker that someone is waiting on, it exists only if you publish it. \
47+
If you already posted, or if silence is genuinely correct for this turn, ignore this and end your turn.";
48+
49+
/// Whether `call` is a recognized attempt to publish a reply to Buzz.
50+
///
51+
/// Recognizes an *attempt*, not a successful publish: the command text is
52+
/// inspected, never the exit status. That is deliberate — a send that fails
53+
/// already returns a non-zero exit and error JSON to the model, which is louder
54+
/// feedback than the reminder this gates.
55+
///
56+
/// `has` + `!is_hook` are the same checks the dispatcher uses to accept a call
57+
/// (see `execute_calls`), so a hallucinated `fake__shell` — rejected at preflight
58+
/// and never executed — cannot disarm the guard. They must stay *before*
59+
/// [`is_reply_shaped`]: together with them, and only with them, the `__shell`
60+
/// suffix is exactly equivalent to "the bare tool name is `shell`".
61+
fn is_buzz_reply_call(call: &ToolCall, mcp: &McpRegistry) -> bool {
62+
mcp.has(&call.name) && !mcp.is_hook(&call.name) && is_reply_shaped(&call.name, &call.arguments)
63+
}
64+
65+
/// Whether a tool name and arguments have the shape of a Buzz publish command.
66+
///
67+
/// Split from [`is_buzz_reply_call`] only so the matcher is testable without a
68+
/// live [`McpRegistry`]; callers must apply the registry checks first.
69+
///
70+
/// On the name: `ends_with("__shell")` is exact rather than approximate *given*
71+
/// those checks. Registration rejects `__` in both server names and bare tool
72+
/// names, and qualified names are `{server}__{bare}`, so a trailing `__shell` can
73+
/// only straddle the separator if the bare name starts with `_` — which `is_hook`
74+
/// already excludes. Dropping the separator would not be exact: `powershell` and
75+
/// `noshell` both end in `shell`.
76+
///
77+
/// On the command: a deliberately coarse substring test, scoped to the structured
78+
/// `command` field so unrelated metadata — a `description` that quotes a send —
79+
/// cannot suppress the guard, and a non-string `command` is rejected rather than
80+
/// coerced. Known limits, both accepted: a command assembled at runtime (`$CMD`)
81+
/// or hidden in a wrapper script is missed, and text that merely quotes a send
82+
/// (`echo "buzz messages send"`) matches. Missing a real post is the expensive
83+
/// direction, and substring matching is the more forgiving one there.
84+
fn is_reply_shaped(name: &str, arguments: &serde_json::Value) -> bool {
85+
name.ends_with("__shell")
86+
&& arguments
87+
.get("command")
88+
.and_then(|v| v.as_str())
89+
.is_some_and(|cmd| {
90+
// `messages send` also covers `messages send-diff`. `reactions
91+
// add` counts because the base prompt directs agents to react
92+
// rather than post a bare acknowledgement, so nagging an agent
93+
// that reacted would punish documented-correct behavior.
94+
cmd.contains("messages send") || cmd.contains("reactions add")
95+
})
96+
}
97+
2498
pub struct RunCtx<'a> {
2599
pub cfg: &'a Config,
26100
/// Effective model for this session. Usually equals `cfg.model`; overridden
@@ -102,6 +176,14 @@ impl RunCtx<'_> {
102176
// session) so a stubborn exchange can't permanently disable the stop
103177
// guard for a long-lived session; `max_rounds` still caps the loop.
104178
let mut stop_rejections = 0u32;
179+
// Reply-guard state for this prompt. `prompt()` *is* the turn, so
180+
// locals here are per-turn by construction — same shape as
181+
// `stop_rejections` above.
182+
//
183+
// Named for what it proves: a *recognized attempt* to publish, not a
184+
// successful publish. See `is_buzz_reply_call`.
185+
let mut buzz_reply_call_seen = false;
186+
let mut reply_nags = 0u32;
105187
loop {
106188
if self.cfg.max_rounds > 0 && round >= self.cfg.max_rounds {
107189
return Ok(StopReason::MaxTurnRequests);
@@ -264,7 +346,7 @@ impl RunCtx<'_> {
264346
if stop_rejections >= self.cfg.stop_max_rejections {
265347
return Ok(stop);
266348
}
267-
let objections = self
349+
let mut objections = self
268350
.mcp
269351
.call_hooks(
270352
"_Stop",
@@ -273,6 +355,17 @@ impl RunCtx<'_> {
273355
&self.cfg.hook_servers,
274356
)
275357
.await;
358+
// Reply guard shares this gate and this budget, so a round
359+
// carrying both a hook objection and a reply reminder costs
360+
// one rejection and delivers both texts.
361+
if self.cfg.require_reply
362+
&& !buzz_reply_call_seen
363+
&& reply_nags < MAX_REPLY_NAGS
364+
{
365+
reply_nags += 1;
366+
objections
367+
.push((REPLY_GUARD_SERVER.to_string(), REPLY_GUARD_NAG.to_string()));
368+
}
276369
if !objections.is_empty() {
277370
stop_rejections = stop_rejections.saturating_add(1);
278371
push_hook_outputs_as_tool_results(self.history, "_Stop", &objections);
@@ -290,6 +383,11 @@ impl RunCtx<'_> {
290383
);
291384
calls.truncate(MAX_TOOL_CALLS_PER_TURN);
292385
}
386+
// Deliberately after truncation: a publish-shaped call that was
387+
// discarded never runs, so it must not suppress the reminder.
388+
if self.cfg.require_reply && !buzz_reply_call_seen {
389+
buzz_reply_call_seen = calls.iter().any(|c| is_buzz_reply_call(c, self.mcp));
390+
}
293391
self.history.push(HistoryItem::Assistant {
294392
text: response.text,
295393
tool_calls: calls.clone(),
@@ -799,6 +897,88 @@ mod tests {
799897
use super::*;
800898
use serde_json::json;
801899

900+
/// The shapes the guard must recognize as a publish attempt. Callers apply
901+
/// the registry checks first; these cover the name suffix and command text.
902+
#[test]
903+
fn reply_shape_matches_documented_send_forms() {
904+
for cmd in [
905+
"buzz messages send --channel X --content Y",
906+
"buzz --relay wss://r messages send --channel X --content Y",
907+
"/abs/path/buzz messages send",
908+
"printf 'hi' | buzz messages send --content -",
909+
"buzz messages send-diff --diff -",
910+
"buzz reactions add --event E --emoji +",
911+
// Assembled through another shell: rev 3's tokenizer missed this.
912+
r#"sh -c "buzz messages send --channel X""#,
913+
] {
914+
assert!(
915+
is_reply_shaped("dev__shell", &json!({ "command": cmd })),
916+
"expected {cmd:?} to count as a publish attempt"
917+
);
918+
}
919+
}
920+
921+
/// Commands that do real work but do not reply in the originating
922+
/// conversation must still be nagged.
923+
#[test]
924+
fn reply_shape_rejects_non_reply_commands() {
925+
for cmd in [
926+
"buzz messages get --channel X",
927+
"buzz channels list",
928+
"buzz reactions remove --event E",
929+
"buzz pr open --title T",
930+
"buzz social publish --content hi",
931+
"buzz notes set --name n",
932+
"cargo test -p buzz-agent",
933+
] {
934+
assert!(
935+
!is_reply_shaped("dev__shell", &json!({ "command": cmd })),
936+
"expected {cmd:?} not to count as a publish attempt"
937+
);
938+
}
939+
}
940+
941+
/// The `__` separator is load-bearing: `ends_with("shell")` alone would
942+
/// accept any registered tool whose name merely ends in those letters, and
943+
/// `has()` proves registration, not the bare name.
944+
#[test]
945+
fn reply_shape_requires_the_qname_separator() {
946+
let args = json!({ "command": "buzz messages send --channel X" });
947+
for name in [
948+
"dev__powershell",
949+
"dev__noshell",
950+
"shell",
951+
"dev__send_message",
952+
] {
953+
assert!(
954+
!is_reply_shaped(name, &args),
955+
"{name} must not satisfy the shell-tool check"
956+
);
957+
}
958+
assert!(is_reply_shaped("dev__shell", &args));
959+
assert!(is_reply_shaped("buzz-dev-mcp__shell", &args));
960+
}
961+
962+
/// Only the field that carries the executable command counts. Searching
963+
/// serialized arguments instead would let arbitrary metadata disarm the
964+
/// guard, turning a description into an attempted send.
965+
#[test]
966+
fn reply_shape_reads_only_the_command_field() {
967+
assert!(!is_reply_shaped(
968+
"dev__shell",
969+
&json!({ "description": "buzz messages send --channel X" })
970+
));
971+
assert!(!is_reply_shaped(
972+
"dev__shell",
973+
&json!({ "workdir": "buzz messages send" })
974+
));
975+
// Malformed `command` is rejected, not coerced — and must not panic.
976+
assert!(!is_reply_shaped("dev__shell", &json!({ "command": 42 })));
977+
assert!(!is_reply_shaped("dev__shell", &json!({ "command": null })));
978+
assert!(!is_reply_shaped("dev__shell", &json!({})));
979+
assert!(!is_reply_shaped("dev__shell", &json!("not an object")));
980+
}
981+
802982
/// A9 regression: `reasoning_details` contributes real bytes to
803983
/// `estimated_bytes` (see `types.rs::HistoryItem::size_with`), so a
804984
/// history item carrying a large opaque reasoning array must actually

‎crates/buzz-agent/src/config.rs‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -720,6 +720,16 @@ pub struct Config {
720720
/// Maximum `_Stop` rejections per prompt. Default 3. Set to 0 to
721721
/// disable `_Stop` hooks entirely (agent always honors end_turn).
722722
pub stop_max_rejections: u32,
723+
/// Remind the model to publish when a turn is about to end without any
724+
/// recognized attempt to post to Buzz. Default off; opt in per agent with
725+
/// `BUZZ_AGENT_REQUIRE_REPLY=1`.
726+
///
727+
/// Advisory only: at most `MAX_REPLY_NAGS` reminders (see `agent.rs`),
728+
/// then the turn ends regardless. Bounded by the same
729+
/// `stop_max_rejections` budget as `_Stop` hooks, which is the outer cap on
730+
/// all end-turn objections — at the default 3 both reminders fit; at 1 only
731+
/// one does; at 0 the guard is off with the hooks.
732+
pub require_reply: bool,
723733
/// Hook server allowlist. See [`HookServers`] for variant semantics.
724734
/// Default (env unset/empty) is `None` — hooks are off unless the
725735
/// operator explicitly opts in.
@@ -851,6 +861,7 @@ impl Config {
851861
max_parallel_tools: parse_env("BUZZ_AGENT_MAX_PARALLEL_TOOLS", 8usize)?,
852862
hook_timeout: Duration::from_millis(parse_env("BUZZ_AGENT_HOOK_TIMEOUT_MS", 2500u64)?),
853863
stop_max_rejections: parse_env("BUZZ_AGENT_STOP_MAX_REJECTIONS", 3u32)?,
864+
require_reply: parse_env("BUZZ_AGENT_REQUIRE_REPLY", 0u8)? != 0,
854865
hook_servers: parse_hook_servers_env("MCP_HOOK_SERVERS"),
855866
hints_enabled: parse_env("BUZZ_AGENT_NO_HINTS", 0u8)? == 0,
856867
thinking_effort: parse_thinking_effort(env("BUZZ_AGENT_THINKING_EFFORT").as_deref())?,
@@ -893,6 +904,7 @@ impl Config {
893904
max_parallel_tools: 1,
894905
hook_timeout: Duration::from_secs(1),
895906
stop_max_rejections: 0,
907+
require_reply: false,
896908
hook_servers: HookServers::None,
897909
hints_enabled: false,
898910
thinking_effort: None,

‎crates/buzz-agent/src/llm.rs‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2355,6 +2355,7 @@ mod tests {
23552355
max_parallel_tools: 1,
23562356
hook_timeout: Duration::from_secs(1),
23572357
stop_max_rejections: 0,
2358+
require_reply: false,
23582359
hook_servers: HookServers::None,
23592360
api_key: "key".into(),
23602361
model: "model".into(),

‎crates/buzz-agent/tests/bin/fake_mcp.rs‎

Lines changed: 25 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,11 @@
3333
//! — expose a `_PostCompact` hook tool
3434
//! FAKE_MCP_POSTCOMPACT_TEXT=text
3535
//! — `_PostCompact` returns this (default: "")
36+
//! FAKE_MCP_SHELL_TOOL=1 — expose a tool whose bare name is `shell`
37+
//! (registered as `<server>__shell`), taking a
38+
//! `command` string. Lets a test drive the
39+
//! reply guard's recognition of a real,
40+
//! registered shell tool.
3641
3742
use std::io::{BufRead, Write};
3843

@@ -76,6 +81,7 @@ fn make_tools(
7681
desc: &str,
7782
include_stop_hook: bool,
7883
include_post_compact_hook: bool,
84+
include_shell_tool: bool,
7985
) -> Vec<Value> {
8086
let mut tools: Vec<Value> = (0..count)
8187
.map(|i| {
@@ -100,6 +106,17 @@ fn make_tools(
100106
"inputSchema": { "type": "object", "properties": {} },
101107
}));
102108
}
109+
if include_shell_tool {
110+
tools.push(json!({
111+
"name": "shell",
112+
"description": "run a shell command",
113+
"inputSchema": {
114+
"type": "object",
115+
"properties": { "command": { "type": "string" } },
116+
"required": ["command"],
117+
},
118+
}));
119+
}
103120
tools
104121
}
105122

@@ -136,6 +153,7 @@ fn main() {
136153
let stop_count_limit: usize = env_usize("FAKE_MCP_STOP_COUNT", usize::MAX);
137154
let mut stop_calls_seen: usize = 0;
138155
let post_compact_hook = env_flag("FAKE_MCP_POSTCOMPACT_HOOK");
156+
let shell_tool = env_flag("FAKE_MCP_SHELL_TOOL");
139157
let post_compact_text = std::env::var("FAKE_MCP_POSTCOMPACT_TEXT").unwrap_or_default();
140158

141159
// Use a channel-based stdin reader so notifications (which carry no id)
@@ -206,7 +224,13 @@ fn main() {
206224
write_response(
207225
id,
208226
json!({
209-
"tools": make_tools(tool_count, &desc, stop_hook, post_compact_hook)
227+
"tools": make_tools(
228+
tool_count,
229+
&desc,
230+
stop_hook,
231+
post_compact_hook,
232+
shell_tool,
233+
)
210234
}),
211235
);
212236
}

0 commit comments

Comments
 (0)