Skip to content

Commit 8a7eb8d

Browse files
tlongwell-blockWrenSami
authored
fix(agent): recover from unsupported image input instead of poisoning the turn (#4896)
## Problem `buzz-dev-mcp` advertises `view_image` to every agent regardless of whether the session's model accepts images. When a text-only model (e.g. DeepSeek V4 Flash) takes the bait, the image lands in session history and every subsequent LLM request 404s with `No endpoints found that support image input`. The error was classified as `LlmModelNotFound` and propagated fatally out of the turn loop — history stays poisoned, buzz-acp retries the batch with exponential backoff, and the session burns its entire clock doing no work. In a recent trial run, **all 57 trials that called `view_image` on a text-only model died this way; none recovered.** ## Fix Capability-gating the advertised tool isn't reliable — there is no image-capability metadata at the agent layer across providers. Instead, recover at the turn loop: - **Typed error**: new `AgentError::UnsupportedImageInput`, classified narrowly on the exact provider phrase `No endpoints found that support image input` on both the generic 404 path and OpenRouter's 404 path. Unknown-model 404s and OpenRouter parameter-routing 404s keep their existing classifications. No deterministic retry. - **In-turn recovery**: on this error, `RunCtx::run` strips every image block from history — keeping the tool result (and therefore tool-call/result pairing) intact — marks the result `is_error`, appends actionable model-facing guidance ("The current model does not support image input. The image was removed from conversation history so this turn can continue. Use a text-based inspection tool…"), and continues the same turn. Base64 never replays again. - **Loop guard**: recovery only fires when at least one image was removed; if the provider says "image" and history has none, the error propagates as before. ## Tests - Unit: phrase classification (typed, not retried; unknown-model 404 unaffected), idempotent image-to-error history mutation preserving call IDs and text. - End-to-end (`fake_llm.rs` + `fake_mcp.rs`): tool call → MCP image result → 404 unsupported-image → same-turn recovery. Captured requests prove round 2 carried the image, round 3 replays no image, carries the guidance text, preserves pairing, and ends `end_turn`. - Loop guard: typed unsupported-image error with **no** image in history fails after exactly one provider request instead of spinning — mutation-testing showed deleting the `removed == 0` guard survived the suite, and `max_rounds` defaults to unlimited in production, so this branch needed direct coverage. Verified at `a210305019b33d5f56677b4c82bab79e4ac52d24`: `cargo test -p buzz-agent` (full package, 381 unit + all integration suites) green; `clippy --all-targets -D warnings` green; `fmt --check` green; pre-push hooks (rust-tests, desktop-tauri-checks, branch-skew) green. **Scope of the classification guarantee**: the classifier runs in the shared `post()` (which Anthropic and OpenAI paths route through) and in `openrouter_post()` — i.e., every 404 path in `llm.rs`. It only runs on 404 responses; providers that reject images with a different status (e.g. a 400) are out of scope for this PR — see the review-comment discussion for why broadening the phrase list alone would not cover them. Authored by Wren, loop-guard test by Sami, reviewed by Eva. --------- Signed-off-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz> Signed-off-by: Sami <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@buzz.block.builderlab.xyz> Co-authored-by: Wren <5217c5c2f7bfb4333e46d17c98a9255a52dadee18dcd43a43536b95e6776dfa0@buzz.block.builderlab.xyz> Co-authored-by: Sami <f4a42a97e594b77bdbd8ee35191c8b28a94a4cb871d96f32921558275421fb68@buzz.block.builderlab.xyz>
1 parent 067c085 commit 8a7eb8d

5 files changed

Lines changed: 335 additions & 13 deletions

File tree

‎crates/buzz-agent/src/agent.rs‎

Lines changed: 87 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,34 @@ use crate::wire::{self, WireSender};
2121
const ERROR_REFLECTION_SUFFIX: &str =
2222
"\n\n[Reflect] Before retrying, identify the cause and change your approach.";
2323

24+
const UNSUPPORTED_IMAGE_TOOL_MESSAGE: &str = "The current model does not support image input. The image was removed from conversation history so this turn can continue. Use a text-based inspection tool or ask the user for a textual description instead.";
25+
26+
/// Remove image blocks that the provider has explicitly rejected while keeping
27+
/// their surrounding tool result (and therefore the tool-call/result pairing)
28+
/// intact. Returns the number of images removed; zero means the provider error
29+
/// cannot be safely recovered by mutating history.
30+
fn replace_unsupported_images(history: &mut [HistoryItem]) -> usize {
31+
let mut replaced = 0;
32+
for item in history {
33+
let HistoryItem::ToolResult(result) = item else {
34+
continue;
35+
};
36+
let before = result.content.len();
37+
result
38+
.content
39+
.retain(|content| !matches!(content, ToolResultContent::Image { .. }));
40+
let removed = before - result.content.len();
41+
if removed > 0 {
42+
replaced += removed;
43+
result.is_error = true;
44+
result.content.push(ToolResultContent::Text(
45+
UNSUPPORTED_IMAGE_TOOL_MESSAGE.to_string(),
46+
));
47+
}
48+
}
49+
replaced
50+
}
51+
2452
/// Maximum reply reminders emitted per prompt when `require_reply` is on.
2553
///
2654
/// After this many, the turn is allowed to end whether or not anything was
@@ -249,10 +277,10 @@ impl RunCtx<'_> {
249277
tools.push(builtin::load_skill_def());
250278
}
251279
round = round.saturating_add(1);
252-
let response = tokio::select! {
280+
let response_result = tokio::select! {
253281
biased;
254282
_ = self.cancel.changed() => return Ok(StopReason::Cancelled),
255-
r = self.llm.complete(self.cfg, self.system_prompt, self.history, &tools, self.effective_model) => r?,
283+
r = self.llm.complete(self.cfg, self.system_prompt, self.history, &tools, self.effective_model) => r,
256284
_ = async {
257285
// Keepalive ticker: emit a lightweight session update every 30s
258286
// while waiting on the LLM provider. This resets the ACP harness
@@ -275,6 +303,22 @@ impl RunCtx<'_> {
275303
}
276304
} => unreachable!(),
277305
};
306+
let response = match response_result {
307+
Ok(response) => response,
308+
Err(AgentError::UnsupportedImageInput(detail)) => {
309+
let removed = replace_unsupported_images(self.history);
310+
if removed == 0 {
311+
return Err(AgentError::UnsupportedImageInput(detail));
312+
}
313+
tracing::warn!(
314+
model = self.effective_model,
315+
removed_images = removed,
316+
"provider rejected image input; removed images from history and continuing turn"
317+
);
318+
continue;
319+
}
320+
Err(error) => return Err(error),
321+
};
278322

279323
// Record provider-reported input usage so the next loop iteration's
280324
// handoff gate can compare it against the token budget. We capture
@@ -1075,6 +1119,47 @@ mod tests {
10751119
assert!(total_after <= max_bytes);
10761120
}
10771121

1122+
#[test]
1123+
fn unsupported_images_become_recoverable_tool_errors() {
1124+
let mut history = vec![
1125+
HistoryItem::Assistant {
1126+
text: String::new(),
1127+
tool_calls: vec![ToolCall {
1128+
provider_id: "call-image".into(),
1129+
name: "dev__view_image".into(),
1130+
arguments: json!({ "source": "spec.png" }),
1131+
provider_extra: Default::default(),
1132+
}],
1133+
reasoning_details: None,
1134+
},
1135+
HistoryItem::ToolResult(ToolResult {
1136+
provider_id: "call-image".into(),
1137+
content: vec![
1138+
ToolResultContent::Text("10x10 image from spec.png".into()),
1139+
ToolResultContent::Image {
1140+
data: "aW1n".into(),
1141+
mime_type: "image/png".into(),
1142+
},
1143+
],
1144+
is_error: false,
1145+
}),
1146+
];
1147+
1148+
assert_eq!(replace_unsupported_images(&mut history), 1);
1149+
let HistoryItem::ToolResult(result) = &history[1] else {
1150+
panic!("tool result must stay paired with the assistant tool call");
1151+
};
1152+
assert_eq!(result.provider_id, "call-image");
1153+
assert!(result.is_error);
1154+
assert!(result
1155+
.content
1156+
.iter()
1157+
.all(|content| !matches!(content, ToolResultContent::Image { .. })));
1158+
assert!(result.text().contains("does not support image input"));
1159+
assert!(result.text().contains("10x10 image from spec.png"));
1160+
assert_eq!(replace_unsupported_images(&mut history), 0);
1161+
}
1162+
10781163
#[test]
10791164
fn truncate_history_noop_when_under_budget() {
10801165
let mut history = vec![

‎crates/buzz-agent/src/llm.rs‎

Lines changed: 43 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1706,6 +1706,11 @@ fn is_retryable_transport_error(e: &reqwest::Error) -> bool {
17061706
e.is_timeout() || e.is_connect() || e.is_request()
17071707
}
17081708

1709+
fn is_unsupported_image_input_error(body: &str) -> bool {
1710+
body.to_ascii_lowercase()
1711+
.contains("no endpoints found that support image input")
1712+
}
1713+
17091714
/// Build the terminal `AgentError::Llm` for a `post()` exit that has given up
17101715
/// retrying — persistent retryable status, transport failure, or a body-read
17111716
/// break. `detail` carries the specific cause (status/body, or the transport
@@ -1864,9 +1869,14 @@ where
18641869
// upstream capacity — no retry was attempted, so cumulative duration
18651870
// would be misleading.
18661871
if status == 404 {
1872+
let error_body = read_error_body(resp).await;
1873+
if is_unsupported_image_input_error(&error_body) {
1874+
return Err(PostError::Agent(AgentError::UnsupportedImageInput(
1875+
error_body,
1876+
)));
1877+
}
18671878
return Err(PostError::Agent(AgentError::LlmModelNotFound(format!(
1868-
"{status}: {}",
1869-
read_error_body(resp).await
1879+
"{status}: {error_body}"
18701880
))));
18711881
}
18721882
if !status.is_success() {
@@ -2117,6 +2127,9 @@ async fn openrouter_post(
21172127
// about the model, and reporting a parameter problem as
21182128
// `LlmModelNotFound` (or vice versa) sends the user to the wrong fix.
21192129
let error_body = read_error_body(resp).await;
2130+
if is_unsupported_image_input_error(&error_body) {
2131+
return Err(AgentError::UnsupportedImageInput(error_body));
2132+
}
21202133
if error_body.contains("No endpoints found that can handle the requested parameters") {
21212134
return Err(openrouter_parameter_routing_error(&error_body));
21222135
}
@@ -6217,6 +6230,34 @@ mod tests {
62176230
);
62186231
}
62196232

6233+
/// A provider's explicit image-capability rejection is a recoverable typed
6234+
/// error, not a missing model. The agent loop uses this signal to remove the
6235+
/// image from history before retrying the next LLM round.
6236+
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
6237+
async fn openrouter_post_404_unsupported_image_is_typed_and_not_retried() {
6238+
let (url, _captured, attempts) = spawn_openrouter_stub(vec![CannedResponse::new(
6239+
404,
6240+
r#"{"error":{"message":"No endpoints found that support image input"}}"#,
6241+
)])
6242+
.await;
6243+
let http = Client::builder()
6244+
.timeout(Duration::from_secs(5))
6245+
.build()
6246+
.unwrap();
6247+
let err = openrouter_post(&http, &format!("{url}/x"), &json!({}), "key")
6248+
.await
6249+
.unwrap_err();
6250+
assert!(
6251+
matches!(&err, AgentError::UnsupportedImageInput(s) if s.contains("support image input")),
6252+
"image rejection must reach the history-recovery path: got {err:?}"
6253+
);
6254+
assert_eq!(
6255+
attempts.load(std::sync::atomic::Ordering::SeqCst),
6256+
1,
6257+
"a deterministic capability rejection must not be retried"
6258+
);
6259+
}
6260+
62206261
/// Every other 404 still maps to `LlmModelNotFound`, including one that
62216262
/// shares the `No endpoints found` prefix but is about the model rather than
62226263
/// the parameters — the discriminator is narrow enough that a genuinely

‎crates/buzz-agent/src/types.rs‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -388,6 +388,10 @@ pub enum AgentError {
388388
Llm(String),
389389
LlmAuth(String),
390390
LlmModelNotFound(String),
391+
/// The provider explicitly rejected image content for the selected model.
392+
/// Kept distinct so the agent loop can remove the unsupported image from
393+
/// replayed history and give the model a recoverable tool error.
394+
UnsupportedImageInput(String),
391395
Mcp(String),
392396
Cancelled,
393397
}
@@ -399,6 +403,7 @@ impl std::fmt::Display for AgentError {
399403
Self::Llm(s) => write!(f, "llm: {s}"),
400404
Self::LlmAuth(s) => write!(f, "llm auth: {s}"),
401405
Self::LlmModelNotFound(s) => write!(f, "llm model not found: {s}"),
406+
Self::UnsupportedImageInput(s) => write!(f, "llm image input unsupported: {s}"),
402407
Self::Mcp(s) => write!(f, "mcp: {s}"),
403408
Self::Cancelled => write!(f, "cancelled"),
404409
}

‎crates/buzz-agent/tests/bin/fake_mcp.rs‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212
//! (use a large value, e.g. 999, to simulate hang)
1313
//! FAKE_MCP_RESULT_SIZE=N — `tools/call` returns an N-byte text result
1414
//! (default: the literal "ok"); grows history
15+
//! FAKE_MCP_IMAGE_RESULT=1 — `tools/call` returns text plus a PNG image block
1516
//! FAKE_MCP_PID_FILE=path — write the child PID to `path` on startup
1617
//! (for tests that want to verify the child died)
1718
//! FAKE_MCP_SPAWN_GRANDCHILD=1
@@ -300,10 +301,18 @@ fn main() {
300301
} else {
301302
"ok".to_owned()
302303
};
304+
let content = if env_flag("FAKE_MCP_IMAGE_RESULT") {
305+
json!([
306+
{ "type": "text", "text": result_text },
307+
{ "type": "image", "data": "aW1n", "mimeType": "image/png" },
308+
])
309+
} else {
310+
json!([{ "type": "text", "text": result_text }])
311+
};
303312
write_response(
304313
id,
305314
json!({
306-
"content": [{ "type": "text", "text": result_text }],
315+
"content": content,
307316
"isError": false,
308317
}),
309318
);

0 commit comments

Comments
 (0)