Repository navigation
Commit b043860
Eva
fix(buzz-agent): recover from 400-shaped image rejections; unbound benchmark agent rounds
Two failure modes from the tb21-glm52-crusoe-1 benchmark run wedged or
killed 13 of 89 trials without the model being at fault.
**Conversation poisoning on text-only endpoints.** Crusoe's serverless
`crusoeai/GLM-5.2-NVFP4` rejects any request whose history contains an
image with `400: ... is not a multimodal model`. The recovery path for
exactly this situation already exists (`AgentError::UnsupportedImageInput`
→ `replace_unsupported_images` strips the image blocks and continues the
turn with a text placeholder), but classification only matched
OpenRouter's 404 body ("no endpoints found that support image input") and
was only consulted on the 404 arms. The Crusoe 400 fell through to
terminal `AgentError::Llm`; the image stayed in history, every subsequent
call failed identically, buzz-acp rode its 10-retry ladder (~40 min), and
the trial idled to budget death. Measured: 8 trials, 12.7h aggregate idle.
Fix: extend `is_unsupported_image_input_error()` to also match the
verbatim "is not a multimodal model" body, and consult it on the 400 arms
of both the shared `post()` ladder and `openrouter_post()` (a BYOK/
passthrough upstream can surface the provider's own 400). The matcher
stays deliberately tight, same doctrine as `is_context_length_error`:
misclassifying a generic 400 as recoverable would mutate history for an
error that removing images cannot fix.
**Bounded agent rounds in benchmark trials.** The harness default
`DEFAULT_MAX_AGENT_ROUNDS = 32` ended solo trials mid-work when a turn
rotated (max_tokens rotation on thinking-heavy models burned rounds
fast). Benchmark trials already have a wall-clock budget as the real
limit, so the round cap only converts recoverable rotation into trial
death. Default is now 0 (unbounded — `BUZZ_AGENT_MAX_ROUNDS=0` is the
agent's own documented unbounded value); per-agent `budget.max_calls`
in manifests still overrides.
Tests: 400-body classification asserted through `complete()` on the
shared-post path and at the `openrouter_post` terminal, both proving
single-attempt (no retry of a deterministic capability rejection);
python tests updated for 0-is-legal with a negative arm at -1.
Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>1 parent 02f640b commit b043860
3 files changed
Lines changed: 107 additions & 9 deletions
File tree
- benchmarks/harbor-buzz-orchestra
- src/harbor_buzz_orchestra
- tests
- crates/buzz-agent/src
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
24 | 24 | | |
25 | 25 | | |
26 | 26 | | |
27 | | - | |
| 27 | + | |
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
| |||
80 | 80 | | |
81 | 81 | | |
82 | 82 | | |
83 | | - | |
84 | | - | |
| 83 | + | |
| 84 | + | |
85 | 85 | | |
86 | 86 | | |
87 | 87 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
247 | 247 | | |
248 | 248 | | |
249 | 249 | | |
250 | | - | |
| 250 | + | |
251 | 251 | | |
252 | 252 | | |
253 | 253 | | |
| |||
290 | 290 | | |
291 | 291 | | |
292 | 292 | | |
293 | | - | |
294 | | - | |
295 | | - | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
296 | 299 | | |
297 | 300 | | |
298 | 301 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
1932 | 1932 | | |
1933 | 1933 | | |
1934 | 1934 | | |
| 1935 | + | |
| 1936 | + | |
| 1937 | + | |
| 1938 | + | |
| 1939 | + | |
| 1940 | + | |
| 1941 | + | |
| 1942 | + | |
1935 | 1943 | | |
1936 | | - | |
1937 | | - | |
| 1944 | + | |
| 1945 | + | |
| 1946 | + | |
| 1947 | + | |
| 1948 | + | |
| 1949 | + | |
| 1950 | + | |
| 1951 | + | |
| 1952 | + | |
1938 | 1953 | | |
1939 | 1954 | | |
1940 | 1955 | | |
| |||
2129 | 2144 | | |
2130 | 2145 | | |
2131 | 2146 | | |
| 2147 | + | |
| 2148 | + | |
| 2149 | + | |
| 2150 | + | |
| 2151 | + | |
| 2152 | + | |
| 2153 | + | |
2132 | 2154 | | |
2133 | 2155 | | |
2134 | 2156 | | |
| |||
2535 | 2557 | | |
2536 | 2558 | | |
2537 | 2559 | | |
| 2560 | + | |
| 2561 | + | |
| 2562 | + | |
| 2563 | + | |
| 2564 | + | |
| 2565 | + | |
2538 | 2566 | | |
2539 | 2567 | | |
2540 | 2568 | | |
| |||
7544 | 7572 | | |
7545 | 7573 | | |
7546 | 7574 | | |
| 7575 | + | |
| 7576 | + | |
| 7577 | + | |
| 7578 | + | |
| 7579 | + | |
| 7580 | + | |
| 7581 | + | |
| 7582 | + | |
| 7583 | + | |
| 7584 | + | |
| 7585 | + | |
| 7586 | + | |
| 7587 | + | |
| 7588 | + | |
| 7589 | + | |
| 7590 | + | |
| 7591 | + | |
| 7592 | + | |
| 7593 | + | |
| 7594 | + | |
| 7595 | + | |
| 7596 | + | |
| 7597 | + | |
| 7598 | + | |
| 7599 | + | |
| 7600 | + | |
| 7601 | + | |
| 7602 | + | |
| 7603 | + | |
| 7604 | + | |
| 7605 | + | |
| 7606 | + | |
| 7607 | + | |
| 7608 | + | |
| 7609 | + | |
| 7610 | + | |
| 7611 | + | |
| 7612 | + | |
| 7613 | + | |
| 7614 | + | |
| 7615 | + | |
| 7616 | + | |
| 7617 | + | |
| 7618 | + | |
| 7619 | + | |
| 7620 | + | |
| 7621 | + | |
| 7622 | + | |
| 7623 | + | |
| 7624 | + | |
| 7625 | + | |
| 7626 | + | |
| 7627 | + | |
| 7628 | + | |
| 7629 | + | |
| 7630 | + | |
| 7631 | + | |
| 7632 | + | |
| 7633 | + | |
| 7634 | + | |
| 7635 | + | |
| 7636 | + | |
| 7637 | + | |
| 7638 | + | |
| 7639 | + | |
| 7640 | + | |
| 7641 | + | |
7547 | 7642 | | |
7548 | 7643 | | |
7549 | 7644 | | |
| |||
0 commit comments