Skip to content

Commit bb2fedd

Browse files
author
Eva
committed
fix(buzz-agent): budget summarizer reasoning separately so it cannot starve the handoff summary
Reasoning models spend output tokens thinking before any visible text, and that spend counts against the summary call's max_tokens. On deepseek-v4-flash this starved the handoff summarizer completely: in a 89-task benchmark run, 13 consecutive handoff attempts across 5 trials length-stopped inside the reasoning channel, returned empty content, and every one degraded to lossy history truncation (~40 minutes of burned reasoning) before a stochastically-short reasoning run finally fit. Give reasoning its own equal-sized budget on top of the text budget (reasoning.max_tokens) and exclude it from the response (reasoning.exclude) — summarize() only reads content. max_tokens is raised to cover both budgets so the text budget the caller asked for is actually available for text. Non-reasoning endpoints ignore the reasoning object; deliberately not paired with provider.require_parameters (see apply_openrouter_mutations). Verification: - cargo test -p buzz-agent (422 unit + 110 integration, all pass) - cargo fmt / clippy clean Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
1 parent 2b873cf commit bb2fedd

1 file changed

Lines changed: 40 additions & 7 deletions

File tree

‎crates/buzz-agent/src/llm.rs‎

Lines changed: 40 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -2249,10 +2249,26 @@ fn openrouter_summary_body(
22492249
user_prompt: &str,
22502250
max_output_tokens: u32,
22512251
) -> Value {
2252+
// Reasoning models spend output tokens thinking before emitting any
2253+
// visible text, and that spend counts against `max_tokens`. Left
2254+
// unseparated, a model can burn the entire cap mid-reasoning and return an
2255+
// empty `content` — observed with deepseek-v4-flash, where 13 consecutive
2256+
// handoff attempts length-stopped inside the reasoning channel and every
2257+
// one degraded to lossy history truncation. Give reasoning its own
2258+
// equal-sized budget on top of the text budget so `max_output_tokens`
2259+
// remains what the caller means: visible summary text. `exclude` keeps the
2260+
// reasoning out of the response body; `summarize()` only reads `content`.
2261+
// Non-reasoning endpoints ignore the `reasoning` object (see
2262+
// `apply_openrouter_mutations` on why it is never paired with
2263+
// `provider.require_parameters`).
22522264
json!({
22532265
"model": effective_model,
22542266
"stream": false,
2255-
"max_tokens": max_output_tokens,
2267+
"max_tokens": max_output_tokens.saturating_mul(2),
2268+
"reasoning": {
2269+
"max_tokens": max_output_tokens,
2270+
"exclude": true,
2271+
},
22562272
"messages": [
22572273
{ "role": "system", "content": system_prompt },
22582274
{ "role": "user", "content": user_prompt },
@@ -6263,8 +6279,14 @@ mod tests {
62636279
assert!(body.get("max_completion_tokens").is_none());
62646280
}
62656281

6282+
/// The summary body reserves `max_output_tokens` for visible text by
6283+
/// granting reasoning a separate, equal budget on top and excluding it
6284+
/// from the response. Without the separation, a reasoning model can spend
6285+
/// the entire cap thinking and length-stop with empty `content`, which
6286+
/// `summarize()` reports as an empty summary and the handoff degrades to
6287+
/// lossy truncation.
62666288
#[test]
6267-
fn openrouter_summary_carries_neither_reasoning_nor_provider() {
6289+
fn openrouter_summary_budgets_reasoning_separately_and_carries_no_provider() {
62686290
let body = openrouter_summary_body(
62696291
"anthropic/claude-opus-4-7",
62706292
"summarize",
@@ -6274,14 +6296,25 @@ mod tests {
62746296
assert_eq!(body["model"], "anthropic/claude-opus-4-7");
62756297
assert_eq!(body["messages"][0]["role"], "system");
62766298
assert_eq!(body["messages"][1]["content"], "text to summarize");
6277-
assert_eq!(body["max_tokens"], 1024);
6299+
assert_eq!(
6300+
body["max_tokens"], 2048,
6301+
"total cap must cover the text budget plus the reasoning budget"
6302+
);
6303+
assert_eq!(
6304+
body["reasoning"]["max_tokens"], 1024,
6305+
"reasoning gets its own budget so it cannot starve the summary text"
6306+
);
6307+
assert_eq!(
6308+
body["reasoning"]["exclude"], true,
6309+
"reasoning must not be included in the response; summarize() reads only content"
6310+
);
62786311
assert!(
6279-
body.get("max_completion_tokens").is_none(),
6280-
"summary body must use OpenRouter's token-limit spelling"
6312+
body["reasoning"].get("effort").is_none(),
6313+
"budget-based cap only; effort stays unset for the summary call"
62816314
);
62826315
assert!(
6283-
body.get("reasoning").is_none(),
6284-
"summary body must not carry reasoning"
6316+
body.get("max_completion_tokens").is_none(),
6317+
"summary body must use OpenRouter's token-limit spelling"
62856318
);
62866319
assert!(
62876320
body.get("provider").is_none(),

0 commit comments

Comments
 (0)