Skip to content

Commit c7b6636

Browse files
tlongwell-blockEva
andauthored
fix(buzz-agent): budget summarizer reasoning separately so it cannot starve the handoff summary (#5248)
## Problem The handoff summarizer sends `max_tokens: 8192` (`HANDOFF_MAX_OUTPUT_TOKENS`) with no reasoning budget separation. On reasoning models, thinking tokens count against that cap: the model can spend the entire budget reasoning, length-stop with empty `content`, and `summarize()` — which only reads `content` — reports an empty summary. The handoff then degrades to lossy history truncation. Observed on deepseek-v4-flash during a terminal-bench 2.1 run (tb21-solo-3, 89 tasks): **13 consecutive handoff attempts across 5 trials failed exactly this way** (`handoff returned empty summary; truncating`), each burning ~3 minutes of full-cap reasoning, before a stochastically-short reasoning run finally fit. circuit-fibsqrt alone: 5 failures, 5 truncations, then success on attempt 6. video-processing failed its task by one frame after 3 context truncations. ## Fix `openrouter_summary_body` now grants reasoning its own equal-sized budget and excludes it from the response: - `reasoning.max_tokens = max_output_tokens` — thinking gets a dedicated budget instead of competing with the summary text - `reasoning.exclude = true` — reasoning is never in the response body; `summarize()` only reads `content` - `max_tokens = max_output_tokens * 2` — the total cap covers both budgets, so the text budget the caller asked for is actually available for text Non-reasoning endpoints ignore the `reasoning` object. Deliberately not paired with `provider.require_parameters`, for the reasons documented at `apply_openrouter_mutations` (it hard-404s valid model ids). The prior test `openrouter_summary_carries_neither_reasoning_nor_provider` asserted `reasoning` absent from the summary body — that assertion guarded against *effort-based* reasoning leaking in from config (the body is built independently of `cfg`, which is still true and still tested: `reasoning.effort` stays unset). Replaced with `openrouter_summary_budgets_reasoning_separately_and_carries_no_provider`. ## Verification - `cargo test -p buzz-agent`: 422 unit + 110 integration tests pass at bb2fedd - `cargo fmt` / `cargo clippy -p buzz-agent --all-targets`: clean - Not yet validated against a live OpenRouter reasoning endpoint — the failing scenario needs a long-context session to trigger organically. Evidence for the mechanism is from run artifacts (13/13 empty-summary length-stops on deepseek-v4-flash) and OpenRouter's documented `reasoning.max_tokens`/`reasoning.exclude` semantics. --------- Signed-off-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz> Co-authored-by: Eva <011987e296fd5006292d2f930b574be47c7801048d1983c46c425d3c95f0cffd@buzz.block.builderlab.xyz>
1 parent 65834d6 commit c7b6636

2 files changed

Lines changed: 147 additions & 18 deletions

File tree

‎crates/buzz-agent/src/handoff.rs‎

Lines changed: 85 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,7 @@ use crate::config::{
33
HANDOFF_MAX_OUTPUT_TOKENS, HANDOFF_MAX_TOOL_NAMES, HANDOFF_MIN_PROMPT_BUDGET_BYTES,
44
HANDOFF_ORIGINAL_TASK_MAX_BYTES, MAX_CONTEXT_RECOVERIES_PER_RUN,
55
};
6+
use crate::llm::summary_completion_cap;
67
use crate::types::HistoryItem;
78

89
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -35,10 +36,21 @@ pub(crate) enum ContextRecovery {
3536
Exhausted,
3637
}
3738

38-
const HANDOFF_SYSTEM_PROMPT: &str = "You are generating a context handoff summary for the next \
39-
turn of an autonomous agent. Be concise but thorough. Cover: what the original task was, what \
40-
you accomplished, key decisions made, what remains, and one concrete next step. Output plain \
41-
text only — no tool calls, no JSON. Stay under 8192 tokens.";
39+
/// System prompt for the handoff summarizer. `LazyLock` + `format!` so the
40+
/// token figure is derived from [`HANDOFF_MAX_OUTPUT_TOKENS`] instead of a
41+
/// duplicated literal, and "visible plain-text summary" makes explicit that
42+
/// the limit is on summary text, not on any hidden reasoning the model does
43+
/// first (which is budgeted separately on the wire — see
44+
/// `openrouter_summary_body`).
45+
static HANDOFF_SYSTEM_PROMPT: std::sync::LazyLock<String> = std::sync::LazyLock::new(|| {
46+
format!(
47+
"You are generating a context handoff summary for the next turn of an autonomous agent. \
48+
Be concise but thorough. Cover: what the original task was, what you accomplished, key \
49+
decisions made, what remains, and one concrete next step. Output plain text only — no \
50+
tool calls, no JSON. Keep the visible plain-text summary under \
51+
{HANDOFF_MAX_OUTPUT_TOKENS} tokens."
52+
)
53+
});
4254

4355
impl RunCtx<'_> {
4456
pub(crate) async fn maybe_handoff(&mut self, handoff_attempts: &mut usize) -> HandoffOutcome {
@@ -170,7 +182,7 @@ impl RunCtx<'_> {
170182
_ = self.cancel.changed() => return HandoffOutcome::Cancelled,
171183
r = self.llm.summarize(
172184
self.cfg,
173-
HANDOFF_SYSTEM_PROMPT,
185+
&HANDOFF_SYSTEM_PROMPT,
174186
&prompt,
175187
HANDOFF_MAX_OUTPUT_TOKENS,
176188
self.effective_model,
@@ -331,7 +343,7 @@ impl RunCtx<'_> {
331343
Some(explicit) => explicit.saturating_sub(fixed_bytes),
332344
None => handoff_prompt_budget_bytes(
333345
self.cfg.max_context_tokens,
334-
HANDOFF_MAX_OUTPUT_TOKENS,
346+
summary_completion_cap(self.cfg.provider, HANDOFF_MAX_OUTPUT_TOKENS),
335347
fixed_bytes,
336348
),
337349
};
@@ -513,8 +525,9 @@ fn byte_fallback_threshold(
513525
mod tests {
514526
use super::{
515527
byte_fallback_threshold, estimate_tokens_from_bytes, handoff_prompt_budget_bytes,
516-
token_threshold,
528+
summary_completion_cap, token_threshold, HANDOFF_SYSTEM_PROMPT,
517529
};
530+
use crate::config::{Provider, HANDOFF_MAX_OUTPUT_TOKENS};
518531

519532
#[test]
520533
fn handoff_prompt_budget_reserves_summary_output_and_fixed_prompt() {
@@ -526,6 +539,71 @@ mod tests {
526539
assert_eq!(handoff_prompt_budget_bytes(1_000, 2_000, 10_000), 0);
527540
}
528541

542+
/// OpenRouter's summary request grants reasoning an equal budget on top of
543+
/// the visible-text budget, so its completion cap is 2× the handoff text
544+
/// budget; the input budget must reserve that doubled cap. At the
545+
/// 1-byte/token upper bound, prompt bytes bound prompt tokens, so the join
546+
/// to pin is: (budget + fixed prompt) + actual completion cap ≤ window.
547+
/// Reserving only `HANDOFF_MAX_OUTPUT_TOKENS` would break this by exactly
548+
/// one extra reasoning budget at the maximum constructed prompt.
549+
#[test]
550+
fn openrouter_prompt_budget_reserves_doubled_completion_cap() {
551+
let cap = summary_completion_cap(Provider::OpenRouter, HANDOFF_MAX_OUTPUT_TOKENS);
552+
assert_eq!(
553+
cap,
554+
2 * HANDOFF_MAX_OUTPUT_TOKENS,
555+
"OpenRouter doubles: text + reasoning"
556+
);
557+
let window = 200_000u64;
558+
let fixed = 1_000usize;
559+
let budget = handoff_prompt_budget_bytes(window, cap, fixed);
560+
assert_eq!(budget, 182_616); // 200_000 - 16_384 - 1_000
561+
let max_prompt_tokens = estimate_tokens_from_bytes(budget + fixed);
562+
assert!(
563+
max_prompt_tokens + u64::from(cap) <= window,
564+
"input + completion allowance must fit the configured window"
565+
);
566+
// The old single reservation violates the same join — the regression
567+
// this guards against.
568+
let stale_budget = handoff_prompt_budget_bytes(window, HANDOFF_MAX_OUTPUT_TOKENS, fixed);
569+
assert!(
570+
estimate_tokens_from_bytes(stale_budget + fixed) + u64::from(cap) > window,
571+
"reserving only the text budget must be observable as an overflow here"
572+
);
573+
}
574+
575+
/// Anthropic/OpenAI/Databricks summary bodies request exactly the caller's
576+
/// budget, so their input reservation is unchanged.
577+
#[test]
578+
fn non_openrouter_completion_cap_is_the_callers_budget() {
579+
for provider in [
580+
Provider::Anthropic,
581+
Provider::OpenAi,
582+
Provider::Databricks,
583+
Provider::DatabricksV2,
584+
] {
585+
assert_eq!(
586+
summary_completion_cap(provider, HANDOFF_MAX_OUTPUT_TOKENS),
587+
HANDOFF_MAX_OUTPUT_TOKENS
588+
);
589+
}
590+
}
591+
592+
/// The prompt's token figure is derived from `HANDOFF_MAX_OUTPUT_TOKENS`
593+
/// and names the *visible plain-text summary* as its target, so hidden
594+
/// reasoning (budgeted separately on the wire) is not the referent.
595+
#[test]
596+
fn handoff_system_prompt_derives_limit_and_targets_visible_text() {
597+
let expected = format!(
598+
"Keep the visible plain-text summary under {HANDOFF_MAX_OUTPUT_TOKENS} tokens."
599+
);
600+
assert!(
601+
HANDOFF_SYSTEM_PROMPT.contains(&expected),
602+
"prompt must derive its token figure from HANDOFF_MAX_OUTPUT_TOKENS: {}",
603+
*HANDOFF_SYSTEM_PROMPT
604+
);
605+
}
606+
529607
#[test]
530608
fn token_threshold_uses_fraction_when_output_is_small() {
531609
// 200k window, 1k output. fractional = 0.9*200000 = 180000;

‎crates/buzz-agent/src/llm.rs‎

Lines changed: 62 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -2237,22 +2237,56 @@ pub(crate) fn build_token_source(cfg: &Config) -> Result<Arc<dyn TokenSource>, A
22372237
}
22382238
}
22392239

2240+
/// Completion-token cap that [`Llm::summarize`] actually requests from
2241+
/// `provider`, given the caller's visible-text budget. OpenRouter grants
2242+
/// reasoning a separate, equal budget on top of the text budget (see
2243+
/// [`openrouter_summary_body`]), so its top-level cap is double the caller's
2244+
/// budget; every other provider requests the caller's budget unchanged.
2245+
/// Callers that reserve output headroom in an input budget
2246+
/// (`handoff_prompt_budget_bytes`) must reserve THIS value, not the text
2247+
/// budget — otherwise input plus the actual completion allowance can exceed
2248+
/// the configured context window.
2249+
pub(crate) fn summary_completion_cap(provider: Provider, max_output_tokens: u32) -> u32 {
2250+
match provider {
2251+
Provider::OpenRouter => max_output_tokens.saturating_mul(2),
2252+
Provider::Anthropic | Provider::OpenAi | Provider::Databricks | Provider::DatabricksV2 => {
2253+
max_output_tokens
2254+
}
2255+
}
2256+
}
2257+
22402258
/// Build the request body for `Llm::summarize` on `Provider::OpenRouter`.
22412259
/// Extracted so tests can assert on the actual wire shape instead of a
2242-
/// hand-rolled literal — summaries never carry `reasoning` (see
2243-
/// `apply_openrouter_mutations`, which the summary path never calls).
2244-
/// It spells the token limit `max_tokens` directly for the same reason: the
2245-
/// mutation that renames it is never applied here.
2260+
/// hand-rolled literal — summaries never carry config-driven reasoning
2261+
/// *effort* (see `apply_openrouter_mutations`, which the summary path never
2262+
/// calls). It spells the token limit `max_tokens` directly for the same
2263+
/// reason: the mutation that renames it is never applied here.
22462264
fn openrouter_summary_body(
22472265
effective_model: &str,
22482266
system_prompt: &str,
22492267
user_prompt: &str,
22502268
max_output_tokens: u32,
22512269
) -> Value {
2270+
// Reasoning models spend output tokens thinking before emitting any
2271+
// visible text, and that spend counts against `max_tokens`. Left
2272+
// unseparated, a model can burn the entire cap mid-reasoning and return an
2273+
// empty `content` — observed with deepseek-v4-flash, where 13 consecutive
2274+
// handoff attempts length-stopped inside the reasoning channel and every
2275+
// one degraded to lossy history truncation. Give reasoning its own
2276+
// equal-sized budget on top of the text budget so `max_output_tokens`
2277+
// remains what the caller means: visible summary text. `exclude` keeps the
2278+
// reasoning out of the response body; `summarize()` only reads `content`.
2279+
// Non-reasoning endpoints ignore the `reasoning` object (see
2280+
// `apply_openrouter_mutations` on why it is never paired with
2281+
// `provider.require_parameters`).
22522282
json!({
22532283
"model": effective_model,
22542284
"stream": false,
2255-
"max_tokens": max_output_tokens,
2285+
"max_tokens": summary_completion_cap(Provider::OpenRouter, max_output_tokens),
2286+
"reasoning": {
2287+
"max_tokens": max_output_tokens,
2288+
"exclude": true,
2289+
},
22562290
"messages": [
22572291
{ "role": "system", "content": system_prompt },
22582292
{ "role": "user", "content": user_prompt },
@@ -6263,8 +6297,14 @@ mod tests {
62636297
assert!(body.get("max_completion_tokens").is_none());
62646298
}
62656299

6300+
/// The summary body reserves `max_output_tokens` for visible text by
6301+
/// granting reasoning a separate, equal budget on top and excluding it
6302+
/// from the response. Without the separation, a reasoning model can spend
6303+
/// the entire cap thinking and length-stop with empty `content`, which
6304+
/// `summarize()` reports as an empty summary and the handoff degrades to
6305+
/// lossy truncation.
62666306
#[test]
6267-
fn openrouter_summary_carries_neither_reasoning_nor_provider() {
6307+
fn openrouter_summary_budgets_reasoning_separately_and_carries_no_provider() {
62686308
let body = openrouter_summary_body(
62696309
"anthropic/claude-opus-4-7",
62706310
"summarize",
@@ -6274,14 +6314,25 @@ mod tests {
62746314
assert_eq!(body["model"], "anthropic/claude-opus-4-7");
62756315
assert_eq!(body["messages"][0]["role"], "system");
62766316
assert_eq!(body["messages"][1]["content"], "text to summarize");
6277-
assert_eq!(body["max_tokens"], 1024);
6317+
assert_eq!(
6318+
body["max_tokens"], 2048,
6319+
"total cap must cover the text budget plus the reasoning budget"
6320+
);
6321+
assert_eq!(
6322+
body["reasoning"]["max_tokens"], 1024,
6323+
"reasoning gets its own budget so it cannot starve the summary text"
6324+
);
6325+
assert_eq!(
6326+
body["reasoning"]["exclude"], true,
6327+
"reasoning must not be included in the response; summarize() reads only content"
6328+
);
62786329
assert!(
6279-
body.get("max_completion_tokens").is_none(),
6280-
"summary body must use OpenRouter's token-limit spelling"
6330+
body["reasoning"].get("effort").is_none(),
6331+
"budget-based cap only; effort stays unset for the summary call"
62816332
);
62826333
assert!(
6283-
body.get("reasoning").is_none(),
6284-
"summary body must not carry reasoning"
6334+
body.get("max_completion_tokens").is_none(),
6335+
"summary body must use OpenRouter's token-limit spelling"
62856336
);
62866337
assert!(
62876338
body.get("provider").is_none(),

0 commit comments

Comments
 (0)