Skip to content

Commit 09c86c5

Browse files
atishpatelclaude
andauthored
fix: report agent usage per provider round, not once per turn (block#4545)
## The bug buzz-agent emitted its `usage_update` notification in exactly one place: after `ctx.run()` returned. Until that moment a turn's token counters lived only in the prompt task's stack frame. **A turn killed mid-flight reported nothing at all** — the provider had already billed every round it completed, and no consumer ever saw any of it. That is not a corner case for anything that ends a turn on a clock. It is the normal case for a long-horizon benchmark run that relaunches its agent between phases. ## How big Measured against a provider's own billing ledger over one run's window: | | provider ledger | what we recorded | |---|---|---| | the relaunched lead seat | $485 / 348M tok | $98.99 / 90.3M tok | | the two seats that were not relaunched | $29.90 / 856M | $25.81 / 765M — reconciles | 97% of that run's usage rows came back all zeros, against 1–4% for comparable runs that never relaunch. In one 450-phase trial exactly 7 phases recorded any usage — and each of those carries 177k–437k input tokens, a whole session's worth landing in the one phase that happened to end gracefully. Worth being precise about what was *not* wrong, since both were plausible and both were checked: - **Not pricing.** The rates were verified against the provider's endpoints API and match what we charge. - **Not a truncation bug.** The usage files were intact and internally consistent. The tokens were never captured in the first place. ## The fix The run loop now emits a session-cumulative `usage_update` after every usage-bearing provider response, so an interrupted turn has reported everything but its single in-flight request. - **Emitting more than once per turn is already part of the contract.** buzz-acp's `UsageTracker` advances its committed baseline only at publish time, and goose behaves the same way — which is why the tracker was written to tolerate it. - **The turn-start session baseline is snapshotted into `RunCtx`** so the mid-turn figure stays *session*-cumulative. A turn-local number would be discarded by a high-water-mark consumer and lose the turn entirely; there is a test for exactly that. - **Snapshot by value, not a session handle.** The loop reports once per round, and taking the sessions lock on each would serialise concurrent sessions behind one another's provider round-trips. Nothing else advances those counters while the turn holds `busy`, so it cannot go stale. - **One shared `wire::usage_update_payload`** for both call sites, so the mid-turn and end-of-turn shapes cannot drift. A drift there would present as tokens silently vanishing, which is the failure this reporting exists to prevent. ## Why not a SIGTERM handler That was the obvious shape and it does not work. At signal time the counters are not sitting anywhere a handler could reach — they are in the turn's stack frame, and the value the handler would need has not been folded into the session yet. Making usage durable *during* the turn is what actually fixes it; once it is, a handler adds nothing beyond the in-flight request, whose cost is unknown until its response lands. ## Tests - `usage_is_reported_after_each_round_not_only_at_turn_end` — two rounds; asserts the **first** notification carries round 1's counts alone, proving it went out before round 2 returned. - `mid_turn_usage_includes_earlier_turns` — a mid-turn report must be session-cumulative, not turn-local. buzz-agent 18/18 on the `fake_llm` suite, 382 unit. `cargo fmt` / `clippy` / `cargo check --workspace --all-targets` clean. ## Scope Agent-side only, against `main`. The matching harness change — settling usage on the timeout path, which was skipped on the reasoning that an incomplete turn has nothing to flush — is **block#4553**, against the benchmark branch, since that harness does not exist on `main`. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Signed-off-by: Atish Patel <[email protected]> Co-authored-by: Claude Code <[email protected]>
1 parent 80315ac commit 09c86c5

5 files changed

Lines changed: 269 additions & 24 deletions

File tree

‎crates/buzz-agent/src/agent.rs‎

Lines changed: 50 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,8 @@ use crate::mcp::McpRegistry;
1313
use crate::mcp::ResultBudget;
1414

1515
use crate::types::{
16-
AgentError, ContentBlock, HistoryItem, ProviderStop, StopReason, ToolCall, ToolResult,
17-
ToolResultContent, TurnTotalState,
16+
AgentError, ContentBlock, HistoryItem, ProviderStop, SessionUsageBaseline, StopReason,
17+
ToolCall, ToolResult, ToolResultContent, TurnTotalState,
1818
};
1919
use crate::wire::{self, WireSender};
2020

@@ -150,9 +150,40 @@ pub struct RunCtx<'a> {
150150
/// Reset to `Unseen` at turn start in `run()`. Callers must not derive a
151151
/// total by summing input+output — that is the UI display approximation only.
152152
pub turn_total_state: &'a mut TurnTotalState,
153+
/// Session-cumulative counters as they stood when this turn began. Added to
154+
/// the `turn_*` accumulators above to report a cumulative figure mid-turn;
155+
/// the session's own copy is only advanced once, after the turn returns.
156+
pub usage_baseline: SessionUsageBaseline,
153157
}
154158

155159
impl RunCtx<'_> {
160+
/// Send a session-cumulative `usage_update` reflecting everything observed
161+
/// up to and including the most recent LLM response.
162+
///
163+
/// The figure is the turn-start baseline plus this turn's running
164+
/// accumulators, which is exactly what `session/prompt` will fold into the
165+
/// session once the turn returns — so a mid-turn notification and the
166+
/// end-of-turn one agree, and a turn that never returns has still reported
167+
/// everything but its final in-flight request.
168+
async fn emit_usage_update(&self) {
169+
let base = self.usage_baseline;
170+
let payload = wire::usage_update_payload(
171+
base.input_tokens
172+
.saturating_add(self.turn_input_tokens.unwrap_or(0)),
173+
base.output_tokens
174+
.saturating_add(self.turn_output_tokens.unwrap_or(0)),
175+
base.cached_input_tokens
176+
.saturating_add(self.turn_cached_input_tokens.unwrap_or(0)),
177+
base.total_state.merge_session(*self.turn_total_state),
178+
self.effective_model,
179+
);
180+
wire::send(
181+
self.wire,
182+
wire::goose_session_update(self.session_id, payload),
183+
)
184+
.await;
185+
}
186+
156187
pub async fn run(&mut self, prompt: Vec<ContentBlock>) -> Result<StopReason, AgentError> {
157188
let user_text = prompt_to_text(prompt)?;
158189
if user_text.len() > MAX_PROMPT_BYTES {
@@ -299,6 +330,23 @@ impl RunCtx<'_> {
299330
// this gate rather than representing absent categories as zero.
300331
if response.input_tokens.is_some() || response.output_tokens.is_some() {
301332
*self.turn_total_state = self.turn_total_state.fold(response.total_tokens);
333+
// Report what the turn has burned SO FAR, before running the
334+
// next round. A turn is many provider round-trips over many
335+
// minutes, and until this point the only report was the one
336+
// `session/prompt` sends after the turn returns — so a turn
337+
// that was cancelled, timed out, or whose process was killed
338+
// reported nothing at all, and its tokens (already billed)
339+
// existed only in this stack frame. Reporting per round bounds
340+
// the loss to the single request in flight.
341+
//
342+
// Emitting more than one `usage_update` per turn is expected by
343+
// the consumer: buzz-acp's UsageTracker advances its committed
344+
// baseline only when the turn's metric is published, so every
345+
// notification within a turn measures from the same frozen
346+
// baseline and the last one seen is the turn's true total.
347+
// goose behaves the same way, which is why the tracker was
348+
// written to tolerate it.
349+
self.emit_usage_update().await;
302350
}
303351

304352
if !response.reasoning.is_empty() {

‎crates/buzz-agent/src/lib.rs‎

Lines changed: 24 additions & 22 deletions
Original file line numberDiff line numberDiff line change
@@ -658,6 +658,7 @@ async fn run_prompt(app: Arc<App>, id: Value, params: Value, wire_tx: WireSender
658658
effective_model_override,
659659
run_id,
660660
mut steer_rx,
661+
usage_baseline,
661662
) = match acquire_session(&app, &p.session_id).await {
662663
Ok(v) => v,
663664
Err(reason) => {
@@ -709,6 +710,7 @@ async fn run_prompt(app: Arc<App>, id: Value, params: Value, wire_tx: WireSender
709710
turn_output_tokens: &mut turn_output_tokens,
710711
turn_cached_input_tokens: &mut turn_cached_input_tokens,
711712
turn_total_state: &mut turn_total_state,
713+
usage_baseline,
712714
};
713715
let result = ctx.run(p.prompt).await;
714716
if let Some(s) = app.sessions.lock().await.get_mut(&sid) {
@@ -766,28 +768,16 @@ async fn run_prompt(app: Arc<App>, id: Value, params: Value, wire_tx: WireSender
766768
if let Some((accumulated_in, accumulated_out, accumulated_cached, accumulated_total)) =
767769
accumulated
768770
{
769-
// Build the usage_update payload. `accumulatedTotalTokens` is only
770-
// included when the cumulative is exactly known — never when Unseen
771-
// (no total ever observed) or Unknown (at least one turn lacked a
772-
// total). A goose consumer that doesn't recognise the field ignores it.
773-
let mut update = serde_json::json!({
774-
"sessionUpdate": "usage_update",
775-
// used: total tokens as a context-usage proxy;
776-
// contextLimit: 0 (buzz-agent has no context limit tracking).
777-
"used": accumulated_in.saturating_add(accumulated_out),
778-
"contextLimit": 0u64,
779-
"accumulatedInputTokens": accumulated_in,
780-
"accumulatedOutputTokens": accumulated_out,
781-
// A subset of accumulatedInputTokens, not an addition to
782-
// it. Extends goose's usage_update shape; a consumer that
783-
// does not know the field ignores it and prices exactly as
784-
// it did before.
785-
"accumulatedCachedInputTokens": accumulated_cached,
786-
"model": effective_model_str,
787-
});
788-
if let crate::types::TurnTotalState::Exact(total) = accumulated_total {
789-
update["accumulatedTotalTokens"] = serde_json::json!(total);
790-
}
771+
// Same builder the run loop uses for its per-round reports, so the
772+
// final notification is shape-identical to the ones that preceded
773+
// it and a consumer taking the high-water mark lands on this one.
774+
let update = wire::usage_update_payload(
775+
accumulated_in,
776+
accumulated_out,
777+
accumulated_cached,
778+
accumulated_total,
779+
effective_model_str,
780+
);
791781
wire::send(&wire_tx, goose_session_update(&sid, update)).await;
792782
}
793783
}
@@ -821,6 +811,7 @@ async fn acquire_session(
821811
Option<String>,
822812
String,
823813
mpsc::UnboundedReceiver<Vec<ContentBlock>>,
814+
crate::types::SessionUsageBaseline,
824815
),
825816
&'static str,
826817
> {
@@ -857,6 +848,17 @@ async fn acquire_session(
857848
effective_model,
858849
run_id,
859850
steer_rx,
851+
// Snapshot rather than a handle: the run loop reports cumulative usage
852+
// after every LLM round, and taking the sessions lock on each of those
853+
// would serialise concurrent sessions behind one another's provider
854+
// round-trips. Nothing else advances these counters while this turn
855+
// holds `busy`, so the snapshot cannot go stale under it.
856+
crate::types::SessionUsageBaseline {
857+
input_tokens: s.accumulated_input_tokens,
858+
output_tokens: s.accumulated_output_tokens,
859+
cached_input_tokens: s.accumulated_cached_input_tokens,
860+
total_state: s.accumulated_total_state,
861+
},
860862
))
861863
}
862864

‎crates/buzz-agent/src/types.rs‎

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -308,6 +308,30 @@ impl TurnTotalState {
308308
}
309309
}
310310

311+
/// The session-cumulative usage counters as of the START of a turn.
312+
///
313+
/// Copied out of the session under the lock when a turn begins and handed to
314+
/// `RunCtx` by value, so the run loop can emit a cumulative `usage_update`
315+
/// after every LLM round without reaching back into `App.sessions` (which it
316+
/// holds no handle to, and which is locked by the turn's own bookkeeping at
317+
/// both ends).
318+
///
319+
/// This exists so that usage is durable *during* a turn rather than only after
320+
/// it. The counters a turn accrues live in the prompt task's stack frame until
321+
/// the turn returns; a process killed mid-turn takes them with it and the
322+
/// tokens are billed by the provider but recorded nowhere. That is not
323+
/// hypothetical — it silently under-reported a long-horizon benchmark's cost by
324+
/// several-fold, because every phase of a `continue_until_timeout` run is
325+
/// terminated mid-turn by design.
326+
#[derive(Debug, Clone, Copy, Default)]
327+
pub struct SessionUsageBaseline {
328+
pub input_tokens: u64,
329+
pub output_tokens: u64,
330+
/// The cache-served subset of `input_tokens`, not an addition to it.
331+
pub cached_input_tokens: u64,
332+
pub total_state: TurnTotalState,
333+
}
334+
311335
#[derive(Debug, Clone, Copy, PartialEq)]
312336
pub enum StopReason {
313337
EndTurn,

‎crates/buzz-agent/src/wire.rs‎

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -148,6 +148,48 @@ pub fn goose_session_update(sid: &str, update: Value) -> Value {
148148
})
149149
}
150150

151+
/// Build the `usage_update` payload for a `_goose/unstable/session/update`.
152+
///
153+
/// Shared by the two places that report usage — after each LLM round inside a
154+
/// turn, and once more when the turn completes — so the wire shape cannot drift
155+
/// between them. A consumer takes the high-water mark per session, so the
156+
/// mid-turn payloads are supersets of each other and the final one wins; a
157+
/// divergence in field names or units between the two call sites would instead
158+
/// show up as tokens silently vanishing, which is the failure this reporting
159+
/// exists to prevent.
160+
///
161+
/// All counts are SESSION-cumulative, matching goose, so buzz-acp's
162+
/// `UsageTracker` can compute per-turn deltas symmetrically for both agents.
163+
pub fn usage_update_payload(
164+
accumulated_input_tokens: u64,
165+
accumulated_output_tokens: u64,
166+
accumulated_cached_input_tokens: u64,
167+
accumulated_total: crate::types::TurnTotalState,
168+
model: &str,
169+
) -> Value {
170+
let mut update = json!({
171+
"sessionUpdate": "usage_update",
172+
// used: total tokens as a context-usage proxy;
173+
// contextLimit: 0 (buzz-agent has no context limit tracking).
174+
"used": accumulated_input_tokens.saturating_add(accumulated_output_tokens),
175+
"contextLimit": 0u64,
176+
"accumulatedInputTokens": accumulated_input_tokens,
177+
"accumulatedOutputTokens": accumulated_output_tokens,
178+
// A subset of accumulatedInputTokens, not an addition to it. Extends
179+
// goose's usage_update shape; a consumer that does not know the field
180+
// ignores it and prices exactly as it did before.
181+
"accumulatedCachedInputTokens": accumulated_cached_input_tokens,
182+
"model": model,
183+
});
184+
// Only when the cumulative is exactly known — never when Unseen (no total
185+
// ever observed) or Unknown (at least one turn lacked a total). A goose
186+
// consumer that doesn't recognise the field ignores it.
187+
if let Some(total) = accumulated_total.exact_value() {
188+
update["accumulatedTotalTokens"] = json!(total);
189+
}
190+
update
191+
}
192+
151193
/// A `session/update` notification carrying a `update._meta.goose.<key>` field.
152194
/// Used to advertise `activeRunId` (so steer-capable clients can target the
153195
/// in-flight run) and `queuedSteer` (so they can correlate an accepted steer

‎crates/buzz-agent/tests/fake_llm.rs‎

Lines changed: 129 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -933,6 +933,135 @@ async fn no_usage_turn_emits_no_usage_notification() {
933933
h.shutdown().await;
934934
}
935935

936+
/// Usage must be reported after EVERY provider round, not only once the turn
937+
/// returns.
938+
///
939+
/// A turn is many provider round-trips over many minutes. While the only report
940+
/// was the one `session/prompt` sends after the turn returns, a turn whose
941+
/// process was killed mid-flight reported nothing at all: its counters lived in
942+
/// the prompt task's stack frame, the provider had already billed them, and no
943+
/// consumer ever saw them. That is not a corner case for a long-horizon
944+
/// benchmark — every phase of a `continue_until_timeout` run is terminated
945+
/// mid-turn by design, which under-reported one measured run's cost several-fold.
946+
///
947+
/// Two rounds with distinct usage. The assertion that matters is the FIRST
948+
/// notification: it must carry round 1's counts alone, proving it was sent
949+
/// before round 2 had returned, so a kill between the rounds would still have
950+
/// left round 1 on the wire.
951+
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
952+
async fn usage_is_reported_after_each_round_not_only_at_turn_end() {
953+
let url = spawn_fake_llm(vec![
954+
openai_tool_call_with_usage("call_round1", "fake__noop", json!({}), 15, 6),
955+
openai_text_with_usage("done", 20, 8),
956+
])
957+
.await;
958+
let mut h = Harness::spawn(&url).await;
959+
let sid = init_session(&mut h).await;
960+
961+
let p_id = h
962+
.send(
963+
"session/prompt",
964+
json!({"sessionId": sid, "prompt": [{"type":"text","text":"go"}]}),
965+
)
966+
.await;
967+
968+
let (frames_before, response) = recv_until_with_drain(&mut h, |v| v["id"] == p_id).await;
969+
assert_eq!(
970+
response["result"]["stopReason"], "end_turn",
971+
"turn must complete with end_turn"
972+
);
973+
974+
let usage: Vec<&Value> = frames_before
975+
.iter()
976+
.filter(|v| is_usage_update(v))
977+
.collect();
978+
assert!(
979+
usage.len() >= 2,
980+
"expected a usage_update per round (2 rounds), got {}; frames: {frames_before:#?}",
981+
usage.len()
982+
);
983+
984+
// Round 1 alone — emitted while round 2 was still outstanding.
985+
assert_eq!(
986+
usage[0]["params"]["update"]["accumulatedInputTokens"],
987+
json!(15u64),
988+
"first notification must carry round 1's input tokens only"
989+
);
990+
assert_eq!(
991+
usage[0]["params"]["update"]["accumulatedOutputTokens"],
992+
json!(6u64),
993+
"first notification must carry round 1's output tokens only"
994+
);
995+
996+
// The last one is the turn total and is what a high-water-mark consumer keeps.
997+
let last = usage[usage.len() - 1];
998+
assert_eq!(
999+
last["params"]["update"]["accumulatedInputTokens"],
1000+
json!(35u64),
1001+
"final notification must carry the turn total 15+20=35"
1002+
);
1003+
assert_eq!(
1004+
last["params"]["update"]["accumulatedOutputTokens"],
1005+
json!(14u64),
1006+
"final notification must carry the turn total 6+8=14"
1007+
);
1008+
1009+
h.shutdown().await;
1010+
}
1011+
1012+
/// A mid-turn report must be SESSION-cumulative, not turn-local.
1013+
///
1014+
/// The baseline handed to the run loop is a snapshot taken when the turn began;
1015+
/// if it were dropped, a consumer taking the high-water mark per session would
1016+
/// see turn 2's first round (a small number) arrive after turn 1's total and
1017+
/// discard it, silently losing turn 2 for any turn that never completed.
1018+
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
1019+
async fn mid_turn_usage_includes_earlier_turns() {
1020+
let url = spawn_fake_llm(vec![
1021+
openai_text_with_usage("turn one", 10, 5),
1022+
openai_tool_call_with_usage("call_t2", "fake__noop", json!({}), 20, 8),
1023+
openai_text_with_usage("turn two done", 30, 9),
1024+
])
1025+
.await;
1026+
let mut h = Harness::spawn(&url).await;
1027+
let sid = init_session(&mut h).await;
1028+
1029+
let p1 = h
1030+
.send(
1031+
"session/prompt",
1032+
json!({"sessionId": sid, "prompt": [{"type":"text","text":"turn 1"}]}),
1033+
)
1034+
.await;
1035+
let (_, _) = recv_until_with_drain(&mut h, |v| v["id"] == p1).await;
1036+
1037+
let p2 = h
1038+
.send(
1039+
"session/prompt",
1040+
json!({"sessionId": sid, "prompt": [{"type":"text","text":"turn 2"}]}),
1041+
)
1042+
.await;
1043+
let (frames_before, _) = recv_until_with_drain(&mut h, |v| v["id"] == p2).await;
1044+
1045+
let first = frames_before
1046+
.iter()
1047+
.find(|v| is_usage_update(v))
1048+
.unwrap_or_else(|| {
1049+
panic!("expected a usage_update during turn 2; frames: {frames_before:#?}")
1050+
});
1051+
assert_eq!(
1052+
first["params"]["update"]["accumulatedInputTokens"],
1053+
json!(30u64),
1054+
"turn 2 round 1 must report 10 (turn 1) + 20 (this round), not 20"
1055+
);
1056+
assert_eq!(
1057+
first["params"]["update"]["accumulatedOutputTokens"],
1058+
json!(13u64),
1059+
"turn 2 round 1 must report 5 (turn 1) + 8 (this round), not 8"
1060+
);
1061+
1062+
h.shutdown().await;
1063+
}
1064+
9361065
/// When a turn is cancelled AFTER the provider has already returned a response
9371066
/// (so token counts are observed), buzz-agent must still emit the usage
9381067
/// notification before the cancelled `session/prompt` response.

0 commit comments

Comments
 (0)