Skip to content

Commit d6a1913

Browse files
npub1mn7jgtj4w2pd0g0zeuhxsa6jy6p0rewxz4kujt98my82ahfmp72sxjexk7wpfleger96
andcommitted
fix(nip-am): address Eva+Wren review findings (migration, delta, cost validation, spec)
Finding 1 (blocker): revert 0001_initial_schema.sql to pristine origin/main state; add additive migrations/0004_agent_turn_metric_fts.sql that DROPs and re-ADDs the search_tsv generated column with the kind-44200 exclusion, following the 0002/0003 pattern. Brownfield relays no longer hit VersionMismatch on startup. Update migration.rs test to assert len=4, migrations[3].version=4, 0001 !contains '44200'. Finding 2 (blocker): fix UsageTracker per-turn delta undercount and wrong turnSeq. Separate the committed baseline (advanced only on take()) from the in-flight cumulative (updated on each record()). Delta is now always measured from the end of the previous published turn, not from the last intermediate notification. turnSeq is now per-published-metric, constant within a turn. Rewrite last_update_wins_multiple_updates_same_turn to assert correct semantics: two in-turn updates 1500/2000 against baseline 1000 → delta 1000/150, seq=2; add turn-3 to prove seq increments per publish, not per notification. Finding 3 (small): add AgentTurnMetricPayload::validate() that rejects negative or non-finite (NaN/inf) costUsd in turn or cumulative. Called from encrypt_agent_turn_metric() so bad payloads can never be published. Add ObserverPayloadError::InvalidPayload variant. Seven new unit tests cover negative, NaN, infinite, and valid cases. Finding 4 (small): two NIP-AM.md wording fixes — (a) unauthenticated → 'AUTH required', authenticated wrong-owner → 'restricted:' (matches code); (b) remove claim that 44200 is regular per NIP-01 (NIP-01 tops out at 10000), say 'regular by Buzz convention (alongside 44100/44101)' instead. E2E: 21/21 assertions pass (fresh keypairs, fresh channel, real relay+acp+agent binaries). Token semantics unchanged for the single-notification-per-turn case. Co-authored-by: Will Pfleger <[email protected]> Signed-off-by: Will Pfleger <[email protected]>
1 parent 760d1ec commit d6a1913

8 files changed

Lines changed: 325 additions & 47 deletions

File tree

‎crates/buzz-acp/src/usage.rs‎

Lines changed: 104 additions & 38 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,15 @@
1919
//! 3. **Session restart** (caller supplies a new `session_id` not seen
2020
//! before): treated as case 1 — fresh baseline, no delta for this turn.
2121
//!
22+
//! Goose may emit **multiple** `usage_update` notifications per turn. The
23+
//! tracker handles this correctly: the committed baseline (and `turn_seq`)
24+
//! advance only when `take()` is called (i.e. at publish time), never on
25+
//! individual notifications. Within a turn all notifications measure their
26+
//! delta from the same frozen baseline — the end of the previous published
27+
//! turn — so the final `pending` record always reflects the full
28+
//! previous-published→current-final delta regardless of how many
29+
//! intermediate notifications arrived.
30+
//!
2231
//! The `TurnUsage` produced after each turn is consumed by the
2332
//! `TurnCompletionGuard` in `pool.rs` to publish a kind 44200 relay event.
2433
@@ -82,14 +91,17 @@ pub(crate) struct UsageUpdatePayload {
8291
/// Per-session normalization state: the last cumulative snapshot we saw.
8392
#[derive(Debug, Clone)]
8493
struct SessionState {
85-
/// Monotonically increasing per-session turn counter (1-based, incremented
86-
/// on every recorded update).
87-
turn_seq: u64,
88-
/// Cumulative input tokens at the end of the previous turn.
94+
/// Per-session turn counter for the LAST PUBLISHED metric (1-based).
95+
/// Advanced only when `take()` drains a pending record — not on every
96+
/// `record()` call. This ensures `turnSeq` counts published metrics, not
97+
/// usage-update notifications.
98+
published_seq: u64,
99+
/// Cumulative input tokens at the end of the LAST PUBLISHED turn.
100+
/// Advanced only on publish (i.e. in `take()`), not on every notification.
89101
last_input: u64,
90-
/// Cumulative output tokens at the end of the previous turn.
102+
/// Cumulative output tokens at the end of the LAST PUBLISHED turn.
91103
last_output: u64,
92-
/// Cumulative cost at the end of the previous turn.
104+
/// Cumulative cost at the end of the LAST PUBLISHED turn.
93105
last_cost: Option<f64>,
94106
}
95107

@@ -131,12 +143,16 @@ pub struct TurnUsage {
131143
/// `session/new` setup) will still update the cumulative baseline but will
132144
/// NOT produce a publishable record.
133145
/// 2. **`record(session_id, payload)`** — called for each
134-
/// `_goose/unstable/session/update` notification. Always updates the
135-
/// cumulative baseline; only produces a publishable record when a turn is
136-
/// currently in-flight for the matching session.
146+
/// `_goose/unstable/session/update` notification. When in-flight, updates
147+
/// `pending` with the latest cumulative values and a delta measured from
148+
/// the committed baseline (end of the previous published turn). Multiple
149+
/// notifications per turn are fine — the last one wins and `turn_seq` stays
150+
/// constant within the turn. When not in-flight, advances the committed
151+
/// baseline so the next turn can compute a correct delta.
137152
/// 3. **`take()`** — called at turn completion by `TurnCompletionGuard`.
138153
/// Drains and returns the pending record (or `None` if no usage was emitted
139-
/// for this turn) and clears the in-flight marker.
154+
/// for this turn), clears the in-flight marker, and advances the committed
155+
/// baseline so the next `record()` call measures from here.
140156
#[derive(Debug, Default)]
141157
pub(crate) struct UsageTracker {
142158
/// One entry per goose `sessionId` ever seen in this process.
@@ -172,21 +188,33 @@ impl UsageTracker {
172188
/// `None` or refers to a different session, the baseline is updated but
173189
/// `pending` is left unchanged.
174190
///
175-
/// When multiple notifications arrive during the same turn, the last one
176-
/// wins (goose may emit several per turn; each increments `turn_seq`).
191+
/// When multiple notifications arrive during the same turn, the **last one
192+
/// wins** on the cumulative totals, and the delta is always measured from
193+
/// the baseline at the end of the **previous published turn** — not from an
194+
/// intermediate notification within the current turn. `turn_seq` stays
195+
/// constant across all notifications within one turn and only increments
196+
/// when a record is actually published (i.e. when `take()` is called).
177197
pub(crate) fn record(&mut self, session_id: &str, payload: &UsageUpdatePayload) {
178198
let current_input = payload.accumulated_input_tokens;
179199
let current_output = payload.accumulated_output_tokens;
180200
let current_cost = payload.accumulated_cost;
181201

202+
// Determine whether this session is currently in-flight so we know
203+
// whether to set `pending`. We compute the delta regardless so that
204+
// setup notifications (no in-flight turn) still advance the baseline.
205+
let is_in_flight = self.in_flight_session.as_deref() == Some(session_id);
206+
182207
let (delta_reliable, turn_input, turn_output, turn_cost, turn_seq) =
183208
match self.sessions.get(session_id) {
184209
None => {
185210
// First notification for this session — no baseline yet.
186211
(false, None, None, None, 1u64)
187212
}
188213
Some(prev) => {
189-
let seq = prev.turn_seq + 1;
214+
// turn_seq for this pending record is one above the last
215+
// *published* seq — constant for all notifications in this
216+
// turn, advanced only on publish.
217+
let seq = prev.published_seq + 1;
190218
// Token counter decrease → unreliable delta.
191219
if current_input < prev.last_input || current_output < prev.last_output {
192220
(false, None, None, None, seq)
@@ -214,20 +242,9 @@ impl UsageTracker {
214242
}
215243
};
216244

217-
// Always advance the session baseline so the next in-flight turn can
218-
// compute a correct delta even if this notification is from setup.
219-
self.sessions.insert(
220-
session_id.to_string(),
221-
SessionState {
222-
turn_seq,
223-
last_input: current_input,
224-
last_output: current_output,
225-
last_cost: current_cost,
226-
},
227-
);
228-
229-
// Only publish a pending record if this session is currently in-flight.
230-
if self.in_flight_session.as_deref() == Some(session_id) {
245+
if is_in_flight {
246+
// Update the pending record with the latest cumulative values.
247+
// Baseline is NOT advanced here — it advances only on take().
231248
self.pending = Some(TurnUsage {
232249
session_id: session_id.to_string(),
233250
turn_seq,
@@ -239,19 +256,48 @@ impl UsageTracker {
239256
cumulative_output_tokens: current_output,
240257
cumulative_cost_usd: current_cost,
241258
});
259+
} else {
260+
// Not in-flight: advance the committed baseline so the next
261+
// in-flight turn computes its delta from this notification.
262+
// This handles setup notifications that fire during `session/new`
263+
// before the first `begin_turn`.
264+
self.sessions.insert(
265+
session_id.to_string(),
266+
SessionState {
267+
published_seq: match self.sessions.get(session_id) {
268+
Some(s) => s.published_seq,
269+
None => 0,
270+
},
271+
last_input: current_input,
272+
last_output: current_output,
273+
last_cost: current_cost,
274+
},
275+
);
242276
}
243277
}
244278

245279
/// Consume and return the most recently computed turn usage record, then
246-
/// clear the in-flight marker.
280+
/// clear the in-flight marker and advance the committed baseline.
247281
///
248282
/// Returns `None` if no `usage_update` arrived during the current in-flight
249283
/// turn (the agent did not emit usage, or no `begin_turn` was called). The
250284
/// caller (`TurnCompletionGuard`) must handle `None`.
251285
#[cfg_attr(not(test), allow(dead_code))]
252286
pub(crate) fn take(&mut self) -> Option<TurnUsage> {
253287
self.in_flight_session = None;
254-
self.pending.take()
288+
let record = self.pending.take()?;
289+
// Advance the committed baseline to this published record so the
290+
// *next* turn measures its delta from here.
291+
self.sessions.insert(
292+
record.session_id.clone(),
293+
SessionState {
294+
published_seq: record.turn_seq,
295+
last_input: record.cumulative_input_tokens,
296+
last_output: record.cumulative_output_tokens,
297+
last_cost: record.cumulative_cost_usd,
298+
},
299+
);
300+
Some(record)
255301
}
256302
}
257303

@@ -497,25 +543,45 @@ mod tests {
497543

498544
#[test]
499545
fn last_update_wins_multiple_updates_same_turn() {
546+
// Goose emits multiple usage_update notifications per turn. The tracker
547+
// must:
548+
// (a) use the LAST notification's cumulative values,
549+
// (b) measure the delta from the baseline at the END OF THE PREVIOUS
550+
// PUBLISHED TURN (not from intermediate notifications), and
551+
// (c) keep turn_seq constant across all notifications within the turn
552+
// (incrementing only on publish, not on each notification).
500553
let mut tracker = UsageTracker::default();
501-
// Turn 1 — baseline.
554+
// Turn 1 — establish baseline. After take(), committed baseline = 1000/100.
502555
tracker.begin_turn("sess-5");
503556
tracker.record("sess-5", &payload(1000, 100, None));
504-
let _ = tracker.take();
557+
let t1 = tracker.take().expect("turn 1");
558+
assert_eq!(t1.turn_seq, 1);
505559

506-
// Two updates arrive before take() — each advances state independently;
507-
// the second delta is computed from the first update's snapshot.
560+
// Turn 2 — two notifications arrive before take(). The second overwrites
561+
// the first in pending; delta is measured from the committed baseline
562+
// (1000/100), not from the intermediate snapshot (1500/150).
508563
tracker.begin_turn("sess-5");
509564
tracker.record("sess-5", &payload(1500, 150, None));
510565
tracker.record("sess-5", &payload(2000, 250, None));
511-
let usage = tracker.take().expect("pending");
566+
let usage = tracker.take().expect("turn 2");
512567

513-
// Cumulative from the last update.
568+
// Cumulative from the last notification.
514569
assert_eq!(usage.cumulative_input_tokens, 2000);
515570
assert_eq!(usage.cumulative_output_tokens, 250);
516-
// Delta is from the previous intermediate snapshot (1500, 150) → (2000, 250).
517-
assert_eq!(usage.turn_input_tokens, Some(500));
518-
assert_eq!(usage.turn_output_tokens, Some(100));
571+
// Delta is from committed baseline (1000, 100) → (2000, 250) = 1000/150.
572+
assert_eq!(usage.turn_input_tokens, Some(1000));
573+
assert_eq!(usage.turn_output_tokens, Some(150));
574+
// seq increments once per publish, not once per notification.
575+
assert_eq!(usage.turn_seq, 2);
576+
577+
// Turn 3 — prove seq continues to increment per publish, not per notification.
578+
tracker.begin_turn("sess-5");
579+
tracker.record("sess-5", &payload(2300, 290, None));
580+
let t3 = tracker.take().expect("turn 3");
581+
assert_eq!(t3.turn_seq, 3);
582+
// Delta from turn-2 committed baseline (2000, 250).
583+
assert_eq!(t3.turn_input_tokens, Some(300));
584+
assert_eq!(t3.turn_output_tokens, Some(40));
519585
}
520586

521587
// ── Wire deserialization ────────────────────────────────────────────────

0 commit comments

Comments
 (0)