Skip to content

Commit c6fe543

Browse files
committed
fix(claude-agent-sdk): promote only new spend; tokens from modelUsage
R1: total_cost_usd is a running total that a resumed, continued or forked session carries forward (sdk.d.ts:5679), and the collector promotes gen_ai.cost.total / gen_ai.usage.* on any span and Observe sums them per trace and per session.id. The wrapper copied the cumulative value, so each resume re-counted earlier spend (real CLI: 0.00018 then 0.00036 promoted for 0.00036 spent). - A process-local LRU (1000 sessions) maps session id -> last running totals. The promoted keys carry the cumulative value minus that baseline: zero for a new session, the session's own totals for resume/continue, the parent's for a fork. A drop within one query (/clear) counts both sides. - No known baseline (resume after restart, continue+forkSession, or a first result below the saved totals): no promoted key and no claude_agent.cost.total_usd; claude_agent.usage.baseline_unknown=true. - Running totals always go on unpromoted claude_agent.cumulative.* keys. - options.continue is now labelled session.is_resumed=true / is_new=false. - A result with nothing usable, or a zeroed crash result, writes no usage key and leaves the baseline alone. R2: tokens came from result.usage, which is main-loop only and per turn in streaming input (sdk.d.ts:5683). They now come from the latest result's modelUsage summed over models (input, output, cache read, cache creation), with the same delta handling. Streaming input stays one conversation span per query() (recorded in the README; Python's ClaudeSDKClient makes one per user turn). Tests: usage.test.ts (21 jest: streaming totals, multi-model modelUsage, no usage on turn/tool/subagent spans, absent usage, resume/continue/fork deltas, unknown baselines, /clear, crash result, LRU). Real-CLI contract scenarios resume, continue_fork, restart (two processes, shared HOME) and streaming assert the promoted keys sum to the final cumulative totals; all four failed before the fix. Fixture results now carry modelUsage. approved: Nikhil 2026-10-03 blanket Refs: TH-8235
1 parent 8b1ab15 commit c6fe543

8 files changed

Lines changed: 833 additions & 33 deletions

File tree

‎typescript/packages/traceai_claude_agent_sdk/README.md‎

Lines changed: 14 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -75,16 +75,27 @@ A foreground subagent span ends with its `tool.Agent` / `tool.Task` result. A su
7575
|---|---|---|
7676
| Model | `gen_ai.request.model`, `claude_agent.model` | `options.model`, the init message, each assistant message |
7777
| Provider | `gen_ai.provider.name` = `anthropic`, or `custom` when `ANTHROPIC_BASE_URL` (from `options.env`, else `process.env`) is not an `anthropic.com` host | |
78-
| Tokens | `gen_ai.usage.input_tokens`, `output_tokens`, `total_tokens`; `gen_ai.usage.cache_read_tokens` / `cache_creation_tokens` when present | result message, on the conversation span |
79-
| Cost | `claude_agent.cost.total_usd` and `gen_ai.cost.total` | result `total_cost_usd`; never computed here |
80-
| Session | `session.id`, `claude_agent.session.id`, `claude_agent.session_id`; `claude_agent.session.fork_from` on fork | init / result `session_id`, `options.sessionId`, `options.resume` |
78+
| Tokens | `gen_ai.usage.input_tokens`, `output_tokens`, `total_tokens`, `cache_read_tokens`, `cache_creation_tokens` | result `modelUsage` summed over every model, on the conversation span only. See "Usage and cost" |
79+
| Cost | `claude_agent.cost.total_usd` and `gen_ai.cost.total` | result `total_cost_usd`; never computed here. See "Usage and cost" |
80+
| Session | `session.id`, `claude_agent.session.id`, `claude_agent.session_id`; `claude_agent.session.is_new` / `is_resumed` (`resume` or `continue`); `claude_agent.session.fork_from` on fork | init / result `session_id`, `options.sessionId`, `options.resume`, `options.continue` |
8181
| Tools | `claude_agent.tool.name`, `gen_ai.tool.name`, `use_id`, `source` (`builtin` / `mcp` / `custom`), `is_error`, `duration_ms` | `tool_use` / `tool_result` blocks |
8282
| Errors | span status ERROR; `claude_agent.error.type` / `error.message`; `claude_agent.cancelled=true` on abort | error result, thrown error, `is_error` tool results, `AbortController` |
8383

8484
The `claude_agent.*` names are the Python package's names (`_attributes.py`); a test fails if one is missing.
8585

8686
An app-set fi-core `session.id` / `user.id` (fi-core context helpers) is kept; the SDK session id still goes to `claude_agent.session.id`.
8787

88+
## Usage and cost
89+
90+
The Future AGI collector promotes `gen_ai.usage.*` and `gen_ai.cost.total` on any span, and Observe sums them over a trace and over a `session.id`. So these keys are written once, on the conversation span, and hold only the spend that is new in this `query()` call. Turn, tool and subagent spans carry none of them.
91+
92+
- Tokens come from the latest result's `modelUsage`, summed over every model (main loop, subagents, compaction). Cost comes from `total_cost_usd`. Both are running totals for the session (`sdk.d.ts:5679`, `5687`). `result.usage` is not read: the SDK documents it as main-loop only and per turn in streaming-input mode (`sdk.d.ts:5683`). This replaces the spec's "result message usage" mapping.
93+
- Streaming input (one `query()`, several user turns) yields one result per turn. The conversation span keeps the latest running totals, so tokens and cost cover every turn. It stays one conversation span per `query()`; the Python `ClaudeSDKClient` path makes one per user turn.
94+
- A resumed (`resume`), continued (`continue`) or forked (`forkSession`) session's first result already carries the earlier turns. The wrapper keeps an in-process map of session id to the last totals it saw (at most 1000 sessions) and writes only the difference. A fork starts from its parent's totals. A drop in the running total within one query (a `/clear`) counts the spend on both sides of it.
95+
- When there is no baseline in this process (resume after a restart, `continue` + `forkSession`, or a first result below the saved totals), no `gen_ai.usage.*` / `gen_ai.cost.total` / `claude_agent.cost.total_usd` is written and `claude_agent.usage.baseline_unknown=true`.
96+
- Every conversation span with a result also carries the running totals on unpromoted keys: `claude_agent.cumulative.cost_usd`, `input_tokens`, `output_tokens`, `cache_read_tokens`, `cache_creation_tokens`.
97+
- A result with no cost and no `modelUsage`, or a zeroed crash result, writes none of these keys and leaves the session's saved totals unchanged.
98+
8899
## Privacy: content is off by default
89100

90101
Prompts, system prompts, tool inputs, tool outputs, assistant text and the final result can contain source code, file contents or secrets. They are **not** recorded unless you opt in:

‎typescript/packages/traceai_claude_agent_sdk/contract/run_real_sdk.mjs‎

Lines changed: 68 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -15,6 +15,14 @@
1515
// tool one query() that runs the Read tool.
1616
// close two query() calls stopped mid-tool: one with close(), one
1717
// with Symbol.asyncDispose.
18+
// resume two query() calls; the second resumes the first's session.
19+
// continue_fork three query() calls: a new session, options.continue,
20+
// then resume + forkSession from the first session id.
21+
// restart one query() per process, sharing WORKDIR (HOME): PHASE=first
22+
// starts a session, PHASE=second resumes RESUME_SESSION_ID.
23+
// streaming one query() with an AsyncIterable prompt of two user turns;
24+
// the second turn is sent after the first result.
25+
// WORKDIR (optional): use and keep this directory instead of a temp dir.
1826
// Prints {"requests": [...], "messages": [...], "queries": [[...], ...]} on stdout.
1927
import http from "node:http";
2028
import fs from "node:fs";
@@ -32,7 +40,9 @@ const PROMPT = "Read README.md and summarize it. SECRET_PROMPT_MARKER";
3240
const SCENARIO = process.env.SCENARIO || "tool";
3341
const TOOL_FLOW = SCENARIO === "tool" || SCENARIO === "close";
3442

35-
const workdir = fs.mkdtempSync(path.join(os.tmpdir(), "th8235-real-sdk-"));
43+
const keepWorkdir = Boolean(process.env.WORKDIR);
44+
const workdir = keepWorkdir ? process.env.WORKDIR : fs.mkdtempSync(path.join(os.tmpdir(), "th8235-real-sdk-"));
45+
fs.mkdirSync(workdir, { recursive: true });
3646
const readmePath = path.join(workdir, "README.md");
3747
fs.writeFileSync(readmePath, "# Demo\nSECRET_TOOL_OUTPUT_MARKER\n");
3848

@@ -161,8 +171,64 @@ const SCENARIOS = {
161171
await disposed[Symbol.asyncDispose]();
162172
return [first, second];
163173
},
174+
async resume() {
175+
const first = await drain(tracedQuery({ prompt: "First question.", options: options({ persistSession: true }) }));
176+
const second = await drain(
177+
tracedQuery({ prompt: "Second question.", options: options({ persistSession: true, resume: sessionOf(first) }) }),
178+
);
179+
return [first, second];
180+
},
181+
async continue_fork() {
182+
const first = await drain(tracedQuery({ prompt: "First question.", options: options({ persistSession: true }) }));
183+
const second = await drain(
184+
tracedQuery({ prompt: "Second question.", options: options({ persistSession: true, continue: true }) }),
185+
);
186+
const third = await drain(
187+
tracedQuery({
188+
prompt: "Third question.",
189+
options: options({ persistSession: true, resume: sessionOf(first), forkSession: true }),
190+
}),
191+
);
192+
return [first, second, third];
193+
},
194+
async restart() {
195+
const resume = process.env.PHASE === "second" ? process.env.RESUME_SESSION_ID : undefined;
196+
if (process.env.PHASE === "second" && !resume) throw new Error("PHASE=second needs RESUME_SESSION_ID");
197+
return [
198+
await drain(
199+
tracedQuery({
200+
prompt: resume ? "Second question." : "First question.",
201+
options: options({ persistSession: true, ...(resume ? { resume } : {}) }),
202+
}),
203+
),
204+
];
205+
},
206+
async streaming() {
207+
let resultSeen;
208+
let waitForResult = new Promise((resolve) => (resultSeen = resolve));
209+
const userTurn = (content) => ({ type: "user", message: { role: "user", content }, parent_tool_use_id: null });
210+
async function* prompt() {
211+
yield userTurn("First question.");
212+
await waitForResult;
213+
waitForResult = new Promise((resolve) => (resultSeen = resolve));
214+
yield userTurn("Second question.");
215+
await waitForResult;
216+
}
217+
const out = [];
218+
for await (const message of tracedQuery({ prompt: prompt(), options: options() })) {
219+
out.push(message);
220+
if (message.type === "result") resultSeen();
221+
}
222+
return [out];
223+
},
164224
};
165225

226+
function sessionOf(messages) {
227+
const init = messages.find((m) => m.type === "system" && m.subtype === "init");
228+
if (!init) throw new Error("no init message");
229+
return init.session_id;
230+
}
231+
166232
const queries = [];
167233
try {
168234
const scenario = SCENARIOS[SCENARIO];
@@ -173,7 +239,7 @@ try {
173239
await shutdown(provider);
174240
await provider.shutdown();
175241
server.close();
176-
fs.rmSync(workdir, { recursive: true, force: true });
242+
if (!keepWorkdir) fs.rmSync(workdir, { recursive: true, force: true });
177243
}
178244

179245
process.stdout.write(JSON.stringify({ mockOrigin, requests, messages: queries.flat(), queries }));

‎typescript/packages/traceai_claude_agent_sdk/contract/test_harness_contract.py‎

Lines changed: 156 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -322,6 +322,162 @@ def test_real_sdk_close_and_async_dispose_end_every_span_cancelled(built_package
322322
_assert_exported_with_both_keys(exported, "th8235-real-sdk-close")
323323

324324

325+
# Keys fi-collector promotes into hot columns on ANY span; Observe sums them per
326+
# trace and per session.id, so they must equal new spend exactly once.
327+
PROMOTED_PREFIXES = ("gen_ai.usage.", "llm.token_count.")
328+
PROMOTED_KEYS = ("gen_ai.cost.total", "llm.cost.total")
329+
330+
331+
def _promoted(attributes: Dict[str, Any]) -> Dict[str, Any]:
332+
return {k: v for k, v in attributes.items() if k.startswith(PROMOTED_PREFIXES) or k in PROMOTED_KEYS}
333+
334+
335+
def _results(messages: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
336+
return [m for m in messages if m["type"] == "result"]
337+
338+
339+
def _init_session(messages: List[Dict[str, Any]]) -> str:
340+
return [m for m in messages if m["type"] == "system" and m.get("subtype") == "init"][0]["session_id"]
341+
342+
343+
def _cumulative(result: Dict[str, Any]) -> Dict[str, Any]:
344+
"""The running totals a result carries: total_cost_usd and modelUsage summed over models."""
345+
usage = result["modelUsage"].values()
346+
return {
347+
"cost": result["total_cost_usd"],
348+
"input": sum(u["inputTokens"] for u in usage),
349+
"output": sum(u["outputTokens"] for u in usage),
350+
}
351+
352+
353+
def _conversations_in_query_order(spans: List[Dict[str, Any]]) -> List[Dict[str, Any]]:
354+
conversations = _by_name(spans, "claude_agent.conversation")
355+
return sorted(conversations, key=lambda span: int(span["startTimeUnixNano"]))
356+
357+
358+
def _assert_only_conversations_carry_usage(spans: List[Dict[str, Any]]) -> None:
359+
for span in spans:
360+
if span["name"] != "claude_agent.conversation":
361+
assert not _promoted(_attrs(span)), (span["name"], _promoted(_attrs(span)))
362+
363+
364+
def _assert_promoted_sum_equals(conversations: List[Dict[str, Any]], final: Dict[str, Any]) -> None:
365+
attrs = [_attrs(c) for c in conversations]
366+
assert sum(a["gen_ai.cost.total"] for a in attrs) == pytest.approx(final["cost"], rel=1e-9)
367+
assert sum(a["gen_ai.usage.input_tokens"] for a in attrs) == final["input"]
368+
assert sum(a["gen_ai.usage.output_tokens"] for a in attrs) == final["output"]
369+
assert sum(a["gen_ai.usage.total_tokens"] for a in attrs) == final["input"] + final["output"]
370+
371+
372+
@requires_real_sdk
373+
def test_real_sdk_resume_counts_cost_and_tokens_once(built_package: Path) -> None:
374+
"""R1: a resumed session's first result already carries the earlier turns (sdk.d.ts:5679).
375+
376+
Two query() calls, the second with options.resume. The promoted cost/tokens summed
377+
over both traces must equal the final cumulative totals, not exceed them.
378+
"""
379+
output, spans, exported = _run_real_sdk("resume", "th8235-real-sdk-resume")
380+
first, second = output["queries"]
381+
(r1,), (r2,) = _results(first), _results(second)
382+
session_id = _init_session(first)
383+
assert _init_session(second) == session_id
384+
final = _cumulative(r2)
385+
# What the reviewer saw: the resumed result is cumulative over both queries.
386+
assert final["cost"] > _cumulative(r1)["cost"] > 0 and final["input"] > _cumulative(r1)["input"]
387+
388+
conversations = _conversations_in_query_order(spans)
389+
assert len(conversations) == 2 and len({c["traceId"] for c in conversations}) == 2
390+
_assert_promoted_sum_equals(conversations, final)
391+
c1, c2 = (_attrs(c) for c in conversations)
392+
assert c1["gen_ai.cost.total"] == pytest.approx(r1["total_cost_usd"])
393+
assert (c2["session.id"], c2["claude_agent.session.is_new"], c2["claude_agent.session.is_resumed"]) == (
394+
session_id, False, True,
395+
)
396+
assert c2["claude_agent.cumulative.cost_usd"] == pytest.approx(final["cost"])
397+
assert (c2["claude_agent.cumulative.input_tokens"], c2["claude_agent.cumulative.output_tokens"]) == (
398+
final["input"], final["output"],
399+
)
400+
assert c2["claude_agent.usage.baseline_unknown"] is False
401+
_assert_only_conversations_carry_usage(spans)
402+
_assert_exported_with_both_keys(exported, "th8235-real-sdk-resume")
403+
404+
405+
@requires_real_sdk
406+
def test_real_sdk_continue_and_fork_count_cost_and_tokens_once(built_package: Path) -> None:
407+
"""R1: options.continue keeps the session id and its totals; a fork gets a new id but
408+
starts from the parent's saved totals. Sum of promoted keys == the fork's final totals."""
409+
output, spans, _ = _run_real_sdk("continue_fork", "th8235-real-sdk-continue-fork")
410+
first, continued, forked = output["queries"]
411+
parent = _init_session(first)
412+
assert _init_session(continued) == parent
413+
fork_id = _init_session(forked)
414+
assert fork_id != parent
415+
final = _cumulative(_results(forked)[-1])
416+
417+
conversations = _conversations_in_query_order(spans)
418+
assert len(conversations) == 3
419+
_assert_promoted_sum_equals(conversations, final)
420+
_, c_continue, c_fork = (_attrs(c) for c in conversations)
421+
assert (c_continue["claude_agent.session.is_new"], c_continue["claude_agent.session.is_resumed"]) == (False, True)
422+
assert c_continue["claude_agent.is_resumed"] is True
423+
assert (c_fork["session.id"], c_fork["claude_agent.session.fork_from"]) == (fork_id, parent)
424+
assert c_fork["claude_agent.session.is_new"] is True
425+
_assert_only_conversations_carry_usage(spans)
426+
427+
428+
@requires_real_sdk
429+
def test_real_sdk_resume_after_process_restart_marks_baseline_unknown(built_package: Path, tmp_path: Path) -> None:
430+
"""R1: a resume with no baseline in this process (restart) must not put promoted keys:
431+
the result is cumulative and the earlier share is unknown. Cumulative values go on
432+
claude_agent.cumulative.* and claude_agent.usage.baseline_unknown=true."""
433+
workdir = tmp_path / "home"
434+
first_out, first_spans, _ = _run_real_sdk("restart", "th8235-real-sdk-restart", {"WORKDIR": str(workdir), "PHASE": "first"})
435+
session_id = _init_session(first_out["queries"][0])
436+
second_out, second_spans, _ = _run_real_sdk(
437+
"restart", "th8235-real-sdk-restart",
438+
{"WORKDIR": str(workdir), "PHASE": "second", "RESUME_SESSION_ID": session_id},
439+
)
440+
(r1,), (r2,) = _results(first_out["queries"][0]), _results(second_out["queries"][0])
441+
final = _cumulative(r2)
442+
assert final["cost"] > r1["total_cost_usd"]
443+
444+
c1 = _attrs(_one(first_spans, "claude_agent.conversation"))
445+
c2 = _attrs(_one(second_spans, "claude_agent.conversation"))
446+
assert c1["gen_ai.cost.total"] == pytest.approx(r1["total_cost_usd"]) # a new session: baseline 0
447+
assert c1["claude_agent.usage.baseline_unknown"] is False
448+
assert _promoted(c2) == {}, _promoted(c2)
449+
assert "claude_agent.cost.total_usd" not in c2
450+
assert c2["claude_agent.usage.baseline_unknown"] is True
451+
assert c2["claude_agent.cumulative.cost_usd"] == pytest.approx(final["cost"])
452+
assert (c2["claude_agent.cumulative.input_tokens"], c2["claude_agent.cumulative.output_tokens"]) == (
453+
final["input"], final["output"],
454+
)
455+
assert c2["session.id"] == session_id and c2["claude_agent.session.is_new"] is False
456+
_assert_only_conversations_carry_usage(first_spans + second_spans)
457+
458+
459+
@requires_real_sdk
460+
def test_real_sdk_streaming_input_counts_tokens_once(built_package: Path) -> None:
461+
"""R2 + R5: one query() with two user turns yields two results. result.usage is per turn
462+
(sdk.d.ts:5683); modelUsage and total_cost_usd are running totals (sdk.d.ts:5679, 5687).
463+
The conversation span must carry the latest running totals, and turn 2 must start
464+
after turn 1 ended."""
465+
output, spans, _ = _run_real_sdk("streaming", "th8235-real-sdk-streaming")
466+
(messages,) = output["queries"]
467+
results = _results(messages)
468+
assert len(results) == 2
469+
final = _cumulative(results[-1])
470+
# What the reviewer saw: usage on the last result is one turn only.
471+
assert results[-1]["usage"]["input_tokens"] < final["input"]
472+
473+
conversation = _one(spans, "claude_agent.conversation")
474+
_assert_promoted_sum_equals([conversation], final)
475+
turns = sorted(_by_name(spans, "claude_agent.assistant_turn"), key=lambda s: int(s["startTimeUnixNano"]))
476+
assert len(turns) == 2
477+
assert int(turns[1]["startTimeUnixNano"]) >= int(turns[0]["endTimeUnixNano"])
478+
_assert_only_conversations_carry_usage(spans)
479+
480+
325481
@requires_real_sdk
326482
def test_real_sdk_query_through_anthropic_base_url_mock(built_package: Path) -> None:
327483
"""The real SDK query() and bundled CLI, wrapped, against a loopback Messages API mock.

‎typescript/packages/traceai_claude_agent_sdk/src/__tests__/fixtures/messages.ts‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -155,7 +155,8 @@ export function resultSuccess(resultText: string, overrides: Partial<SDKResultSu
155155
stop_reason: "end_turn",
156156
total_cost_usd: TOTAL_COST_USD,
157157
usage: resultUsage,
158-
modelUsage: {},
158+
// modelUsage is the running total the wrapper reads (sdk.d.ts:5687); usage is main-loop only.
159+
modelUsage: { [MODEL]: modelUsage(120, 45, TOTAL_COST_USD, 11, 7) },
159160
permission_denials: [],
160161
uuid: uuid(),
161162
session_id: SESSION_ID,
@@ -174,7 +175,7 @@ export function resultError(overrides: Partial<SDKResultError> = {}): SDKResultE
174175
stop_reason: null,
175176
total_cost_usd: 0.002,
176177
usage: resultUsage,
177-
modelUsage: {},
178+
modelUsage: { [MODEL]: modelUsage(120, 45, 0.002, 11, 7) },
178179
permission_denials: [],
179180
errors: ["Reached maximum number of turns (3)"],
180181
uuid: uuid(),

0 commit comments

Comments
 (0)