Skip to content

Commit a06e80f

Browse files
byapparovclaude
andcommitted
fix(run): report subagent token usage as subagent_message_complete (#213)
A run that delegated through the `task` tool reported only the parent's turns: child-session assistant messages are excluded from message_complete by contract (EVENTS.md, #93), and nothing else carried their usage, so consumers undercounted every delegating run. Emit an additive `subagent_message_complete` event for each finished assistant turn of a tracked child session, with the same payload as message_complete plus `subagentSessionID`. The payload construction is extracted into completedMessage() and shared by both events, so child turns get identical usage provenance and context-window handling. message_complete keeps its primary-session meaning. EVENTS.md explains how to total a run: sum message_complete and subagent_message_complete. Co-Authored-By: Claude Opus 5.5 <[email protected]> Claude-Session: https://claude.ai/code/session_015nwvfUHdM3bgMgD9Zpz7Mj
1 parent 7e16685 commit a06e80f

5 files changed

Lines changed: 237 additions & 57 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@
1111
- **Messages written in the same millisecond keep their order** — Reading a session's messages now breaks `time_created` ties by message ID. Before, messages written in the same millisecond came back oldest first. A compaction writes its request, its summary and the continue message in quick succession, so the prompt loop could miss the finished summary and compact again, sending an extra model request. (#200)
1212
- **Retry waits are bounded for every status** — No single wait between retries exceeds 5 minutes. A retryable error of any status whose `Retry-After` / `retry-after-ms` asks for longer is no longer retried: a 429 still fails as `usage_limit` (#174), and any other status (for example a 503 or 529 asking for `retry-after: 18000`, which used to sleep about 5 hours per attempt) fails at once with the provider error, `session_error.reason` from its status (`provider` for a 5xx) and exit 1, with no `retry` event. Exponential backoff without a retry hint now stops at 30 s even when the response has headers, as every AI SDK HTTP error does, so a hintless 429 or 5xx retries for about 3.5 minutes in all instead of about 34. A zero or negative `Retry-After` now counts as no hint and backs off, instead of retrying with no wait. With 10 retries the waits total at most 50 minutes. (closes #175)
1313
- **`session_start.permissions` is complete** — it now lists the full ruleset tool calls are evaluated against (agent defaults, config `permission` rules, agent rules and session rules) in evaluation order; it was empty for the default agent and omitted session rules (#212).
14+
- **Subagent token usage is reported** — `aictrl run --format json` emits a new `subagent_message_complete` event for each finished assistant turn of a `task` subagent, with the same usage fields as `message_complete` plus `subagentSessionID`. Summing `message_complete` and `subagent_message_complete` gives the run's total; before, a run that delegated reported only the parent's turns. `message_complete` is unchanged and still covers the primary session only (#213).
1415

1516
### Internal
1617

‎EVENTS.md‎

Lines changed: 28 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -255,7 +255,8 @@ The StructuredOutput corrective reminder is not a stored user message. It is add
255255
### `message_complete`
256256

257257
Emitted when a primary-session assistant message finishes (one per LLM turn). Child-session assistant messages are
258-
excluded; use the child-session lifecycle events when tracking subagents.
258+
excluded; they are reported as [`subagent_message_complete`](#subagent_message_complete). To total a run, see
259+
[Totalling a run](#totalling-a-run).
259260

260261
```json
261262
{
@@ -535,6 +536,32 @@ Emitted when a child session completes.
535536
}
536537
```
537538

539+
### `subagent_message_complete`
540+
541+
Emitted when an assistant message of a child session (a `task` subagent announced by `subagent_start`) finishes, one per child LLM turn. It carries the same fields as `message_complete` (`messageID`, `modelID`, `providerID`, `agent`, `status`, `usageStatus`, `cost`, `tokens`, `context`, `finish`) plus `subagentSessionID`. `sessionID` is the primary session, as on every event. A child may use a different model from the parent, so price each event by its own `providerID`/`modelID`.
542+
543+
```json
544+
{
545+
"type": "subagent_message_complete",
546+
"sessionID": "session_01abc...",
547+
"subagentSessionID": "session_01xyz...",
548+
"messageID": "msg_01def...",
549+
"modelID": "claude-sonnet-4-20250514",
550+
"providerID": "anthropic",
551+
"agent": "general",
552+
"status": "completed",
553+
"usageStatus": "reported",
554+
"cost": 0.004,
555+
"tokens": { "total": 2055, "input": 2000, "output": 55, "reasoning": 0, "cache": { "read": 0, "write": 0 } },
556+
"context": { "used": 2000, "limit": 200000, "ratio": 0.01 },
557+
"finish": "stop"
558+
}
559+
```
560+
561+
### Totalling a run
562+
563+
A run's model usage is the sum of `tokens` (and `cost`) over every `message_complete` (the primary session's turns) and every `subagent_message_complete` (its subagents' turns). Each message appears in exactly one of the two events, at most once, so nothing is double-counted. An event with `usageStatus: "missing"` has `tokens: null`; when any event in the run has it, the sum is a lower bound. Subagents are the primary session's direct children (those announced by `subagent_start`); turns of a subagent's own nested `task` calls are not reported.
564+
538565
## Error Events
539566

540567
### `error`

‎packages/cli/src/cli/cmd/run.ts‎

Lines changed: 49 additions & 39 deletions
Original file line numberDiff line numberDiff line change
@@ -622,6 +622,48 @@ export const RunCommand = cmd({
622622
return false
623623
}
624624

625+
/** The `message_complete` payload for a finished assistant message: status, usage and context window. */
626+
async function completedMessage(info: MessageV2.Assistant) {
627+
const usage = terminalUsage(info)
628+
629+
// Context-window utilization: used = input + cache.read + cache.write
630+
// (all prompt tokens that occupy the model's context window this turn).
631+
// cache.write tokens are written to the cache ON this turn — they are
632+
// part of the prompt sent to the model and count against the context
633+
// window, just billed at the cache-write rate. Excluding them
634+
// undercounts utilization on the first turn of a conversation.
635+
// limit comes from the provider registry (the built-in catalog plus config, or the attached server's).
636+
// On lookup failure (or limit===0 for custom models) buildContextWindow returns null.
637+
const contextLimit = await (args.attach
638+
? attachedContextLimit(info.providerID, info.modelID, attachedProviders())
639+
: Provider.getModel(info.providerID, info.modelID)
640+
.then((m) => m.limit.context)
641+
.catch((e) => {
642+
if (e instanceof Provider.ModelNotFoundError) return null
643+
throw e
644+
}))
645+
const contextUsed = usage.tokens
646+
? usage.tokens.input + usage.tokens.cache.read + usage.tokens.cache.write
647+
: null
648+
const context = contextUsed === null ? null : buildContextWindow(contextLimit, contextUsed)
649+
650+
return {
651+
messageID: info.id,
652+
modelID: info.modelID,
653+
providerID: info.providerID,
654+
agent: info.agent,
655+
context,
656+
// cost is sourced from info.cost which accumulates real per-step costs
657+
// from StepFinishPart. Do NOT use the new step.ended event cost field
658+
// which emits cost:0 and is reconciled later (the cost:0 trap).
659+
cost: info.cost,
660+
tokens: usage.tokens,
661+
usageStatus: usage.usageStatus,
662+
status: info.error?.name === "MessageAbortedError" ? "aborted" : info.error ? "error" : "completed",
663+
finish: info.finish,
664+
}
665+
}
666+
625667
const subscriptionAbort = new AbortController()
626668
const events = await sdk.event.subscribe({ signal: subscriptionAbort.signal })
627669
let error: string | undefined
@@ -742,46 +784,14 @@ export const RunCommand = cmd({
742784
if (event.type === "message.updated" && event.properties.info.role === "assistant") {
743785
const info = event.properties.info
744786
if (args.format === "json") {
745-
if (info.sessionID === sessionID && info.time.completed !== undefined && !emitted.has(info.id)) {
787+
const own = info.sessionID === sessionID
788+
const finished = info.time.completed !== undefined && !emitted.has(info.id)
789+
if (finished && (own || childSessions.has(info.sessionID))) {
746790
emitted.add(info.id)
747-
const usage = terminalUsage(info)
748-
749-
// Context-window utilization: used = input + cache.read + cache.write
750-
// (all prompt tokens that occupy the model's context window this turn).
751-
// cache.write tokens are written to the cache ON this turn — they are
752-
// part of the prompt sent to the model and count against the context
753-
// window, just billed at the cache-write rate. Excluding them
754-
// undercounts utilization on the first turn of a conversation.
755-
// limit comes from the provider registry (the built-in catalog plus config, or the attached server's).
756-
// On lookup failure (or limit===0 for custom models) buildContextWindow returns null.
757-
const contextLimit = await (args.attach
758-
? attachedContextLimit(info.providerID, info.modelID, attachedProviders())
759-
: Provider.getModel(info.providerID, info.modelID)
760-
.then((m) => m.limit.context)
761-
.catch((e) => {
762-
if (e instanceof Provider.ModelNotFoundError) return null
763-
throw e
764-
}))
765-
const contextUsed = usage.tokens
766-
? usage.tokens.input + usage.tokens.cache.read + usage.tokens.cache.write
767-
: null
768-
const context = contextUsed === null ? null : buildContextWindow(contextLimit, contextUsed)
769-
770-
emit("message_complete", {
771-
messageID: info.id,
772-
modelID: info.modelID,
773-
providerID: info.providerID,
774-
agent: info.agent,
775-
context,
776-
// cost is sourced from info.cost which accumulates real per-step costs
777-
// from StepFinishPart. Do NOT use the new step.ended event cost field
778-
// which emits cost:0 and is reconciled later (the cost:0 trap).
779-
cost: info.cost,
780-
tokens: usage.tokens,
781-
usageStatus: usage.usageStatus,
782-
status: info.error?.name === "MessageAbortedError" ? "aborted" : info.error ? "error" : "completed",
783-
finish: info.finish,
784-
})
791+
const message = await completedMessage(info)
792+
// message_complete stays primary-session only; child turns get their own event (#213)
793+
if (own) emit("message_complete", message)
794+
else emit("subagent_message_complete", { subagentSessionID: info.sessionID, ...message })
785795
}
786796
} else if (toggles.get("start") !== true) {
787797
UI.empty()
Lines changed: 144 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,144 @@
1+
import { describe, expect, test } from "bun:test"
2+
import path from "path"
3+
import { tmpdir } from "../fixture/fixture"
4+
5+
const entry = path.resolve(import.meta.dir, "../../src/index.ts")
6+
const command = process.env.AICTRL_TEST_BINARY
7+
? [process.env.AICTRL_TEST_BINARY]
8+
: ["bun", "run", "--conditions=browser", entry]
9+
10+
type Usage = { prompt_tokens: number; completion_tokens: number; total_tokens: number }
11+
type Tokens = {
12+
total: number
13+
input: number
14+
output: number
15+
reasoning: number
16+
cache: { read: number; write: number }
17+
}
18+
type Event = {
19+
type: string
20+
sessionID: string
21+
subagentSessionID?: string
22+
messageID?: string
23+
tokens?: Tokens | null
24+
usageStatus?: string
25+
}
26+
27+
function usage(prompt: number, completion: number): Usage {
28+
return { prompt_tokens: prompt, completion_tokens: completion, total_tokens: prompt + completion }
29+
}
30+
31+
// Model turns in request order: the parent delegates through `task`, the child answers, the parent finishes.
32+
const turns = [
33+
{
34+
delta: {
35+
tool_calls: [
36+
{
37+
index: 0,
38+
id: "call_task",
39+
type: "function",
40+
function: {
41+
name: "task",
42+
arguments: JSON.stringify({ description: "Search", prompt: "Find the file.", subagent_type: "general" }),
43+
},
44+
},
45+
],
46+
},
47+
finish: "tool_calls",
48+
usage: usage(100, 10),
49+
},
50+
{ delta: { content: "Found it." }, finish: "stop", usage: usage(50, 5) },
51+
{ delta: { content: "Done." }, finish: "stop", usage: usage(120, 7) },
52+
]
53+
54+
async function run() {
55+
await using tmp = await tmpdir({ git: true })
56+
const pending = [...turns]
57+
const server = Bun.serve({
58+
hostname: "127.0.0.1",
59+
port: 0,
60+
idleTimeout: 0,
61+
fetch() {
62+
const turn = pending.shift() ?? { delta: { content: "extra" }, finish: "stop", usage: usage(0, 0) }
63+
const chunks = [
64+
{ choices: [{ index: 0, delta: { role: "assistant", ...turn.delta }, finish_reason: null }] },
65+
{ choices: [{ index: 0, delta: {}, finish_reason: turn.finish }], usage: turn.usage },
66+
].map(
67+
(chunk) =>
68+
`data: ${JSON.stringify({ id: "fixture", object: "chat.completion.chunk", created: 1, model: "fixture", ...chunk })}\n\n`,
69+
)
70+
return new Response(chunks.join("") + "data: [DONE]\n\n", { headers: { "content-type": "text/event-stream" } })
71+
},
72+
})
73+
const config = {
74+
provider: {
75+
fixture: {
76+
npm: "@ai-sdk/openai-compatible",
77+
options: { apiKey: "fixture", baseURL: `http://127.0.0.1:${server.port}/v1` },
78+
models: { fixture: { name: "fixture", tool_call: true, limit: { context: 100000, output: 1000 } } },
79+
},
80+
},
81+
agent: { title: { disable: true } },
82+
permission: { "*": "allow" },
83+
}
84+
const home = path.join(tmp.path, "home")
85+
const proc = Bun.spawn([...command, "run", "--format", "json", "--model", "fixture/fixture", "Delegate a search."], {
86+
cwd: tmp.path,
87+
env: {
88+
...process.env,
89+
AICTRL_CONFIG_CONTENT: JSON.stringify(config),
90+
AICTRL_DISABLE_DEFAULT_PLUGINS: "true",
91+
AICTRL_DISABLE_AUTOCOMPACT: "true",
92+
HOME: home,
93+
AICTRL_TEST_HOME: home,
94+
AICTRL_TEST_MANAGED_CONFIG_DIR: path.join(tmp.path, "managed"),
95+
XDG_DATA_HOME: path.join(tmp.path, "data"),
96+
XDG_CONFIG_HOME: path.join(tmp.path, "config"),
97+
XDG_STATE_HOME: path.join(tmp.path, "state"),
98+
XDG_CACHE_HOME: path.join(tmp.path, "cache"),
99+
},
100+
stdout: "pipe",
101+
stderr: "pipe",
102+
})
103+
const timeout = setTimeout(() => proc.kill("SIGKILL"), 20000)
104+
try {
105+
const [stdout, stderr, exit] = await Promise.all([
106+
new Response(proc.stdout).text(),
107+
new Response(proc.stderr).text(),
108+
proc.exited,
109+
])
110+
const events: Event[] = stdout
111+
.split("\n")
112+
.filter((line) => line.startsWith("{"))
113+
.map((line) => JSON.parse(line))
114+
return { exit, output: stderr + stdout, events, unused: pending.length }
115+
} finally {
116+
clearTimeout(timeout)
117+
proc.kill("SIGKILL")
118+
server.stop(true)
119+
}
120+
}
121+
122+
describe("subagent token usage (#213)", () => {
123+
test("reports each child turn so parent + children add up to the provider total", async () => {
124+
const result = await run()
125+
expect(result.exit, result.output).toBe(0)
126+
expect(result.unused, result.output).toBe(0)
127+
128+
const child = result.events.find((event) => event.type === "subagent_start")?.subagentSessionID
129+
expect(child, result.output).toBeDefined()
130+
const parentTurns = result.events.filter((event) => event.type === "message_complete")
131+
const childTurns = result.events.filter((event) => event.type === "subagent_message_complete")
132+
133+
// message_complete keeps its primary-session meaning.
134+
expect(parentTurns).toHaveLength(2)
135+
expect(childTurns).toHaveLength(1)
136+
expect(childTurns[0]).toMatchObject({ subagentSessionID: child, usageStatus: "reported" })
137+
expect(childTurns[0].messageID).not.toBe(parentTurns[0].messageID)
138+
139+
const total = [...parentTurns, ...childTurns].reduce((sum, event) => sum + event.tokens!.total, 0)
140+
const provider = turns.reduce((sum, turn) => sum + turn.usage.total_tokens, 0)
141+
expect(childTurns[0].tokens!.total).toBe(55)
142+
expect(total).toBe(provider)
143+
}, 30_000)
144+
})

‎packages/cli/test/cli/usage-token-breakdown.test.ts‎

Lines changed: 15 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -110,30 +110,28 @@ describe("message_complete emit block shape (source-verified, #86)", () => {
110110
expect(block).toContain("write")
111111
})
112112

113-
test("emit block calls buildContextWindow (not inline ternary)", async () => {
113+
// The payload is built by completedMessage() and emitted as message_complete
114+
// (primary session) or subagent_message_complete (child sessions, #213).
115+
async function payloadBlock() {
114116
const source = await Bun.file(RUN_SRC).text()
115-
const emitIdx = source.indexOf('emit("message_complete"')
116-
expect(emitIdx).toBeGreaterThan(-1)
117-
const blockStart = Math.max(0, emitIdx - 1500)
118-
const block = source.slice(blockStart, emitIdx + 200)
119-
expect(block).toContain("buildContextWindow")
117+
expect(source).toContain('emit("message_complete", message)')
118+
const start = source.indexOf("async function completedMessage(")
119+
const end = source.indexOf("\n }\n", start)
120+
expect(start).toBeGreaterThan(-1)
121+
expect(end).toBeGreaterThan(start)
122+
return source.slice(start, end)
123+
}
124+
125+
test("emit block calls buildContextWindow (not inline ternary)", async () => {
126+
expect(await payloadBlock()).toContain("buildContextWindow")
120127
})
121128

122129
test("emit block includes context field", async () => {
123-
const source = await Bun.file(RUN_SRC).text()
124-
const emitIdx = source.indexOf('emit("message_complete"')
125-
expect(emitIdx).toBeGreaterThan(-1)
126-
// emit object spans ~400 chars; search up to closing paren
127-
const block = source.slice(emitIdx, emitIdx + 500)
128-
expect(block).toContain("context")
130+
expect(await payloadBlock()).toContain("context,")
129131
})
130132

131133
test("cost field is still emitted (not regressed)", async () => {
132-
const source = await Bun.file(RUN_SRC).text()
133-
const idx = source.indexOf('emit("message_complete"')
134-
expect(idx).toBeGreaterThan(-1)
135-
const block = source.slice(idx, idx + 800)
136-
expect(block).toContain("cost:")
134+
expect(await payloadBlock()).toContain("cost:")
137135
})
138136

139137
test("getModel catch rethrows non-ModelNotFoundError (targeted catch, not swallow-all)", async () => {

0 commit comments

Comments
 (0)