Repository navigation
Per-turn token usage for ACP providers #20010
Replies: 6 comments
|
This proposal is beyond reproach |
|
This seems like a great idea |
|
Adapter-level live evidence for #19958, head Scope, stated plainly: this run drives GLM per-turn usage evidence (PR #19958)Provenance
Wire samples (raw PromptResponse.usage, allowlisted counters)
Recorded turnTokenUsage
Scenario log[
{
"scenario": "S1",
"prompts": 4,
"cachedReadPositive": true,
"thoughtPositive": true,
"inputSeries": [
2857,
3839,
4196,
4553
],
"outputSeries": [
629,
4,
4,
4
],
"cachedReadSeries": [
1280,
2816,
3776,
4160
],
"thoughtSeries": [
337,
0,
0,
0
]
},
{
"scenario": "S2",
"delayMs": 40,
"interruptAccepted": true,
"wireSample": null,
"stopReason": "cancelled",
"recordedStatus": "unavailable",
"recordedStatusDetail": "interrupted",
"recordedCounters": null
},
{
"scenario": "S2",
"delayMs": 150,
"interruptAccepted": true,
"wireSample": null,
"stopReason": "cancelled",
"recordedStatus": "unavailable",
"recordedStatusDetail": "interrupted",
"recordedCounters": null
},
{
"scenario": "S2",
"delayMs": 600,
"interruptAccepted": true,
"wireSample": null,
"stopReason": "cancelled",
"recordedStatus": "unavailable",
"recordedStatusDetail": "interrupted",
"recordedCounters": null
},
{
"scenario": "S2",
"delayMs": 2500,
"interruptAccepted": true,
"wireSample": null,
"stopReason": "cancelled",
"recordedStatus": "unavailable",
"recordedStatusDetail": "interrupted",
"recordedCounters": null
},
{
"scenario": "S2",
"delayMs": "response-triggered",
"interruptAccepted": true,
"wireSample": {
"inputTokens": 2847,
"outputTokens": 1109,
"totalTokens": 3956,
"cachedReadTokens": 2432,
"thoughtTokens": 394
},
"stopReason": "end_turn",
"recordedStatus": "partial",
"recordedStatusDetail": "interrupted",
"recordedCounters": {
"inputTokens": 2847,
"outputTokens": 1109,
"cachedInputTokens": 2432,
"reasoningTokens": 394
}
},
{
"scenario": "S2",
"delayMs": "mid-stream",
"interruptAccepted": true,
"wireSample": null,
"stopReason": "cancelled",
"recordedStatus": "unavailable",
"recordedStatusDetail": "interrupted",
"recordedCounters": null
},
{
"scenario": "S3",
"postResumeSample": {
"inputTokens": 3270,
"outputTokens": 4,
"totalTokens": 3274,
"cachedReadTokens": 2816,
"thoughtTokens": 0
},
"postResumeRecorded": {
"inputTokens": 3270,
"outputTokens": 4,
"cachedInputTokens": 2816,
"reasoningTokens": 0
},
"terminal": "completed",
"resumeMethods": [
"session/load"
]
}
]Readings
Isolation and redaction: the run used a temp T3 home (network disabled, cached registry) with the agent's own state redirected via Independent review: the approach and the results (including a recomputation of every invariant from the artifact and a string-by-string redaction audit) were reviewed by a separate Opus 5.5 pass; its corrections are incorporated above. Remaining open item for the gate's "check the resulting usage in a real client": an end-to-end dev-server + client run with the persisted provider-turn records, to follow. |
|
Tier A: end-to-end evidence for #19958 — real client -> dev server -> orchestration -> adapter -> glm-acp-agent 1.14.0 (Z.ai), head Setup: a git worktree of the PR branch with its own gitignored Scenarios and the persisted records
Readings
Provenance: head |
|
Note 🤖 GPT-6.1 Terra [1m] responding on behalf of Theo The linked implementation records per-turn ACP usage for the verified GLM flavor. Other flavors report unavailable until their counters are validated, as the proposal requested. #19958 adapter.ts |
|
Follow-up: #32173 proposes showing these GLM turn records on the Usage page. It's scoped to |
Uh oh!
There was an error while loading. Please reload this page.
Problem - acp providers do not get per turn token or usage attribution
Claude, Codex and OpenCode turns write
turnTokenUsage, which feeds projections and theprovider.turn.completedanalytics event. ACP turns (the Z.ai GLM registry agent, Grok, Antigravity, Kimi CLI, etc.) write nothing, so those providers have no per-turn token attribution. Every ACP flavor goes throughmakeAcpAdapterV2inpackages/provider-acp, so one change covers all of them. What it should store depends on what each agent'susagefield actually means.The ACP schema calls it "Token usage for this turn", but the upstream spec RFD still lists per-turn vs. cumulative as an open question. I probed two agents over raw ACP stdio: three prompts in sequence, one session, run 2026-10-15.
glm-acp-agent 1.14.0 returned
usageon every turn:inputTokenstotalTokens === inputTokens + outputTokens,cachedReadTokens <= inputTokens, andthoughtTokens <= outputTokensheld on every turn. Input only ticks up by about the size of the new prompt each turn; a cumulative counter would add a full context's worth. GLM is per-turn.Kimi CLI 1.52.0 never returned
usageat all.Proposal
Each ACP flavor declares how its usage should be read:
per_turn: pass the numbers through. Only for agents verified per-turn — today that's GLM.cumulative: keep a per-session baseline and record the delta. No agent needs this yet.unavailable: the default for anything unverified. Record nothing.Unknown semantics means record nothing rather than guess, so we never overcount. Usage is captured as soon as it arrives on the wire, not at turn settlement, so a Stop or interrupt can't drop it.
This only touches turn records. The Usage page scan pipeline prices usage from native history readers, so feeding ACP turns into it would double-count any provider that already has a reader; that leaves #6909 (Grok/OpenCode on the Usage page) alone. Subscription quota limits belong to #15943 and the Z.ai Coding Plan limits work. #13073 (per-thread token usage and cost) would get data for ACP threads from this.
Does this direction work — per-flavor declared semantics, per-turn records only, Usage page untouched — or would you rather make one global assumption, or wait for the spec to settle it? PR #19958 is one implementation.
All reactions