fix(agents): Responses turns that end incomplete report zero tokens and zero cost#109904
Conversation
The agent-side Responses processor only had a terminal branch for response.completed. A stream that ends with response.incomplete — the max_output_tokens and content_filter cases, and what Azure emits on early truncation — matched no branch at all, so the event was dropped: usage was never recorded and stopReason was never set. The turn reports zero tokens and zero cost, which is the drift in openclaw#100954. openclaw#109615 fixed the same split on the package-side processor by finalizing completed and incomplete through one finalizeResponse. This does the same for the agent path, which openclaw#109615 did not touch: both terminal events now record usage, cost and service-tier pricing through one helper. The helper moves to its own module rather than growing openai-responses-transport.ts, which is a legacy file the max-lines ratchet will not let grow; extracting the block drops it from 2502 to 2444 lines. Content-filtered turns are mapped to a provider error instead of a plain length stop, matching what the package side already does, so the two terminal surfaces do not disagree.
|
Codex review: needs real behavior proof before merge. Reviewed July 18, 2026, 5:47 PM ET / 21:47 UTC. Summary PR surface: Source +89, Tests +333. Total +422 across 10 files. Reproducibility: yes. in source: current main has no agent-side Review metrics: none identified. Stored data model Root-cause cluster Members:
Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything. Merge readiness Overall follows the weaker of proof and patch quality, so missing proof can cap an otherwise strong patch. Rank-up moves:
Proof guidance:
Risk before merge
Maintainer options:
Next step before merge
Security Review detailsBest possible solution: Retain the shared terminal mapper and guarded agent adapter, then add a redacted current-head real-socket or live-provider transcript showing terminal-only length-limited text is recovered once, while content-filtered and tool-call cases remain non-recoverable. Do we have a high-confidence way to reproduce the issue? Yes in source: current main has no agent-side Is this the best way to solve the issue? Yes structurally: sharing terminal usage and stop-reason mapping avoids drift with the package-side processor, while the agent adapter preserves its distinct reasoning-token and transcript responsibilities; current-head real behavior proof remains the missing merge gate. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning high; reviewed against 4e11da2872ec. Label changesLabel changes:
Label justifications:
Evidence reviewedPR surface: Source +89, Tests +333. Total +422 across 10 files. View PR surface stats
What I checked:
Likely related people:
What the crustacean ranks mean
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics. How this review workflow works
Review history (12 earlier review cycles; latest 8 shown)
|
…ncomplete-agent-usage
Terminal handling now covers response.incomplete, which also routed those events into backfillCompletedResponseOutput. That reconstruction exists to recover a final answer when item events never arrived; an incomplete turn has no final answer, so replaying its partial output persisted truncated text the streaming path never emitted. Usage and stop-reason recording still apply to both terminal events.
The agent transport mapped terminal usage buckets, cost, and stop reasons in parallel with the package-side processor, so the two could drift on token buckets, service-tier pricing, or future terminal-event semantics. Both now call one canonical mapper. openai-responses-shared.ts cannot be imported across the package boundary, so the mapper lives in its own module re-exported through the existing internal/openai entry point; no new subpath is introduced. The agent module keeps only the reasoning-token accounting the package path does not track. Merging the two revealed a real disagreement on totalTokens: the package took the reported total, the agent summed the split buckets. The canonical rule is now max(bucket sum, reported total), which keeps the reported value while covering both payloads that omit total_tokens and payloads whose cached_tokens exceed input_tokens, where clamping leaves the reported total short.
…ncomplete-agent-usage
…ncomplete-agent-usage
…ncomplete-agent-usage
…ncomplete-agent-usage
Co-authored-by: Yigtwxx <[email protected]>
The head's partial-output preservation is only safe while it stays a recovery path. Pin both sides of that guard: text that already streamed must not be replayed from the terminal payload, and a non-length incomplete stop must not surface partial text as an answer.
Co-authored-by: Yigtwxx <[email protected]>
…ncomplete-agent-usage
Co-authored-by: Yiğit ERDOĞAN <[email protected]>
|
Merged via squash.
|
…nd zero cost (openclaw#109904) * fix(agents): record token usage when a Responses turn ends incomplete The agent-side Responses processor only had a terminal branch for response.completed. A stream that ends with response.incomplete — the max_output_tokens and content_filter cases, and what Azure emits on early truncation — matched no branch at all, so the event was dropped: usage was never recorded and stopReason was never set. The turn reports zero tokens and zero cost, which is the drift in openclaw#100954. openclaw#109615 fixed the same split on the package-side processor by finalizing completed and incomplete through one finalizeResponse. This does the same for the agent path, which openclaw#109615 did not touch: both terminal events now record usage, cost and service-tier pricing through one helper. The helper moves to its own module rather than growing openai-responses-transport.ts, which is a legacy file the max-lines ratchet will not let grow; extracting the block drops it from 2502 to 2444 lines. Content-filtered turns are mapped to a provider error instead of a plain length stop, matching what the package side already does, so the two terminal surfaces do not disagree. * fix(agents): keep Responses output backfill on completed turns Terminal handling now covers response.incomplete, which also routed those events into backfillCompletedResponseOutput. That reconstruction exists to recover a final answer when item events never arrived; an incomplete turn has no final answer, so replaying its partial output persisted truncated text the streaming path never emitted. Usage and stop-reason recording still apply to both terminal events. * refactor(ai): share one Responses terminal usage mapper The agent transport mapped terminal usage buckets, cost, and stop reasons in parallel with the package-side processor, so the two could drift on token buckets, service-tier pricing, or future terminal-event semantics. Both now call one canonical mapper. openai-responses-shared.ts cannot be imported across the package boundary, so the mapper lives in its own module re-exported through the existing internal/openai entry point; no new subpath is introduced. The agent module keeps only the reasoning-token accounting the package path does not track. Merging the two revealed a real disagreement on totalTokens: the package took the reported total, the agent summed the split buckets. The canonical rule is now max(bucket sum, reported total), which keeps the reported value while covering both payloads that omit total_tokens and payloads whose cached_tokens exceed input_tokens, where clamping leaves the reported total short. * fix(agents): preserve partial incomplete Responses output Co-authored-by: Yigtwxx <[email protected]> * test(agents): cover incomplete terminal output backfill boundaries The head's partial-output preservation is only safe while it stays a recovery path. Pin both sides of that guard: text that already streamed must not be replayed from the terminal payload, and a non-length incomplete stop must not surface partial text as an answer. * fix(agents): avoid shadowing Responses options Co-authored-by: Yigtwxx <[email protected]> * style(agents): apply oxfmt to the incomplete backfill test * test(agents): register incomplete Responses suite Co-authored-by: Yiğit ERDOĞAN <[email protected]> --------- Co-authored-by: Peter Steinberger <[email protected]>
Closes #100954
What Problem This Solves
When an OpenAI/Azure Responses turn ends with
response.incomplete— themax_output_tokensandcontent_filtercases, and what Azure emits on early truncation — the agent records zero tokens andzero cost for that turn. The tokens were spent and billed; the session transcript,
usage.costRPC andusage bars just do not see them, so per-session accounting drifts low every time a turn is truncated.
This is the drift reported in #100954.
Why This Change Was Made
processResponsesStreaminsrc/agents/openai-responses-transport.tshad a terminal branch forresponse.completedonly, soresponse.incompletematched no branch in the chain and was silentlydropped:
output.usagewas never written,output.stopReasonnever set. Unlike the package-sideprocessor, this one has no "stream ended before a terminal response event" guard, so nothing surfaced the
miss — the loop just ended and the turn was reported as free.
The package side already fixed exactly this. #109615 finalizes both terminal events through one
finalizeResponse(packages/ai/src/providers/openai-responses-shared.ts), which records usage, cost andservice-tier pricing. It did not touch the agent path, and that path is live for the
openclaw-openai-responses-transport/openclaw-azure-openai-responses-transportruntimes — the Azureone being #100954's subject. So this is a sibling-parity fix: same bug, same shape of fix, the surface
#109615 left behind.
The recording lives in
src/agents/openai-responses-terminal-outcome.tsrather than growingopenai-responses-transport.ts, which is a legacy file the max-lines ratchet will not let grow. Pullingthe block out drops it from 2502 to 2444 lines, so the ratchet gets slack instead of pressure.
Addressing the Review
Both P1 findings from the last ClawSweeper cycle are fixed in this revision.
1. Incomplete terminal output: recovery only, never replay
The first version of this change refused terminal-output reconstruction for
response.incompleteoutright. @steipete revised that in
b2419b0a1a1: some compatible endpoints carry the generated textonly on the terminal event, so discarding it loses a partial answer the user should still see. The
head now backfills partial message text on an incomplete turn while still refusing to materialize an
incomplete
function_call— a truncated argument string must never become an executable tool call.That narrows the contract to a guard rather than a rule, so this push pins both sides of the guard
(
openai-transport-stream.incomplete-output.test.ts):payload repeats it, the turn keeps exactly one copy. This is the duplicate-transcript case: the
backfill is gated on
output.content.length === 0, so it stays a recovery path for streams thatemitted nothing.
content_filterstop is surfaced asan error, so its partial text is not a recoverable answer and must not be persisted as one.
Both tests are load-bearing, shown by mutating the source and observing exactly one failure each:
Neither mutation trips the other test, so the two cases are independently covered rather than
overlapping. Full file: 49 tests pass across the incomplete-output and base transport suites.
Scope note on the runtime evidence below: the main-versus-branch SSE transcript was recorded on
d050cf4da5e, before the partial-output revision, so it demonstrates the accounting fix — the originalsubject of this PR — and not the newer output-preservation behavior. The two tests above are what cover
the revised behavior; I have not produced a fresh live-socket transcript for it.
Verified on the current head
4b3362b864c, which includes both follow-up commits(
fix(agents): preserve partial incomplete Responses outputandfix(agents): avoid shadowing Responses options): 49 tests pass acrossopenai-transport-stream.incomplete-output.test.tsandopenai-transport-stream.base.test.ts, and both mutations above still fail exactly one test each onthat head.
2. One canonical terminal-usage mapper
The parallel mapper is gone. Terminal usage buckets, the stop-reason mapping and the two shared overrides
(content-filter → provider error, tool calls →
toolUse) now live in one module,packages/ai/src/providers/openai-responses-terminal-usage.ts, and bothfinalizeResponseand the agenttransport call it. The duplicated
mapStopReasonandResponsesInputTokensDetailsinopenai-responses-shared.tsare deleted; the agent module drops from 103 to 68 lines and keeps only thereasoning-token accounting the package-side
Usagetype does not carry — the thin adapter the reviewasked for.
On placement:
openai-responses-shared.tsis not reachable across the package boundary — it is not inpackages/aiexports,tsconfigpaths, or the plugin-SDK resolver, and a deep import would resolve undervitest's catch-all alias while failing
tscand runtime. So the mapper is its own module, re-exportedthrough the existing
internal/openaientry point. No new subpath, no resolver or alias-test change.One real disagreement surfaced by merging them
The two implementations computed
totalTokensdifferently: the package took the reportedtotal_tokens, the agent summed the split buckets. The canonical rule is nowmax(bucket sum, reported total), which keeps the provider's number while covering both directions theold pair disagreed on — payloads that omit
total_tokens(proxies routinely do; reporting 0 understatesthe turn) and payloads whose
cached_tokensexceedsinput_tokens, where clamping the input bucket to 0leaves the reported total below what the buckets actually price. Both cases are pinned by tests that fail
without the rule.
Evidence
The added tests fail on
origin/mainand pass here. On main the assertions report exactly thesymptom from #100954:
The usage test drives the real
processResponsesStreamwith an Azure Responses model and an exactresponse.incompleteevent (max_output_tokens, populated usage with cached + cache-write tokens), thenasserts the split buckets and priced cost. Sub-cent totals round to zero at
toBeCloseTo's defaultprecision, so it also asserts
toBeGreaterThan(0).Runtime proof: the real transport over a real socket, main vs this branch
Beyond the unit tests, the fix was exercised end to end through the actual network path. A local HTTP
server speaks the Responses SSE wire protocol and ends the turn with
response.incomplete(
incomplete_details.reason = max_output_tokens, populated usage), and the realcreateOpenAIResponsesTransportStreamFn()is pointed at it viamodel.baseUrl. Nothing is mocked —the OpenAI SDK opens a socket, parses the event stream and hands events to
processResponsesStreamexactly as it does against
api.openai.com. The only difference from a live run is the hostname, as thetransport's own debug output confirms on both branches:
Same harness, same event, priced with real cost points — run against
origin/main(e01a880) and againstthis branch:
origin/mainstopReasonstoplengthusage.inputusage.outputusage.cacheReadusage.reasoningTokensundefinedusage.totalTokensusage.cost.totalThis is #100954 reproduced against current
mainon the agent path, and it is worse than the issuedescribes: main does not merely lose the tokens, it also records
stopReason: "stop". A turn that hit theoutput cap and was truncated mid-answer is persisted as a turn that finished successfully having spent
nothing. Downstream consumers cannot tell a truncated turn from a completed one, so nothing retries or
warns — which is why the drift is silent.
The scenario is routine rather than an edge case:
max_output_tokensis what every long answer hits.On proof shape: the usage payload in the harness is authored, so this proves the transport records
what the provider reports — not that these particular numbers came off an OpenAI invoice. For comparison,
#109615 pinned the identical package-side behavior with a synthetic terminal event
(
it("finalizes incomplete terminal events with usage"),packages/ai/src/providers/openai-responses-shared.test.ts) and no captured provider transcript. If aredacted live Azure/OpenAI transcript is still required on top of the above, a maintainer run against a
real endpoint is the remaining gap — I do not have live credentials.
Merge Risk
Low. The
response.completedpath is unchanged in behavior — the reconstruction it depends on is nowexplicitly scoped to it, and the existing transport suites cover it unchanged.
Three behaviors move, all of them the corrections above:
response.incompleterecords usage/cost and sets a stop reason instead of being dropped, soper-session totals go up for truncated turns — worth calling out for anyone watching usage
dashboards.
stopReason: "error"withProvider incomplete_reason: content_filter,matching the package side instead of a plain
lengthstop.totalTokensfollows themax(bucket sum, reported total)rule on both Responses paths, which changesthe package-side value only for payloads that omit
total_tokensor report one below the pricedbuckets.
This supersedes the agent-side half of #101608, which I opened before #109615 landed the package-side fix;
that PR is now conflicted and half-redundant, so I am closing it in favor of this narrower one.
AI-assisted.