Repository navigation
fix(zen): stop double-counting reasoning_tokens in oa-compat usage - #24367
tiffanychum wants to merge 1 commit into
Conversation
…nomalyco#24268) The OpenAI chat-completions usage spec says `completion_tokens` already includes `completion_tokens_details.reasoning_tokens`. Zen's downstream `calculateCost` bills `outputCost + reasoningCost` separately, so when the oa-compat normalizer reported `outputTokens = completion_tokens` and `reasoningTokens = reasoning_tokens`, reasoning was billed twice. Mirror the OpenAI Responses helper (openai.ts) and subtract reasoning from completion before returning. Clamp at 0 because some providers (e.g. Moonshot Kimi K2.6) report `reasoning_tokens > completion_tokens`. Adds unit tests for the reporter's exact payloads: - Kimi K2.6 "Hi": prompt 22 / completion 77 / reasoning 78 -> output 0 - Real session: prompt N / completion 1226 / reasoning 790 -> output 436 - No-reasoning case: outputTokens unchanged - Parity with `openaiHelper.normalizeUsage` for the same logical usage
|
This PR doesn't fully meet our contributing guidelines and PR template. What needs to be fixed:
Please edit this PR description to address the above within 2 hours, or it will be automatically closed. If you believe this was flagged incorrectly, please let a maintainer know. |
|
This pull request has been automatically closed because it was not updated to meet our contributing guidelines within the 2-hour window. Feel free to open a new pull request that follows our guidelines. |
|
(I'm the reporter of this issue: #24268) Thanks for the quick pull request! You clearly understood the points I made in the issue. I really appreciate it. I see a minor issue:
I think in this case |
Summary
Fixes #24268.
The OpenAI chat-completions usage spec says
completion_tokensalready includescompletion_tokens_details.reasoning_tokens. Zen'soaCompatHelper.normalizeUsagewas reportingoutputTokens = completion_tokensandreasoningTokens = reasoning_tokenswithout subtracting, so downstreamcalculateCost(which billsoutputCost + reasoningCostseparately at the samecost.outputrate) double-billed reasoning.The reporter's evidence:
prompt_tokens=22, completion_tokens=77, completion_tokens_details.reasoning_tokens=78for a single "Hi"output: 436, reasoning: 790for a real session (correctly excludes reasoning from output)output: 1226, reasoning: 790, total: 2016— i.e.completion_tokens (1226) + reasoning_tokens (790), double-countingFix
Mirror
openaiHelper.normalizeUsagefrom the OpenAI Responses helper (openai.ts) and subtractreasoning_tokensfromcompletion_tokensbefore returning. Clamp at 0 because some OA-compatible providers reportreasoning_tokens > completion_tokens(e.g. Moonshot Kimi K2.6 returningreasoning=78, completion=77).This keeps
outputTokens + reasoningTokens === completion_tokens(or close to it under provider quirks), which is the invariantcalculateCostdepends on to get the same total bill the upstream API uses.Tests
Adds
packages/console/app/test/zen-usage.test.tswith 4 cases that lock in the wire-level invariant for both helpers:completion=1226, reasoning=790→outputTokens=436, reasoningTokens=790, sum=1226✅completion=77, reasoning=78→outputTokens=0, reasoningTokens=78(clamped) ✅outputTokensleft untouched ✅openaiHelper.normalizeUsagefor the same logical usage ✅Test plan
bun testinpackages/console/app/— 7/7 pass (was 3/7 with the new tests written against the old code, all 4 new tests fail before the fix and pass after)bun typecheckfrom repo root — all 13 tasks successfulNotes
openaiHelper.normalizeUsage(which already subtracts but does not clamp), since the OpenAI Responses API doesn't exhibit thereasoning > completionquirk and changing it is out of scope.reasoning > completioncause weird per-message rows) becomes a cleanoutputTokens=0, reasoningTokens=Nrow instead of a negativeoutputTokensafter this fix. Total cost ends up matching the upstream API'scompletion_tokens * rateto within one token of rounding.