Skip to content

fix(zen): stop double-counting reasoning_tokens in oa-compat usage - #24367

Closed
tiffanychum wants to merge 1 commit into
anomalyco:devfrom
tiffanychum:fix/24268-zen-reasoning-tokens-double-count
Closed

tiffanychum wants to merge 1 commit into
anomalyco:devfrom
tiffanychum:fix/24268-zen-reasoning-tokens-double-count

Conversation

@tiffanychum

Copy link
Copy Markdown

Summary

Fixes #24268.

The OpenAI chat-completions usage spec says completion_tokens already includes completion_tokens_details.reasoning_tokens. Zen's oaCompatHelper.normalizeUsage was reporting outputTokens = completion_tokens and reasoningTokens = reasoning_tokens without subtracting, so downstream calculateCost (which bills outputCost + reasoningCost separately at the same cost.output rate) double-billed reasoning.

The reporter's evidence:

  • API: prompt_tokens=22, completion_tokens=77, completion_tokens_details.reasoning_tokens=78 for a single "Hi"
  • Local opencode db: output: 436, reasoning: 790 for a real session (correctly excludes reasoning from output)
  • Zen Usage console (broken): output: 1226, reasoning: 790, total: 2016 — i.e. completion_tokens (1226) + reasoning_tokens (790), double-counting
  • Cost was being computed off the inflated 2016 total

Fix

Mirror openaiHelper.normalizeUsage from the OpenAI Responses helper (openai.ts) and subtract reasoning_tokens from completion_tokens before returning. Clamp at 0 because some OA-compatible providers report reasoning_tokens > completion_tokens (e.g. Moonshot Kimi K2.6 returning reasoning=78, completion=77).

This keeps outputTokens + reasoningTokens === completion_tokens (or close to it under provider quirks), which is the invariant calculateCost depends on to get the same total bill the upstream API uses.

Tests

Adds packages/console/app/test/zen-usage.test.ts with 4 cases that lock in the wire-level invariant for both helpers:

  • Reporter's real session: completion=1226, reasoning=790 → outputTokens=436, reasoningTokens=790, sum=1226 ✅
  • Reporter's "Hi" with inverted ratio: completion=77, reasoning=78 → outputTokens=0, reasoningTokens=78 (clamped) ✅
  • No reasoning at all: outputTokens left untouched ✅
  • Parity with openaiHelper.normalizeUsage for the same logical usage ✅

Test plan

  • bun test in packages/console/app/ — 7/7 pass (was 3/7 with the new tests written against the old code, all 4 new tests fail before the fix and pass after)
  • bun typecheck from repo root — all 13 tasks successful
  • No new lint errors on the modified file

Notes

  • I deliberately did not also touch openaiHelper.normalizeUsage (which already subtracts but does not clamp), since the OpenAI Responses API doesn't exhibit the reasoning > completion quirk and changing it is out of scope.
  • The bookkeeping nuance the reporter raised in the "Extended Evidence" section (cases where reasoning > completion cause weird per-message rows) becomes a clean outputTokens=0, reasoningTokens=N row instead of a negative outputTokens after this fix. Total cost ends up matching the upstream API's completion_tokens * rate to within one token of rounding.

…nomalyco#24268)

The OpenAI chat-completions usage spec says `completion_tokens` already
includes `completion_tokens_details.reasoning_tokens`. Zen's downstream
`calculateCost` bills `outputCost + reasoningCost` separately, so when
the oa-compat normalizer reported `outputTokens = completion_tokens` and
`reasoningTokens = reasoning_tokens`, reasoning was billed twice.

Mirror the OpenAI Responses helper (openai.ts) and subtract reasoning
from completion before returning. Clamp at 0 because some providers
(e.g. Moonshot Kimi K2.6) report `reasoning_tokens > completion_tokens`.

Adds unit tests for the reporter's exact payloads:
- Kimi K2.6 "Hi": prompt 22 / completion 77 / reasoning 78 -> output 0
- Real session: prompt N / completion 1226 / reasoning 790 -> output 436
- No-reasoning case: outputTokens unchanged
- Parity with `openaiHelper.normalizeUsage` for the same logical usage
@github-actions github-actions Bot added the needs:compliance This means the issue will auto-close after 2 hours. label Apr 25, 2026
@github-actions

Copy link
Copy Markdown
Contributor

This PR doesn't fully meet our contributing guidelines and PR template.

What needs to be fixed:

  • PR description is missing required template sections. Please use the PR template.

Please edit this PR description to address the above within 2 hours, or it will be automatically closed.

If you believe this was flagged incorrectly, please let a maintainer know.

@github-actions

Copy link
Copy Markdown
Contributor

This pull request has been automatically closed because it was not updated to meet our contributing guidelines within the 2-hour window.

Feel free to open a new pull request that follows our guidelines.

@github-actions github-actions Bot removed the needs:compliance This means the issue will auto-close after 2 hours. label Apr 25, 2026
@github-actions github-actions Bot closed this Apr 25, 2026
@ceshine

ceshine commented Apr 26, 2026

Copy link
Copy Markdown

@tiffanychum

(I'm the reporter of this issue: #24268)

Thanks for the quick pull request! You clearly understood the points I made in the issue. I really appreciate it.

I see a minor issue:

Reporter's "Hi" with inverted ratio: completion=77, reasoning=78 → outputTokens=0, reasoningTokens=78 (clamped) ✅

I think in this case reasoningTokens also needs to be clamped to 77, not the inflated 78. This will make outputTokens + reasoningTokens = completion, which matches the actual token count being billed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bug: OpenCode Go/Zen appears to overcount output tokens when reasoning tokens are present

2 participants