fix(qwen): stop counting cached and reasoning tokens twice - #1527

Summary

Qwen CLI usage is overcounted. Every cached prompt token is counted twice, once inside input and once as cache_read, so it is priced at both the input rate and the cache-read rate. On OpenAI-compatible providers (Qwen OAuth, DashScope, OpenRouter and similar) every reasoning token is counted twice as well. A turn with a 10,000-token prompt (9,000 of them cached) and 300 completion tokens (120 of them reasoning) is reported as 19,420 tokens, while Qwen Code recorded totalTokenCount: 10300 for it.

Cause

parse_qwen_file in crates/tokscale-core/src/sessions/qwen.rs maps promptTokenCount to input, cachedContentTokenCount to cache_read, candidatesTokenCount to output and thoughtsTokenCount to reasoning, as if the four were disjoint. In Qwen Code's chat recordings they overlap:

• promptTokenCount includes the cached prefix for every provider. The OpenAI-compatible converter copies prompt_tokens (converter.ts, streaming path L2085-L2135). The Responses converter copies input_tokens (responses-converter.ts). The Anthropic converter adds cache reads and writes to input_tokens unless the provider already counted them there (usage.ts). The Gemini API counts cached content inside the prompt.
• On OpenAI-compatible providers candidatesTokenCount is completion_tokens, which already includes the reasoning that thoughtsTokenCount reports again. Qwen Code treats it that way itself: when it has to rebuild a missing total, getEventTotalTokenCount adds thoughts_token_count on top of the output only for auth types other than openai and qwen-oauth (uiTelemetry.ts). Only the Gemini API reports thoughts beside the candidates, and the parser assumed that for every provider.

recordAssistantTurn stores that object unchanged as the record's usageMetadata (chatRecordingService.ts). TokenBreakdown::total() and compute_cost add every bucket, so the overlapping tokens were counted and billed twice. The Gemini parser got the same fix for its own promptTokenCount in #452, and Hindsight got it in #1393.

Fix

A new qwen_token_breakdown helper removes each overlap once:

• input = promptTokenCount - cachedContentTokenCount when the cached count is a valid subset of the prompt. A cached count above the prompt is kept as reported, as fix(hindsight): avoid counting cached input twice #1393 does, so the existing test_parse_qwen_with_cache_and_reasoning fixture is unchanged.
• When totalTokenCount equals promptTokenCount + candidatesTokenCount (OpenAI-compatible and Responses providers), the thoughts come out of the candidates: output = candidates - thoughts, clamped like the Codex split. The Gemini API's total adds thoughts on top of the candidates, so those records keep reasoning as a separate bucket. Records without a total also keep the current split.

For both the OpenAI-compatible and the Gemini shape, a parsed record's total now equals the totalTokenCount Qwen Code recorded. ClientId::Qwen moves to parser version 2 so cached parses of unchanged transcripts are rebuilt rather than replaying the old buckets.

Verification

• New test_parse_qwen_counts_cached_and_reasoning_subsets_once (OpenAI-compatible shape) and test_parse_qwen_keeps_gemini_thoughts_beside_candidates (Gemini shape) assert every bucket and that tokens.total() equals the recorded totalTokenCount. test_qwen_parser_version_invalidates_v1_entries pins the version bump.
• Three existing tests have fixtures with a cached count (test_parse_qwen_multi_turn, test_parse_qwen_malformed_lines, test_parse_qwen_unknown_model_fallback). They now expect input without the cached tokens: 100 becomes 95 and 300 becomes 290.
• Not compiled. The environment I prepared this in had no Rust toolchain, so cargo fmt, cargo clippy and cargo test have not been run, and CI will be the first build. To check the expected values, I ran every fixture in the qwen.rs test module through a line-by-line Python port of the new mapping. All 49 assertions held. With the old mapping, the two new tests and the three updated ones fail, and every other assertion still passes.

🤖 Generated with Claude Code

Summary by cubic

Fixes Qwen CLI usage overcounting: cached prompt tokens were counted twice (in both input and cache_read) and priced at both rates, and on OpenAI-compatible providers reasoning tokens were counted twice as well. A 10,000-token prompt (9,000 cached) with 300 completion tokens (120 reasoning) was reported as 19,420 tokens instead of 10,300.

• input now excludes cachedContentTokenCount when the cached count is a valid subset of the prompt; counts above the prompt are kept as reported.
• On OpenAI-compatible providers output now excludes thoughtsTokenCount when the recorded total shows the thoughts inside the candidates, while Gemini API records keep reasoning as a separate bucket.
• Parser version bumps to 2 so cached parses of unchanged transcripts are rebuilt instead of replaying the old buckets.
• Not compiled or tested yet: CI is the first build. Expected values were verified with a Python port of the new mapping rather than the Rust test suite.

^{Written for commit 6ad8ec6. Summary will update on new commits.}