Skip to content

Commit c02df89

Browse files
fix: bill cached and thinking tokens and carry refusal usage to the dispatcher
Review findings on the Gemini comparison adapter: - Cached and thinking tokens were subtracted from the billable totals, so real tokens were priced at $0. promptTokenCount already includes the cached content and thoughtsTokenCount is separate from candidatesTokenCount (UsageMetadata), and output is billed including thinking. Input now keeps the cached tokens (an upper bound at the input rate), output adds the thoughts, and the output ceiling counts thinking, as max_output_tokens does. - Refusal counts sat on an attribute live_dispatcher never reads, so a billed void cell settled as unknown usage and halted the pilot. GeminiRefused now carries input_tokens / output_tokens, every refusal after a billed response sets them, and a void finish with no text part keeps its usage. - The usage shape the API actually sends still settled as unknown: it omits zero counters (proto3 JSON) and adds per-modality detail lists (googleapis/python-genai#782). An omitted candidatesTokenCount is now a zero when totalTokenCount proves it, and the prompt, cache and candidate detail lists are accepted when they hold only text tokens that add up to their counter; another modality is priced at its own rate, so it refuses. - A STOP answer with only empty text was graded; it is now void and carries its counts. - A blocked prompt is reported as blocked before the model identity check, and carries counts only when they are this model's and known. - Text beside a zero candidate count (reported or omitted) was graded and its output settled at $0. Text is not generated in zero tokens, so the output is now unknown: the prompt count is carried, the output is None, and the dispatcher keeps a cost floor and stops the pilot for review. - A refusal before anything was sent (a changed authorization record, an unknown profile, a ceiling above the profile maximum, an invalid request, an exhausted MockTransport) carried no counts, so the dispatcher halted the pilot with nothing to reconcile. It now carries a known zero and settles at $0. - A part holding only a thoughtSignature voided a STOP answer; it now adds no text. A part with any other payload still refuses. Tests: 37 (25 at b5a0e46). End-to-end tests through live_dispatcher: a void cell in the real response shape settles as "observed" with nothing pending and the pilot not halted, its cost pinned at the prices-2 rates; an answer with unreported output keeps a floor and halts; a refusal before sending settles at $0. Co-Authored-By: Claude Opus 5.5 <[email protected]>
1 parent b5a0e46 commit c02df89

3 files changed

Lines changed: 552 additions & 70 deletions

File tree

‎docs/GEMINI_COMPARISON_ADAPTER.md‎

Lines changed: 52 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -9,7 +9,7 @@ comparison is a separate, explicitly authorized change (see *Bindings* below).
99

1010
Files: `tools/gemini_adapter.py` (mapping, gates and a `MockTransport` for dry
1111
runs and tests), `tools/gemini_comparison_authorization.json` (the unqualified
12-
host record), `tools/test_gemini_adapter.py` (25 mock tests, in the regression
12+
host record), `tools/test_gemini_adapter.py` (37 mock tests, in the regression
1313
gate).
1414

1515
## What it maps
@@ -32,20 +32,60 @@ Responses map back to text and billable usage:
3232

3333
| Response field | Use | Refusal |
3434
| --- | --- | --- |
35-
| `promptFeedback.blockReason` | — | prompt blocked |
35+
| `promptFeedback.blockReason` | checked first: a blocked prompt returns no candidates and may not name its model | prompt blocked |
3636
| `modelVersion` | must equal the qualified model | any other id (never a fallback) |
37-
| `candidates` | exactly one, with non-empty text parts | zero or several candidates, missing text |
37+
| `candidates` | exactly one, with non-empty text for a `STOP` answer; a part holding only a `thoughtSignature` adds no text | zero or several candidates, a `STOP` answer with no text or only empty text, a part with any other payload |
3838
| `finishReason` | must be `STOP` for a graded answer | `MAX_TOKENS`, `SAFETY`, `RECITATION`, `OTHER`, missing — the cell is void |
3939
| `usageMetadata` absent | `(None, None)` — usage stays **null**, never zero | — |
40-
| `promptTokenCount`, `candidatesTokenCount` | billable totals | missing, negative or non-integer |
41-
| `cachedContentTokenCount` | **subtracted** from the prompt total | exceeds the prompt total |
42-
| `thoughtsTokenCount` | **subtracted** from the candidate total | exceeds the candidate total |
43-
| `totalTokenCount` | must equal `prompt + candidates` | any other total |
44-
| any other usage field | — | unmodeled usage (charge unknown) |
45-
46-
A partial finish reason refuses the answer but carries the parsed counts on the
47-
exception (`err.usage`), so a future host can record spend without grading a
48-
truncated or filtered answer as capability.
40+
| `promptTokenCount`, `candidatesTokenCount` | billable totals; an omitted `candidatesTokenCount` is a zero only when `totalTokenCount` proves it | prompt count missing; candidate count missing with no total; negative or non-integer; a zero candidate count beside generated text (output unknown, prompt kept) |
41+
| `cachedContentTokenCount` | kept **inside** the prompt total (it is already included there) and priced at the full input rate, a conservative upper bound | exceeds the prompt total |
42+
| `thoughtsTokenCount` | **added** to the candidate total: thinking is separate from `candidatesTokenCount` and billed as output | negative or non-integer |
43+
| `totalTokenCount` | must equal `prompt + candidates + thoughts` | any other total |
44+
| `promptTokensDetails`, `cacheTokensDetails`, `candidatesTokensDetails` | per-modality breakdowns: each must hold only `TEXT` entries that add up to its counter | another modality (priced at its own rate), a sum that disagrees, a malformed entry |
45+
| billed output (`candidates + thoughts`) | must fit the declared `maxOutputTokens` | above the ceiling |
46+
| any other usage field (`toolUsePromptTokenCount`, `toolUsePromptTokensDetails`, `serviceTier`, …) | — | unmodeled usage (charge unknown) |
47+
48+
Provider facts behind these rows: `promptTokenCount` "is still the total effective
49+
prompt size" when content is cached, and `totalTokenCount` is the sum of prompt,
50+
tool-use prompt, thoughts and candidates (https://ai.google.dev/api/generate-content,
51+
`UsageMetadata`); output is priced "including thinking tokens" and cached tokens at
52+
a reduced, nonzero rate, and audio input at a higher rate than text, image or video
53+
(https://ai.google.dev/gemini-api/docs/pricing). The dispatcher prices only
54+
`(input, output)` at two rates, so subtracting either subset would bill real tokens
55+
at $0, and a modality other than text would be priced at the wrong rate.
56+
`max_output_tokens` "applies to the combined total of thinking tokens and output
57+
tokens", and a response cut there still bills its thinking
58+
(https://ai.google.dev/gemini-api/docs/thinking), so the ceiling row is a sanity
59+
check on a cap the provider enforces.
60+
61+
The API omits a counter whose value is zero (the proto3 JSON mapping,
62+
https://protobuf.dev/programming-guides/json/). A thinking model that spends its
63+
whole budget therefore answers `MAX_TOKENS` with no `candidatesTokenCount` and no
64+
text part, as in googleapis/python-genai#782 (`prompt_token_count=1674`,
65+
`prompt_tokens_details=[TEXT 1674]`, `thoughts_token_count=2000`,
66+
`total_token_count=3674`); the reported total proves the omitted zero. A usage
67+
block with no total proves nothing, so a missing candidate count there stays
68+
unknown.
69+
70+
Every refusal raised after a response from the qualified model carries the parsed
71+
counts as `err.input_tokens` / `err.output_tokens`, the attributes
72+
`tools/live_dispatcher.py` reads from a failed send. That covers a partial finish
73+
reason (which may carry no text at all), a `STOP` answer with only empty text,
74+
a malformed candidate and output above the ceiling, so the dispatcher records the
75+
spend without grading a truncated or filtered answer as capability. A blocked
76+
prompt carries its counts only when the response names the qualified model and its
77+
usage is known. When the counts are unknown (no `usageMetadata`, a total that
78+
disagrees with its parts, a missing candidate count with no total, a detail list
79+
that disagrees or is not text, an unmodeled field, a different or unnamed model)
80+
both stay `None`, and the dispatcher holds the whole reservation instead of
81+
settling a guess. Text is not generated in zero tokens, so text beside a zero
82+
candidate count (reported or omitted) leaves only the output unknown: the prompt
83+
count is carried, the output is `None`, and the dispatcher keeps the prompt as a
84+
cost floor and stops the pilot for review. A refusal before anything is sent (a
85+
changed authorization record, an unknown profile, a ceiling above the profile's
86+
maximum, an invalid request, an exhausted `MockTransport`) carries a known zero,
87+
`(0, 0)`, so the dispatcher settles it at $0 instead of halting with nothing to
88+
reconcile.
4989

5090
## Fail-closed gates
5191

0 commit comments

Comments
 (0)