You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit c02df89
Browse filesBrowse the repository at this point in the historyBrowse files
fix: bill cached and thinking tokens and carry refusal usage to the dispatcher
Review findings on the Gemini comparison adapter:
- Cached and thinking tokens were subtracted from the billable totals, so
real tokens were priced at $0. promptTokenCount already includes the
cached content and thoughtsTokenCount is separate from
candidatesTokenCount (UsageMetadata), and output is billed including
thinking. Input now keeps the cached tokens (an upper bound at the input
rate), output adds the thoughts, and the output ceiling counts thinking,
as max_output_tokens does.
- Refusal counts sat on an attribute live_dispatcher never reads, so a
billed void cell settled as unknown usage and halted the pilot.
GeminiRefused now carries input_tokens / output_tokens, every refusal
after a billed response sets them, and a void finish with no text part
keeps its usage.
- The usage shape the API actually sends still settled as unknown: it
omits zero counters (proto3 JSON) and adds per-modality detail lists
(googleapis/python-genai#782). An omitted candidatesTokenCount is now a
zero when totalTokenCount proves it, and the prompt, cache and candidate
detail lists are accepted when they hold only text tokens that add up to
their counter; another modality is priced at its own rate, so it refuses.
- A STOP answer with only empty text was graded; it is now void and
carries its counts.
- A blocked prompt is reported as blocked before the model identity check,
and carries counts only when they are this model's and known.
- Text beside a zero candidate count (reported or omitted) was graded and
its output settled at $0. Text is not generated in zero tokens, so the
output is now unknown: the prompt count is carried, the output is None,
and the dispatcher keeps a cost floor and stops the pilot for review.
- A refusal before anything was sent (a changed authorization record, an
unknown profile, a ceiling above the profile maximum, an invalid
request, an exhausted MockTransport) carried no counts, so the
dispatcher halted the pilot with nothing to reconcile. It now carries a
known zero and settles at $0.
- A part holding only a thoughtSignature voided a STOP answer; it now adds
no text. A part with any other payload still refuses.
Tests: 37 (25 at b5a0e46). End-to-end tests through live_dispatcher: a
void cell in the real response shape settles as "observed" with nothing
pending and the pilot not halted, its cost pinned at the prices-2 rates;
an answer with unreported output keeps a floor and halts; a refusal
before sending settles at $0.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Copy file name to clipboardExpand all lines: docs/GEMINI_COMPARISON_ADAPTER.md
+52-12Lines changed: 52 additions & 12 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -9,7 +9,7 @@ comparison is a separate, explicitly authorized change (see *Bindings* below).
9
9
10
10
Files: `tools/gemini_adapter.py` (mapping, gates and a `MockTransport` for dry
11
11
runs and tests), `tools/gemini_comparison_authorization.json` (the unqualified
12
-
host record), `tools/test_gemini_adapter.py` (25 mock tests, in the regression
12
+
host record), `tools/test_gemini_adapter.py` (37 mock tests, in the regression
13
13
gate).
14
14
15
15
## What it maps
@@ -32,20 +32,60 @@ Responses map back to text and billable usage:
32
32
33
33
| Response field | Use | Refusal |
34
34
| --- | --- | --- |
35
-
|`promptFeedback.blockReason`|—| prompt blocked |
35
+
|`promptFeedback.blockReason`|checked first: a blocked prompt returns no candidates and may not name its model| prompt blocked |
36
36
|`modelVersion`| must equal the qualified model | any other id (never a fallback) |
37
-
|`candidates`| exactly one, with non-empty text parts | zero or several candidates, missing text |
37
+
|`candidates`| exactly one, with non-empty text for a `STOP` answer; a part holding only a `thoughtSignature` adds no text | zero or several candidates, a `STOP` answer with no text or only empty text, a part with any other payload|
38
38
|`finishReason`| must be `STOP` for a graded answer |`MAX_TOKENS`, `SAFETY`, `RECITATION`, `OTHER`, missing — the cell is void |
39
39
|`usageMetadata` absent |`(None, None)` — usage stays **null**, never zero | — |
40
-
|`promptTokenCount`, `candidatesTokenCount`| billable totals | missing, negative or non-integer |
41
-
|`cachedContentTokenCount`|**subtracted** from the prompt total | exceeds the prompt total |
42
-
|`thoughtsTokenCount`|**subtracted** from the candidate total | exceeds the candidate total |
43
-
|`totalTokenCount`| must equal `prompt + candidates`| any other total |
44
-
| any other usage field | — | unmodeled usage (charge unknown) |
45
-
46
-
A partial finish reason refuses the answer but carries the parsed counts on the
47
-
exception (`err.usage`), so a future host can record spend without grading a
48
-
truncated or filtered answer as capability.
40
+
|`promptTokenCount`, `candidatesTokenCount`| billable totals; an omitted `candidatesTokenCount` is a zero only when `totalTokenCount` proves it | prompt count missing; candidate count missing with no total; negative or non-integer; a zero candidate count beside generated text (output unknown, prompt kept) |
41
+
|`cachedContentTokenCount`| kept **inside** the prompt total (it is already included there) and priced at the full input rate, a conservative upper bound | exceeds the prompt total |
42
+
|`thoughtsTokenCount`|**added** to the candidate total: thinking is separate from `candidatesTokenCount` and billed as output | negative or non-integer |
43
+
|`totalTokenCount`| must equal `prompt + candidates + thoughts`| any other total |
44
+
|`promptTokensDetails`, `cacheTokensDetails`, `candidatesTokensDetails`| per-modality breakdowns: each must hold only `TEXT` entries that add up to its counter | another modality (priced at its own rate), a sum that disagrees, a malformed entry |
45
+
| billed output (`candidates + thoughts`) | must fit the declared `maxOutputTokens`| above the ceiling |
46
+
| any other usage field (`toolUsePromptTokenCount`, `toolUsePromptTokensDetails`, `serviceTier`, …) | — | unmodeled usage (charge unknown) |
47
+
48
+
Provider facts behind these rows: `promptTokenCount` "is still the total effective
49
+
prompt size" when content is cached, and `totalTokenCount` is the sum of prompt,
50
+
tool-use prompt, thoughts and candidates (https://ai.google.dev/api/generate-content,
51
+
`UsageMetadata`); output is priced "including thinking tokens" and cached tokens at
52
+
a reduced, nonzero rate, and audio input at a higher rate than text, image or video
53
+
(https://ai.google.dev/gemini-api/docs/pricing). The dispatcher prices only
54
+
`(input, output)` at two rates, so subtracting either subset would bill real tokens
55
+
at $0, and a modality other than text would be priced at the wrong rate.
56
+
`max_output_tokens` "applies to the combined total of thinking tokens and output
57
+
tokens", and a response cut there still bills its thinking
58
+
(https://ai.google.dev/gemini-api/docs/thinking), so the ceiling row is a sanity
59
+
check on a cap the provider enforces.
60
+
61
+
The API omits a counter whose value is zero (the proto3 JSON mapping,
62
+
https://protobuf.dev/programming-guides/json/). A thinking model that spends its
63
+
whole budget therefore answers `MAX_TOKENS` with no `candidatesTokenCount` and no
64
+
text part, as in googleapis/python-genai#782 (`prompt_token_count=1674`,
0 commit comments