Skip to content

Commit 99d72a5

Browse files
committed
Extend the reasoning-text fix to Gemini: includeThoughts, and split thought parts out of the answer
GeminiProvider.chat() had the same "generates reasoning, bills for it, discards it" bug as Groq/Cerebras, but one layer earlier and in two compounding parts: 1. The request never set thinkingConfig.includeThoughts. thinkingLevel and thinkingBudget only control how much the model thinks; Gemini only returns thought content in the response when includeThoughts is explicitly requested. It never was, so thought content was generated, billed, and never even arrived at the gateway to discard. 2. Even if it had arrived, the response parser would have contaminated the answer with it: text = "".join(p.get("text","") for p in parts if "text" in p) concatenates every part with a text key, with no check of the sibling part["thought"] boolean Gemini sets on reasoning parts. These two had to ship together, not just be bundled for tidiness: turning on includeThoughts without also fixing the parser would have introduced a live contamination bug that doesn't exist today, since today no thought parts return at all. Fix, same file, same method: - Request side: add "includeThoughts": True to both existing thinkingConfig dict literals (the thinkingLevel branch and the thinkingBudget branch). - Response side: split parts by their thought flag -- thought-flagged parts join into reasoning_text, everything else joins into text as before. Add reasoning_text to the return dict, reusing the same schema field and routes/chat.py wiring this PR already added for Groq/Cerebras -- no further schema changes needed. Confirmed by reading every class in providers.py and every provider registry construction site: NVIDIA, OpenRouter, and GitHub need no separate work (all three inherit chat() unchanged from OpenAICompatProvider, already covered by this PR's Groq/Cerebras fix). Ollama is separately covered by theschoolofai#29. No Anthropic/Claude chat provider exists in this codebase, and the embedding/STT/TTS provider hierarchies are a different domain, out of scope for this bug. Explicitly out of scope for this change, named rather than silently missed: - output_tokens undercounting thoughtsTokenCount (Gemini's usage fields are additive, unlike OpenAI-compat's completion_tokens, which already includes reasoning tokens as a documented subset) -- a separate, real cost-accounting gap, logged for its own fix. - googleapis/python-genai#2121: some Gemini configurations return thought content as a plain "THOUGHT:"-prefixed text part instead of setting part.thought = True. Named in a code comment, not fixed. - maxOutputTokens defaults: thinking and the visible answer share one token budget, and a tight max_tokens can starve the answer once thinking is turned on. A deployment-time config choice, not a code change. 3 new tests in tests/test_provider_reasoning.py (extending the existing file, not a new one): reasoning_text captured from a thought part; text excludes thought content when both are present -- the actual regression test for the contamination bug, confirmed RED before the fix producing literally "scratch work: 6*7=4242"; reasoning_text is None when no thought parts return. 18 passed on the full file. Full suite: 471 passed, the same two pre-existing failures already cited in this PR's original body, nothing new.
1 parent fc7e75d commit 99d72a5

2 files changed

Lines changed: 142 additions & 3 deletions

File tree

‎glc/providers.py‎

Lines changed: 26 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -850,10 +850,16 @@ async def chat(
850850
if reasoning and reasoning != "off":
851851
knob = _gemini_thinking_knob(m)
852852
if knob == "level":
853-
body["generationConfig"]["thinkingConfig"] = {"thinkingLevel": reasoning}
853+
body["generationConfig"]["thinkingConfig"] = {
854+
"thinkingLevel": reasoning,
855+
"includeThoughts": True,
856+
}
854857
reasoning_applied = True
855858
elif knob == "budget":
856-
body["generationConfig"]["thinkingConfig"] = {"thinkingBudget": _GEMINI_BUDGETS[reasoning]}
859+
body["generationConfig"]["thinkingConfig"] = {
860+
"thinkingBudget": _GEMINI_BUDGETS[reasoning],
861+
"includeThoughts": True,
862+
}
857863
reasoning_applied = True
858864

859865
url = f"{self.base_url}/models/{m}:generateContent?key={self.api_key}"
@@ -885,7 +891,23 @@ async def chat(
885891
f"gemini no candidates: {json.dumps(d)[:200]}", status=200, retryable=True
886892
)
887893
parts = cands[0].get("content", {}).get("parts", []) or []
888-
text = "".join(p.get("text", "") for p in parts if "text" in p)
894+
# Gemini marks reasoning parts with a sibling `thought: true`, not a
895+
# separate response field -- split before joining so thought content
896+
# never lands in `text`. Known gap, intentionally not handled here:
897+
# some configurations return thought content as a plain text part
898+
# prefixed "THOUGHT:" instead of setting the `thought` flag
899+
# (googleapis/python-genai#2121, upstream, unfixed as of this
900+
# writing) -- those slip through as answer text same as before.
901+
answer_chunks: list[str] = []
902+
thought_chunks: list[str] = []
903+
for p in parts:
904+
if "text" not in p:
905+
continue
906+
(thought_chunks if p.get("thought") is True else answer_chunks).append(
907+
p.get("text", "")
908+
)
909+
text = "".join(answer_chunks)
910+
reasoning_text = "".join(thought_chunks) or None
889911
tool_calls_out = []
890912
for p in parts:
891913
fc = p.get("functionCall") or p.get("function_call")
@@ -920,6 +942,7 @@ async def chat(
920942
"model": m,
921943
"tool_call_dialect": "native",
922944
"reasoning_applied": reasoning_applied,
945+
"reasoning_text": reasoning_text,
923946
}
924947

925948

‎tests/test_provider_reasoning.py‎

Lines changed: 116 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -221,6 +221,122 @@ def test_chat_response_accepts_reasoning_text():
221221
assert without_reasoning.reasoning_text is None
222222

223223

224+
# ── Gemini's own reasoning channel ───────────────────────────────────────────
225+
#
226+
# Gemini marks reasoning content with a sibling `thought: true` on the response
227+
# part, not a separate response field like Groq/Cerebras's `message.reasoning`.
228+
# Two compounding gaps: the request never set `includeThoughts`, so thought
229+
# parts never came back at all; and if they had, the old single-line
230+
# `text = "".join(p.get("text","") for p in parts if "text" in p)` would have
231+
# merged them straight into the visible answer, since it only ever checked
232+
# `"text" in p`, never the `thought` flag.
233+
234+
235+
@pytest.mark.asyncio
236+
async def test_geminis_own_reasoning_field_is_captured_not_discarded(monkeypatch):
237+
class _R:
238+
status_code = 200
239+
240+
def json(self):
241+
return {
242+
"candidates": [{
243+
"content": {"parts": [
244+
{"text": "17*24=408, sqrt(289)=17, 408-17=391", "thought": True},
245+
{"text": "391"},
246+
]},
247+
"finishReason": "STOP",
248+
}],
249+
"usageMetadata": {"promptTokenCount": 12, "candidatesTokenCount": 4},
250+
}
251+
252+
class _Client:
253+
async def __aenter__(self):
254+
return self
255+
256+
async def __aexit__(self, *a):
257+
return False
258+
259+
async def post(self, url, json=None):
260+
return _R()
261+
262+
monkeypatch.setattr(P.httpx, "AsyncClient", lambda **kw: _Client())
263+
out = await P.GeminiProvider("k", "gemini-2.5-flash", None).chat(
264+
[{"role": "user", "content": "hi"}], reasoning="high", max_tokens=64
265+
)
266+
267+
assert out["reasoning_text"] == "17*24=408, sqrt(289)=17, 408-17=391"
268+
269+
270+
@pytest.mark.asyncio
271+
async def test_gemini_text_excludes_thought_content_when_both_are_present(monkeypatch):
272+
"""The regression test: thought parts must never land in the visible answer."""
273+
class _R:
274+
status_code = 200
275+
276+
def json(self):
277+
return {
278+
"candidates": [{
279+
"content": {"parts": [
280+
{"text": "scratch work: 6*7=42", "thought": True},
281+
{"text": "42"},
282+
]},
283+
"finishReason": "STOP",
284+
}],
285+
"usageMetadata": {"promptTokenCount": 10, "candidatesTokenCount": 1},
286+
}
287+
288+
class _Client:
289+
async def __aenter__(self):
290+
return self
291+
292+
async def __aexit__(self, *a):
293+
return False
294+
295+
async def post(self, url, json=None):
296+
return _R()
297+
298+
monkeypatch.setattr(P.httpx, "AsyncClient", lambda **kw: _Client())
299+
out = await P.GeminiProvider("k", "gemini-2.5-flash", None).chat(
300+
[{"role": "user", "content": "hi"}], reasoning="high", max_tokens=64
301+
)
302+
303+
assert out["text"] == "42"
304+
assert "scratch work" not in out["text"]
305+
306+
307+
@pytest.mark.asyncio
308+
async def test_gemini_reasoning_text_is_none_when_no_thought_parts_return(monkeypatch):
309+
class _R:
310+
status_code = 200
311+
312+
def json(self):
313+
return {
314+
"candidates": [{
315+
"content": {"parts": [{"text": "391"}]},
316+
"finishReason": "STOP",
317+
}],
318+
"usageMetadata": {"promptTokenCount": 12, "candidatesTokenCount": 4},
319+
}
320+
321+
class _Client:
322+
async def __aenter__(self):
323+
return self
324+
325+
async def __aexit__(self, *a):
326+
return False
327+
328+
async def post(self, url, json=None):
329+
return _R()
330+
331+
monkeypatch.setattr(P.httpx, "AsyncClient", lambda **kw: _Client())
332+
out = await P.GeminiProvider("k", "gemini-2.5-flash", None).chat(
333+
[{"role": "user", "content": "hi"}], max_tokens=64
334+
)
335+
336+
assert out["reasoning_text"] is None
337+
assert out["text"] == "391"
338+
339+
224340
# ── the timeout cooldown ────────────────────────────────────────────────────
225341

226342

0 commit comments

Comments
 (0)