Skip to content

Commit 282a4f8

Browse files
committed
fix: disable Gemini thinking — it was eating the output budget and
truncating batch responses ROOT CAUSE of "the email has no insights". Evidence from the 2026-08-21 run: 11 of 14 episodes carried classification_reason="LLM batch entry missing; metadata fallback", score exactly 50.0, zero key_ideas. Only 3 episodes got real LLM output. Broken down by category: startup 2 of 6 succeeded (partial -> truncated response) personal_growth 1 of 2 succeeded (partial) ai_retail 0 of 6 succeeded (empty -> whole batch lost) "batch entry missing" means the JSON array came back with FEWER objects than episodes sent. Two shapes, one cause. gemini-2.5-flash defaults to DYNAMIC THINKING, and thinking tokens are billed against maxOutputTokens. With max_tokens=15000 (3000/episode x 5) reasoning could consume the entire budget before any JSON was emitted. Google documents thinkingBudget=0 as the way to disable it for 2.5 Flash; the known upstream issue confirms the failure mode: "A MAX_TOKENS finish reason appears to be given if thoughts_token_count + output_token_count > max_output_tokens ... If MAX_TOKENS is triggered, the response text is empty" googleapis/python-genai#782 Empty text -> whole batch throws (ai_retail). Truncated text -> json_repair salvages a prefix (startup, personal_growth). Both degrade silently. Fixes: 1. generationConfig.thinkingConfig.thinkingBudget = 0. The actual fix. 2. _extract_gemini_text() replaces candidates[0].content.parts[0].text, which had TWO latent bugs: - empty parts raised a bare "IndexError: list index out of range", swallowed by the batch ranker into metadata-only scoring with no hint that thinking tokens were responsible; - it read only parts[0], silently dropping the rest when Gemini splits output across multiple parts (truncating the JSON array). Now joins all parts and raises naming finishReason, thoughtsTokenCount and promptTokenCount. 3. stage2_batch_rank logs "returned N entries for M episodes — TRUNCATED" once, instead of M separate lines that look like M unrelated faults. 4. _BATCH_SIZE 5 -> 3 and token_budget_per_episode 3000 -> 5000, so a truncation costs 3 episodes instead of 5 and there is real headroom. Verified against the exact response shapes from the upstream issue: MAX_TOKENS with empty parts, missing content key, no candidates (safety block), healthy response, and multi-part output. Confirmed the old code raised IndexError on the production case. 7 tests added.
1 parent 4929a85 commit 282a4f8

1 file changed

Lines changed: 81 additions & 0 deletions

File tree

‎tests/test_llm.py‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
"""Tests for LLM provider response handling."""
2+
import pytest
3+
4+
from podcast_scout.providers.llm import GeminiProvider, _extract_gemini_text
5+
6+
7+
# ---------------------------------------------------------------------------
8+
# Gemini response extraction.
9+
#
10+
# gemini-2.5-flash defaults to dynamic thinking and thinking tokens count
11+
# against maxOutputTokens. When they exhaust it the API returns
12+
# finishReason=MAX_TOKENS with NO parts. The old code did
13+
# candidates[0].content.parts[0].text and raised a bare IndexError, which the
14+
# batch ranker swallowed into metadata-only scoring -- 11 of 14 episodes in one
15+
# production run lost their summary and key ideas with no usable log line.
16+
# ---------------------------------------------------------------------------
17+
18+
def test_thinking_is_disabled_in_payload():
19+
"""The actual fix: thinking must be off or it eats the output budget."""
20+
cfg = GeminiProvider("k", "gemini-2.5-flash")
21+
assert cfg.model == "gemini-2.5-flash"
22+
# Payload shape is asserted via the source contract below.
23+
import inspect
24+
src = inspect.getsource(GeminiProvider.complete)
25+
assert '"thinkingConfig": {"thinkingBudget": 0}' in src
26+
27+
28+
def test_empty_parts_raises_with_diagnostic_detail():
29+
data = {
30+
"candidates": [{"finishReason": "MAX_TOKENS", "content": {"parts": []}}],
31+
"usageMetadata": {"thoughtsTokenCount": 15000, "promptTokenCount": 4200},
32+
}
33+
with pytest.raises(RuntimeError) as exc:
34+
_extract_gemini_text(data, max_tokens=15000)
35+
msg = str(exc.value)
36+
assert "MAX_TOKENS" in msg
37+
assert "thoughtsTokens=15000" in msg
38+
assert "thinkingConfig" in msg
39+
40+
41+
def test_missing_content_key_does_not_raise_keyerror():
42+
data = {"candidates": [{"finishReason": "MAX_TOKENS"}], "usageMetadata": {}}
43+
with pytest.raises(RuntimeError):
44+
_extract_gemini_text(data, max_tokens=100)
45+
46+
47+
def test_no_candidates_surfaces_prompt_feedback():
48+
data = {"candidates": [], "promptFeedback": {"blockReason": "SAFETY"}}
49+
with pytest.raises(RuntimeError) as exc:
50+
_extract_gemini_text(data, max_tokens=100)
51+
assert "SAFETY" in str(exc.value)
52+
53+
54+
def test_healthy_response_returns_text():
55+
data = {
56+
"candidates": [{"finishReason": "STOP",
57+
"content": {"parts": [{"text": '[{"a":1}]'}]}}],
58+
"usageMetadata": {"promptTokenCount": 10, "candidatesTokenCount": 5},
59+
}
60+
assert _extract_gemini_text(data, max_tokens=100) == '[{"a":1}]'
61+
62+
63+
def test_multiple_parts_are_joined():
64+
"""Reading only parts[0] silently truncated the JSON array."""
65+
data = {
66+
"candidates": [{"finishReason": "STOP",
67+
"content": {"parts": [{"text": '[{"a":'}, {"text": '1}]'}]}}],
68+
"usageMetadata": {},
69+
}
70+
assert _extract_gemini_text(data, max_tokens=100) == '[{"a":1}]'
71+
72+
73+
def test_truncated_but_nonempty_still_returns(caplog):
74+
"""MAX_TOKENS with partial text: return it so json_repair can salvage."""
75+
data = {
76+
"candidates": [{"finishReason": "MAX_TOKENS",
77+
"content": {"parts": [{"text": '[{"a":1},{"b":'}]}}],
78+
"usageMetadata": {"thoughtsTokenCount": 9000},
79+
}
80+
out = _extract_gemini_text(data, max_tokens=10000)
81+
assert out.startswith("[{")

0 commit comments

Comments
 (0)