You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 4487f1b
Browse filesBrowse the repository at this point in the historyBrowse files
OSAC-4741: Fix empty/truncated Gemini responses, retry for a real answer (#499)
Confirmed live: run 33965773078's diagnosis came back completely empty —
"(empty response from Gemini)", "Confidence: not reported by the model",
"Estimated cost: unavailable (response had no usage data)" — with zero
console output in the job log to explain why.
gemini-2.5-pro's internal "thinking" tokens count against the same
output-token budget as its visible answer (a documented, widely-hit
reliability issue — see
[googleapis/python-genai#782](googleapis/python-genai#782),
[#811](googleapis/python-genai#811)). Left
unset, thinking defaults to dynamic/unbounded, so a genuinely hard
diagnosis (our prompts are long, `MAX_TOOL_CALLS` allows up to 15
evidence-gathering round trips, and the 92%-confidence bar demands
thorough reasoning) can spend its entire output budget on internal
reasoning alone and return nothing at all. The failures most in need of
a real answer — the hard, ambiguous ones — are exactly the ones most at
risk of this.
Two layers, since the goal is a genuine answer, not just a clearly
labeled failure:
1. **Prevention**: `max_output_tokens` set to the model's actual
documented maximum (65536, verified — not a guessed cap), and
`thinking_config.thinking_budget` bounded to 24576, guaranteeing 40k+
tokens of headroom for the visible answer regardless of how much the
model reasons first (real diagnoses run 1-5k output tokens per the cost
data already collected).
2. **Recovery**: even bounded, the visible answer itself (Evidence
quotes, Causal chain bullets) can still hit `max_output_tokens` on a
genuinely long diagnosis. `_generate_with_retry` detects
`finish_reason=MAX_TOKENS` specifically (the one failure mode retrying
can actually fix — a safety block or recitation flag isn't caused by
length) and sends ONE follow-up turn on the same chat asking for a
significantly more concise rewrite. A fresh turn gets its own full token
budget, and the model already has every gathered tool-call result in
context, so this is a real repair, not a relabeled failure. If the retry
also hits the limit, whichever attempt has actual text is used,
explicitly flagged `⚠️ Truncated: hit the output token limit even after
a retry -- treat as incomplete` in the footer, rather than silently
presenting a cut-off answer as if it were complete.
Also, so an empty response is never a silent mystery again:
`_describe_empty_response()` pulls `finish_reason` (or a prompt-level
`block_reason`) out of the response defensively, prints it to the job
log, and folds it into the fallback text itself — e.g. "(empty response
from Gemini: finish_reason=MAX_TOKENS)" instead of a bare, unexplained
one.
## Test plan
- [x] `_describe_empty_response` against 4 response shapes (MAX_TOKENS,
a prompt-level SAFETY block, neither present, and a response object that
raises on attribute access) — all sensible, none crash
- [x] `_generate_with_retry` against 5 scripted chat scenarios: clean
first response (no retry), first hits the limit empty then retry
succeeds, first is truncated-but-non-empty then retry succeeds, both
attempts hit the limit with the retry non-empty (flagged truncated),
both hit the limit but the retry is empty (falls back to the first
attempt's partial text, flagged truncated) — all 5 correct
- [x] `python3 .github/scripts/test_ai_diagnose_failure.py` — 14/14 pass
- [x] `python3 .github/scripts/test_upsert_pr_comment.py` — 15/15 pass
- [ ] Confirm on the next real hard-to-diagnose E2E failure that we get
a genuine (or at worst, explicitly-flagged-truncated) answer instead of
an empty one
0 commit comments