Skip to content

Commit 4487f1b

Browse files
authored
OSAC-4741: Fix empty/truncated Gemini responses, retry for a real answer (#499)
Confirmed live: run 33965773078's diagnosis came back completely empty — "(empty response from Gemini)", "Confidence: not reported by the model", "Estimated cost: unavailable (response had no usage data)" — with zero console output in the job log to explain why. gemini-2.5-pro's internal "thinking" tokens count against the same output-token budget as its visible answer (a documented, widely-hit reliability issue — see [googleapis/python-genai#782](googleapis/python-genai#782), [#811](googleapis/python-genai#811)). Left unset, thinking defaults to dynamic/unbounded, so a genuinely hard diagnosis (our prompts are long, `MAX_TOOL_CALLS` allows up to 15 evidence-gathering round trips, and the 92%-confidence bar demands thorough reasoning) can spend its entire output budget on internal reasoning alone and return nothing at all. The failures most in need of a real answer — the hard, ambiguous ones — are exactly the ones most at risk of this. Two layers, since the goal is a genuine answer, not just a clearly labeled failure: 1. **Prevention**: `max_output_tokens` set to the model's actual documented maximum (65536, verified — not a guessed cap), and `thinking_config.thinking_budget` bounded to 24576, guaranteeing 40k+ tokens of headroom for the visible answer regardless of how much the model reasons first (real diagnoses run 1-5k output tokens per the cost data already collected). 2. **Recovery**: even bounded, the visible answer itself (Evidence quotes, Causal chain bullets) can still hit `max_output_tokens` on a genuinely long diagnosis. `_generate_with_retry` detects `finish_reason=MAX_TOKENS` specifically (the one failure mode retrying can actually fix — a safety block or recitation flag isn't caused by length) and sends ONE follow-up turn on the same chat asking for a significantly more concise rewrite. A fresh turn gets its own full token budget, and the model already has every gathered tool-call result in context, so this is a real repair, not a relabeled failure. If the retry also hits the limit, whichever attempt has actual text is used, explicitly flagged `⚠️ Truncated: hit the output token limit even after a retry -- treat as incomplete` in the footer, rather than silently presenting a cut-off answer as if it were complete. Also, so an empty response is never a silent mystery again: `_describe_empty_response()` pulls `finish_reason` (or a prompt-level `block_reason`) out of the response defensively, prints it to the job log, and folds it into the fallback text itself — e.g. "(empty response from Gemini: finish_reason=MAX_TOKENS)" instead of a bare, unexplained one. ## Test plan - [x] `_describe_empty_response` against 4 response shapes (MAX_TOKENS, a prompt-level SAFETY block, neither present, and a response object that raises on attribute access) — all sensible, none crash - [x] `_generate_with_retry` against 5 scripted chat scenarios: clean first response (no retry), first hits the limit empty then retry succeeds, first is truncated-but-non-empty then retry succeeds, both attempts hit the limit with the retry non-empty (flagged truncated), both hit the limit but the retry is empty (falls back to the first attempt's partial text, flagged truncated) — all 5 correct - [x] `python3 .github/scripts/test_ai_diagnose_failure.py` — 14/14 pass - [x] `python3 .github/scripts/test_upsert_pr_comment.py` — 15/15 pass - [ ] Confirm on the next real hard-to-diagnose E2E failure that we get a genuine (or at worst, explicitly-flagged-truncated) answer instead of an empty one
1 parent 37183ca commit 4487f1b

1 file changed

Lines changed: 217 additions & 4 deletions

File tree

‎.github/scripts/ai-diagnose-failure.py‎

Lines changed: 217 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -287,6 +287,25 @@ def extract_log_signal(artifact_dir):
287287
# per call; raised from 5000 now that MAX_TOOL_CALLS affords more budget.
288288
MAX_TOOL_READ_CHARS = 8000
289289
MAX_LISTED_FILES = 300
290+
291+
# gemini-2.5-pro's "thinking" tokens count against the SAME output-token
292+
# budget as its visible answer (confirmed against multiple real reports of
293+
# this exact failure mode, e.g. googleapis/python-genai#782 and #811) --
294+
# left unset, thinking defaults to dynamic/unbounded, so a genuinely hard
295+
# diagnosis (long prompt, high MAX_TOOL_CALLS budget, a 92%-confidence bar
296+
# forcing thorough reasoning) can spend its ENTIRE output budget on
297+
# internal reasoning and return a completely empty response with
298+
# finish_reason=MAX_TOKENS, sometimes with no usage_metadata either --
299+
# confirmed live (run 33965773078: "(empty response from Gemini)",
300+
# "Estimated cost: unavailable (response had no usage data)"). Ironically
301+
# the hardest, most-in-need-of-a-real-answer failures are the ones most at
302+
# risk of this. MAX_OUTPUT_TOKENS is the model's actual hard maximum (not
303+
# a made-up cap); THINKING_BUDGET_TOKENS reserves a bounded, generous slice
304+
# of it for thinking specifically, guaranteeing real diagnoses (typically
305+
# 1-5k output tokens per format_cost_line's own observed numbers) always
306+
# have room left over regardless of how much the model reasons first.
307+
MAX_OUTPUT_TOKENS = 65536
308+
THINKING_BUDGET_TOKENS = 24576
290309
# The bar a diagnosis must clear before it's presented as definitive, rather
291310
# than deferring to "go check the logs yourself" -- see CONFIDENCE_PATTERN.
292311
# Raised from 85: real artifacts are often dominated by noise unrelated to
@@ -690,6 +709,55 @@ def compute_cost(usage_metadata, model):
690709
return cost_usd, input_tokens, output_tokens
691710

692711

712+
def aggregate_cost(usage_metadata_list, model):
713+
"""Sum compute_cost's numbers across EVERY generation attempt actually
714+
made for one diagnosis -- the initial send_message plus a retry turn,
715+
if _generate_with_retry sent one -- rather than just the final
716+
attempt's own usage_metadata.
717+
718+
This matters specifically because of the retry: each attempt is billed
719+
on its own (Gemini resends the whole growing conversation as input on
720+
every turn, including the prior attempt's own output as context, and
721+
bills a fresh set of output tokens for whatever it generates this
722+
turn). Looking only at the final attempt's usage_metadata silently
723+
drops the first attempt's real, already-incurred output-token cost --
724+
which, for exactly the hard-to-diagnose cases this retry exists for,
725+
can be tens of thousands of thinking/output tokens at gemini-2.5-pro's
726+
output rate. Not double counting: per-turn billing really does charge
727+
for the resent context each turn, so summing each turn's own
728+
(input, output) as reported by that turn's own usage_metadata is the
729+
actual total cost, not an overcount.
730+
731+
Skips (rather than aborting on) any attempt whose usage_metadata is
732+
unavailable, same fail-open behavior as compute_cost. Returns
733+
(None, None, None) only if NONE of the attempts had usable token
734+
counts; if at least one did, but pricing for `model` is unknown,
735+
returns (None, total_input_tokens, total_output_tokens) -- same
736+
None-cost-but-real-tokens contract as compute_cost.
737+
"""
738+
total_cost = 0.0
739+
total_input_tokens = 0
740+
total_output_tokens = 0
741+
any_usage = False
742+
cost_known = True
743+
for usage_metadata in usage_metadata_list:
744+
cost_usd, input_tokens, output_tokens = compute_cost(usage_metadata, model)
745+
if input_tokens is None or output_tokens is None:
746+
continue
747+
any_usage = True
748+
total_input_tokens += input_tokens
749+
total_output_tokens += output_tokens
750+
if cost_usd is None:
751+
cost_known = False
752+
else:
753+
total_cost += cost_usd
754+
if not any_usage:
755+
return None, None, None
756+
if not cost_known:
757+
return None, total_input_tokens, total_output_tokens
758+
return total_cost, total_input_tokens, total_output_tokens
759+
760+
693761
def format_cost_line(cost_usd, input_tokens, output_tokens, model):
694762
"""Render compute_cost's numbers as the human-readable line appended
695763
to the confidence/cost footer. A bare "$0.0000" for the unavailable
@@ -727,6 +795,131 @@ def count_tool_calls(chat):
727795
)
728796

729797

798+
def _finish_reason(resp):
799+
"""First candidate's finish_reason, or None if unavailable. Defensive:
800+
candidates can be absent entirely depending on why generation stopped
801+
(e.g. a prompt-level safety block never produces one at all), and this
802+
must never raise -- shared by _describe_empty_response and
803+
_hit_max_tokens below, which each need this same defensive lookup.
804+
"""
805+
try:
806+
candidates = resp.candidates or []
807+
if candidates:
808+
return getattr(candidates[0], "finish_reason", None)
809+
except Exception: # noqa: BLE001
810+
pass
811+
return None
812+
813+
814+
# Stringified rather than compared against a specific enum type: whether
815+
# finish_reason comes back as a plain string or a google.genai enum member
816+
# depends on SDK details we shouldn't have to track here, but str(...) on
817+
# either reliably contains this substring.
818+
MAX_TOKENS_FINISH_REASON = "MAX_TOKENS"
819+
820+
821+
def _hit_max_tokens(resp):
822+
"""True if this response's own finish_reason shows it hit
823+
max_output_tokens before finishing -- the one failure mode a "please
824+
be more concise" retry can actually fix. A safety block or recitation
825+
flag isn't caused by length, so retrying with that ask wouldn't
826+
address why generation actually stopped.
827+
"""
828+
return MAX_TOKENS_FINISH_REASON in str(_finish_reason(resp) or "")
829+
830+
831+
def _describe_empty_response(resp):
832+
"""Best-effort explanation for why resp.text came back empty --
833+
printed to the job log and folded into the visible fallback text, so a
834+
future occurrence is self-diagnosing instead of the silent mystery run
835+
33965773078 was (zero console output, "(empty response from Gemini)"
836+
with no cost data either, and no way to tell why after the fact).
837+
838+
Every attribute access is defensive: candidates/finish_reason/
839+
prompt_feedback can each be absent depending on why generation
840+
stopped, and this must never itself raise -- a footer-diagnostics bug
841+
is not worth losing the real (if unhelpful) fallback text over.
842+
"""
843+
finish_reason = _finish_reason(resp)
844+
if finish_reason is not None:
845+
return f"finish_reason={finish_reason}"
846+
try:
847+
block_reason = getattr(getattr(resp, "prompt_feedback", None), "block_reason", None)
848+
if block_reason is not None:
849+
return f"prompt blocked: {block_reason}"
850+
except Exception: # noqa: BLE001
851+
pass
852+
return "no candidates/finish_reason available"
853+
854+
855+
def _generate_with_retry(chat, prompt):
856+
"""Send `prompt`, retrying ONCE with a "be more concise" follow-up
857+
turn if the response hit max_output_tokens (see _hit_max_tokens for
858+
why only that specific finish_reason is worth retrying at all).
859+
860+
A fresh turn gets its own full max_output_tokens budget again, and the
861+
model already has every read_artifact_file call's evidence sitting in
862+
this same chat's history -- asking it to compress what it already
863+
figured out into a tighter answer is a real repair, not just a
864+
relabeled failure. Only one retry: if it's still this verbose on a
865+
second attempt, a third pass is unlikely to help and just doubles the
866+
cost again for no gain.
867+
868+
Returns (resp, truncated, usage_metadata_list). `truncated` is True
869+
only if the response being returned STILL hit max_output_tokens after
870+
the retry (with or without any text) -- the caller uses this to flag
871+
an incomplete diagnosis explicitly rather than silently presenting a
872+
cut-off answer as if it were a complete one. `usage_metadata_list`
873+
carries every attempt actually made (one entry, or two if a retry was
874+
sent) so the caller can add up the REAL total cost across attempts --
875+
see aggregate_cost's docstring for why the final attempt's
876+
usage_metadata alone would silently drop the first attempt's already-
877+
incurred cost.
878+
"""
879+
resp = chat.send_message(prompt)
880+
usage_metadata_list = [resp.usage_metadata]
881+
if not _hit_max_tokens(resp):
882+
return resp, False, usage_metadata_list
883+
print(
884+
f"WARNING: Gemini hit its output token limit ({_describe_empty_response(resp)}); "
885+
"retrying once for a more concise answer.",
886+
file=sys.stderr,
887+
)
888+
retry_resp = chat.send_message(
889+
"Your previous response was cut off for exceeding the output token limit. "
890+
"Respond again from scratch with the SAME required section headers and format, "
891+
"but be significantly more concise: shorter Evidence quotes (one line each is "
892+
"enough), fewer Causal chain bullets, and no filler -- prioritize actually "
893+
"reaching the Conclusion and the final Confidence line over exhaustive detail."
894+
)
895+
usage_metadata_list.append(retry_resp.usage_metadata)
896+
if _hit_max_tokens(retry_resp):
897+
print(
898+
f"WARNING: Retry also hit the output token limit ({_describe_empty_response(retry_resp)}); "
899+
"giving up after one retry.",
900+
file=sys.stderr,
901+
)
902+
# Prefer whichever attempt actually has SOME text -- a truncated
903+
# but non-empty answer is still more useful (once flagged below)
904+
# than a totally empty one.
905+
return (retry_resp if retry_resp.text else resp), True, usage_metadata_list
906+
if retry_resp.text:
907+
return retry_resp, False, usage_metadata_list
908+
# The retry did NOT hit the length limit (e.g. a safety/recitation
909+
# block on the retry turn, unrelated to length) but still came back
910+
# with no text -- do not present that as the "clean" result. Fall back
911+
# to the ORIGINAL response instead: it's the one that actually hit
912+
# max_output_tokens, so `truncated=True` still accurately describes it
913+
# whether or not it has any text of its own.
914+
print(
915+
f"WARNING: Retry returned no text without hitting the token limit "
916+
f"({_describe_empty_response(retry_resp)}); falling back to the "
917+
"original (truncated) response.",
918+
file=sys.stderr,
919+
)
920+
return resp, True, usage_metadata_list
921+
922+
730923
def call_gemini(prompt, artifact_dir):
731924
from google import genai
732925
from google.genai import types
@@ -744,17 +937,27 @@ def call_gemini(prompt, artifact_dir):
744937
automatic_function_calling=types.AutomaticFunctionCallingConfig(
745938
maximum_remote_calls=MAX_TOOL_CALLS
746939
),
940+
max_output_tokens=MAX_OUTPUT_TOKENS,
941+
thinking_config=types.ThinkingConfig(thinking_budget=THINKING_BUDGET_TOKENS),
747942
),
748943
)
749-
resp = chat.send_message(prompt)
750-
text = resp.text or "(empty response from Gemini)"
944+
resp, truncated, usage_metadata_list = _generate_with_retry(chat, prompt)
945+
if resp.text:
946+
text = resp.text
947+
else:
948+
reason = _describe_empty_response(resp)
949+
print(f"WARNING: Gemini returned an empty response ({reason}).", file=sys.stderr)
950+
text = f"(empty response from Gemini: {reason})"
751951
category = None
752952
cost_usd = input_tokens = output_tokens = None
753953
try:
754954
text, category = extract_category(text)
755955
text, confidence = extract_confidence(text)
756956
tool_calls = count_tool_calls(chat)
757-
cost_usd, input_tokens, output_tokens = compute_cost(resp.usage_metadata, GEMINI_MODEL)
957+
# Sums every attempt's own usage_metadata (see aggregate_cost's
958+
# docstring) so a retry's real, already-incurred first-attempt
959+
# cost is never silently dropped from the reported total.
960+
cost_usd, input_tokens, output_tokens = aggregate_cost(usage_metadata_list, GEMINI_MODEL)
758961
cost_line = format_cost_line(cost_usd, input_tokens, output_tokens, GEMINI_MODEL)
759962
if tool_calls:
760963
tool_calls_text = f"{tool_calls} tool call{'s' if tool_calls != 1 else ''}"
@@ -763,7 +966,17 @@ def call_gemini(prompt, artifact_dir):
763966
# nonzero tool-call count is real signal on its own and
764967
# shouldn't disappear just because usage_metadata was empty.
765968
cost_line = f"{cost_line}, {tool_calls_text}" if cost_line else tool_calls_text
766-
parts = filter(None, [format_confidence_line(confidence), cost_line])
969+
# Even after the retry in _generate_with_retry, the response can
970+
# still be truncated (verbose model, hard failure) -- surfaced
971+
# explicitly rather than silently presenting a cut-off answer
972+
# (missing its Conclusion/Confidence line, or worse, mid-sentence)
973+
# as if it were a complete one.
974+
truncated_line = (
975+
"⚠️ Truncated: hit the output token limit even after a retry -- treat as incomplete"
976+
if truncated
977+
else None
978+
)
979+
parts = filter(None, [truncated_line, format_confidence_line(confidence), cost_line])
767980
footer = f"<sub>{' | '.join(parts)}</sub>"
768981
except Exception: # noqa: BLE001 -- a footer-formatting bug must never lose a real diagnosis
769982
footer = ""

0 commit comments

Comments
 (0)