[issue-8134] [SDK] fix(g-eval): locate the score token by content instead of a fixed offset - #8152

Details

• The G-Eval logprob scorer located the score digit at a hardcoded token offset and assumed single-digit scores. A two-digit score ("score": 10) tokenizes across two tokens, so only the first digit was averaged: a confident 10 parsed as ~0.099. A tokenizer folding the whitespace after the colon into the score token (" 0") made every candidate fail the decimal check and raised MetricComputationError on a perfectly parseable response.
• The score digits are now located by reconstructing the token stream and walking the first JSON object once, stepping over whole strings and reading the "score" key only at depth 1, stopping when that object closes. Last assignment wins, so duplicate top-level keys resolve the way json.loads does, and a nested "score" cannot capture the position because it sits deeper. Candidate tokens are stripped before the decimal check; a two-token span combines both positions' candidates ("1" + "0" -> 10).
• When the located span carries no probability mass (digits span punctuation-carrying tokens), the parser degrades to the text path like the short-stream and no-logprob branches instead of raising.
• Falls back to the legacy fixed offset when the first object holds no top-level key.

Change checklist

• User facing
• Documentation update

Issues

Related: #8134 (defect 3 of 4 — Resolves intentionally avoided so the four-defect issue stays open for defects 1 and 2)

AI-WATERMARK

AI-WATERMARK: yes

• Tools: OpenCode (Sisyphus agent session)

• Model(s): muse-spark

• Scope: review-feedback commits only — 5486cc3 (typehints, _to_dict wrap, no-mass degrade, content-free log line, test dedup/parametrize, punctuation-split regression test), 3b8cfe9 (DEBUG fallback log + message assertion), 0e48cce (locator value-agreement fix, two regression tests, two docstring corrections), ae048eadf (locator brace-depth fix + one regression test), 670fdd61e (one test-only commit closing a coverage hole the depth rule opened), 3b056b5d9 (locator rewritten to a single-pass structural walk, design and reference implementation from petrotiurin on the thread, plus three regression tests)

• Human verification: pytest tests/unit/evaluation/metrics/llm_judges/g_eval/ (18 passed at 3b056b5d9, 15 at 670fdd61e, 13 at 0e48cce), pytest tests/unit/evaluation/metrics/llm_judges/ 126 passed, new tests verified to fail on the pre-fix parser, [email protected] check + format clean, [email protected] clean; mutation table below re-run whole at each head, each mutation restoring the file byte-for-byte

Testing

At 3b056b5d9:

• PYTHONPATH=src python -m pytest tests/unit/evaluation/metrics/llm_judges/g_eval/ -q → 18 passed
• PYTHONPATH=src python -m pytest tests/unit/evaluation/metrics/llm_judges/ -q → 126 passed
• The three tests added in 3b056b5d9 fail against 670fdd61e's src/ and only those three do; the fifteen existing tests pass unmodified on the new locator
• uvx [email protected] check + format --check on both touched files → clean; uvx [email protected] --config-file pyproject.toml --disable-error-code misc --ignore-missing-imports on parser.py → Success: no issues found in 1 source file
• PYTHONPATH=src python -m pytest tests/unit -q --tb=no -rf → 1 failed, 5161 passed, 3 skipped in 243.78s. The failure is tests/unit/evaluation/metrics/test_heuristics.py::test_evaluation__levenshtein_ratio, in a file this diff does not touch; importlib.util.find_spec("rapidfuzz") returns None here, so an optional dependency is missing rather than anything in this change being wrong
• E2E not run (needs a live backend stack)

Historical, at earlier heads: test_punctuation_carrying_split_digits_degrade_to_text_path fails on the pre-PR parser with MetricComputationError and passes post-fix with 1.0. A broader tests/unit/evaluation/ sweep at that time showed the same failure count on the clean tree (91 failed / 927 passed) and on this branch (91 failed / 930 passed); those 91 are pre-existing environment failures unrelated to this diff.

Documentation

N/A — no user-facing API or config change.

Follow-up round (3b8cfe9). On Baz's question, the zero-probability-mass fallback now logs at DEBUG with the metric name, matching the sibling locator-failure branch, and test_punctuation_carrying_split_digits_degrade_to_text_path asserts the message as well as the value. Measured against its immediate predecessor 5486cc3, which degrades silently, the assertion fails on an empty capture; against the pre-PR parser the test fails outright with MetricComputationError. Post-fix: 11 passed in tests/unit/evaluation/metrics/llm_judges/g_eval/, ruff check and ruff format --check clean, mypy clean on parser.py.

Follow-up round (0e48cce, addressing the nested-key concern). Reproduced first, then applied the suggested agreement check. Measured on {"score":7, "reason": "ok", "breakdown": {"score":3}}, with the top-level digit at entry 3 and the nested one last in text order:

Two tests added:

• test_nested_score_key_does_not_capture_the_located_token — fails on 3b8cfe9 with AssertionError: scored from the nested key: 0.27397830512740046, passes here.
• test_unparseable_reconstruction_keeps_the_positional_match — pins the fallback when the reconstructed stream is not valid JSON.

Mutations, each restoring parser.py byte-for-byte (sha256 checked) and re-running the suite:

The third row is why the fallback stays: removing it fails 6 of the 13 tests, not just the new one, because the reconstructed text is not valid JSON in several existing fixtures (their token streams stop before the document closes). In those cases the positional choice is kept, which is today's behaviour.

Two notes on the suggestion. I did not hoist the json.loads into the one further down: that one parses choice_dict's message content, this one parses the reconstructed token text, and those can differ (the truncated-stream case), so the parses are not interchangeable. And the two docstring claims that said last-match-wins mirrors json.loads were true for top-level duplicates only, so both now state the value-agreement rule.

Verification: 13 passed in tests/unit/evaluation/metrics/llm_judges/g_eval/; uvx [email protected] check and format --check clean on both files; uvx [email protected] --config-file sdks/python/pyproject.toml --disable-error-code misc clean on parser.py (run with --ignore-missing-imports locally, since this environment has no litellm stubs).

Follow-up round (ae048eadf + 670fdd61e, on the three remaining locator claims).
Each claim was measured as one response per claim across three trees — origin/main, this
PR at 0e48cce, and after the change at 670fdd61e — instead of being reasoned about:

Row 1 is real and is fixed here. Value agreement does not identify a position: when a
breakdown restates the top-level score, both digits agree, so the reversed scan kept
returning the nested one and the weighted average came off the breakdown token's
distribution. main scores 0.6219349153822014 on that response, so this was a regression
the previous round introduced — and a worse shape than the differing-value case, because it
yields a plausible number rather than an obviously wrong one. The rule is now: take the
agreeing matches, restrict them to the shallowest brace depth, then take the last of those,
which keeps duplicate top-level keys resolving to the final one.

test_nested_score_key_with_the_same_value_does_not_win_the_locator fails pre-fix with
assert [13] == [3] and passes post-fix. Selection differential over 32 reconstructed texts
(top level only; breakdown with the same and with a different value; two criteria; criteria
inside an array; braces inside a string; duplicate top-level keys; doubly nested) changes
the chosen span in 5 of them, and all 5 are the equal-valued-nested family — the other 27
select the same span as before.

Rows 2 and 3 are not changed, and the numbers say why. The escaped key returns None
from the locator, so the caller falls back to the fixed offset and lands on 0.7, the same
value json.loads gives, and better than main, which raises MetricComputationError on
that response. This PR therefore does not worsen the escaped-key case; it leaves it on a
path that already resolves it. The exponent row is identical on all three trees, so it is
inherited rather than introduced; consuming the full number grammar would also extend the
digit span onto e1, which _weighted_score_sums cannot weight because it filters
candidates on isdecimal(). Fixing that means changing the weighting code, and it is only
reachable if the integer contract in _COT_SYSTEM_PROMPT ("SCORE VALUE MUST BE AN INTEGER")
is relaxed. Left out rather than widening this diff.

Corrections to my own previous round, found while measuring this one.

• The matches[0] row in the table above is retracted. Re-run at 0e48cce it kills nothing
  (13 passed): the value-agreement step overrides the positional default whenever any match
  agrees, so first-vs-last only survives on an unparseable reconstruction, and the test for
  that path used a single score key. The row was true of an earlier head and I carried it
  forward without re-measuring it.
• 670fdd61e closes the coverage hole that explanation describes: two score keys in a
  truncated stream, where last-wins is the only rule left standing. It passes on 0e48cce
  too, so it demonstrates missing coverage rather than a fixed defect.

Mutation table at 670fdd61e, 15 tests as the baseline. Each mutation was applied to
parser.py, the suite run, then the file restored byte-for-byte (sha256 e81072a313fb,
checked before and after). This table is superseded by the one at 3b056b5d9 below: the
code it mutates no longer exists.

The last row is why the fallback stays load-bearing: it takes the punctuation-split
degradation test and both unparseable-reconstruction tests, plus all four provider-path e2e
cases (sync and async, two parameterizations each).

Verification at 670fdd61e: PYTHONPATH=src pytest tests/unit/evaluation/metrics/llm_judges/g_eval/ → 15 passed; uvx [email protected] check and
format --check clean on both files; uvx [email protected] --config-file pyproject.toml --ignore-missing-imports clean on parser.py (the --ignore-missing-imports is local
only, this machine has no litellm stubs; CI runs without it, and tests/ is excluded from
mypy by .pre-commit-config.yaml). Source diff is 23 lines added / 5 removed in
parser.py; the rest is tests.

Follow-up round (3b056b5d9, the depth rule replaced by a structural walk).

petrotiurin reproduced two defects in the rule the previous round added, and supplied the
design and a reference implementation. Both reproduce here, measured before changing
anything.

The depth rule counted { and } characters in the text preceding each match, so braces
inside a JSON string value read as structure. With }}} in a reason the top-level key came
out at depth 1 and the nested one at depth 0, so "shallowest agreeing match" selected the
nested key. The same rule copied a growing prefix and ran two full .count() scans per
agreeing match, which is quadratic in the response length.

Timing on a rubric-shaped response that doubles each step (top-level score plus N criteria,
each {"name": ..., "score": 7}, four-character tokens):

The walk is one pass: _STRUCTURE_RE matches either a whole JSON string, escapes included
and an unterminated tail tolerated, or a single brace. Strings are consumed whole, so
neither a brace nor a "score" echo inside a reason can move the structure. The key is read
only at depth 1 and the walk stops when that object closes. Last assignment wins, so
duplicate top-level keys still resolve as json.loads does. Nothing is validated, so a
truncated reconstruction behaves exactly as before, which lets the json.loads agreement
check and its bare except go away. json stays imported: parser.py:111 still uses it.

Three tests added. The fifteen existing ones pass unmodified, and against 670fdd61e's
src/ the file gives three failures which are exactly the three new tests.

• test_braces_inside_a_string_do_not_shift_the_score_depth — fails pre-fix with
  assert [13] == [3].
• test_score_only_nested_falls_back_instead_of_scoring_from_it — pins the decision on the
  consequence below; fails pre-fix, which scored 0.6219349153822014 from the nested key.
• test_walk_stops_at_the_first_object_close — {"score": 7} and {"score": 3} located
  entry 9 pre-fix, entry 4 now.

The consequence flagged on the thread is taken, not absorbed. A response whose only
"score" is nested now returns None, falls through to the legacy offset and raises,
matching main and the text path. Otherwise the same response succeeds or fails purely on
whether the provider returned logprobs.

One shape where the walk is stricter than the rule it replaces. Prose carrying a complete
brace pair before the JSON, Use {this} format: {"score": 7}: the walk closes on the prose
brace and returns None, where the old rule reached the real key. On that input all three
trees still end in MetricComputationError, because parser.py:111 parses the message
content with a strict json.loads and rejects anything prose-wrapped before the located
score matters. So the narrowing changes no outcome today, and where it eventually shows up it
degrades to the legacy offset, which is main's behaviour. Raised on the thread rather than
silently widened; continuing past a close while span is still None is a one-line change
and is deliberately not made here.

A distinction Baz's report and the thread did not separate. Baz's literal example,
{"reason":"{", "score":7}, does not reproduce: it has one match, so the shallowest-of-
agreeing step has nothing to choose between and it locates correctly. main is what raises
there, because the fixed offset lands on the { token and the zero-mass degrade did not
exist before this PR. Only the }}} variant breaks the depth rule, since the comparison
needs a second match to pick wrongly.

Mutation table at 3b056b5d9, 18 tests as the baseline, each mutation restored
byte-for-byte with sha256 765c826e checked before and after:

No test pins the linear scan. A timing bound loose enough to survive a loaded runner would
not kill the quadratic version either, since 58 ms at 64KB passes anything sane, so that
measurement lives in this description rather than in the suite.

Follow-up, not touched here. parser.py:111 reads the reason with a strict json.loads
on the raw message content, while every other read goes through
parsing_helpers.extract_json_content_or_raise (:21). That is why a prose-wrapped or
markdown-fenced response fails on the logprob path and succeeds without logprobs. Verified,
and out of scope for this diff.

Verification at 3b056b5d9: 18 passed in
tests/unit/evaluation/metrics/llm_judges/g_eval/, 126 passed in
tests/unit/evaluation/metrics/llm_judges/; uvx [email protected] check and format --check
clean on both touched files; uvx [email protected] --config-file pyproject.toml --disable-error-code misc --ignore-missing-imports on parser.py → Success: no issues found in 1 source file. The commit is 36 lines added / 36 removed in parser.py plus 108
lines of tests. Full tests/unit at this head: 1 failed, 5161 passed, 3 skipped in 243.78s,
the failure being test_evaluation__levenshtein_ratio on a missing optional rapidfuzz in a
file outside the diff.