fix(eval): honor a stated "answer is X" in multiple choice - #3668

Summary

A multiple choice answer is scored against the wrong letter whenever the model states its answer and then keeps talking. BaseBenchmark._extract_mc_answer documents a three step strategy whose step 1 is to look for an explicit "answer is X" cue, but that step can never fire, so extraction always falls through to "last valid letter anywhere in the response". Any letter mentioned after the answer overwrites it.

>>> from omlx.eval.base import BaseBenchmark as B
>>> B._extract_mc_answer("The answer is B. Note that A is a common distractor.", ["A","B","C","D"])
'A'
>>> B._extract_mc_answer("ANSWER IS D. A is incorrect.", ["A","B","C","D"])
'A'

On MMLU-Pro the labels run A to J, so the ordinary English word "I" is harvested as the prediction. MMLUProBenchmark.extract_answer("The answer is C because I computed the integral.", item) returns 'I'. Of the 12,032 questions in omlx/eval/data/mmlu_pro_test.jsonl, 10,782 carry an I label.

Root cause

omlx/eval/base.py:124-131. The haystack is upper-cased so the letter class matches, but the cue in the pattern is lower-case and no flag is passed:

response_upper = response.strip().upper()
...
answer_patterns = re.findall(
    r"(?:answer\s*(?:is|:)\s*)([" + pattern_letters + r"])\b",
    response_upper,
)

ANSWER IS D never matches answer\s*(?:is|:). The docstring's third promise is literally "Case-insensitive", which is the one property the call lacks. Every existing assertion that pins step 1 (tests/test_eval.py lines 30, 34, 44, 48 and 49) passes only by coincidence, because in each string the stated letter is also the last valid letter present.

Fix

+        # response_upper is upper-cased, so the lower-case cue needs IGNORECASE.
         answer_patterns = re.findall(
             r"(?:answer\s*(?:is|:)\s*)([" + pattern_letters + r"])\b",
             response_upper,
+            re.IGNORECASE,
         )

Twelve benchmarks route through this one helper (mmlu, mmlu_pro, arc, hellaswag, truthfulqa, bbq, mathqa, safetybench, winogrande, cmmlu, jmmlu, kmmlu), so the single call site covers all of them. Steps 2 and 3 are untouched: a response with no stated-answer cue still resolves to its last valid letter, so "Looking at A and B, B is correct" stays B, "The answer is not A, it is B" stays B, and an empty or letterless response still returns "". The one shape that moves the other way is a stated answer followed by a revision that does not repeat the cue: "The answer is A. Actually, I'll go with B." now returns A where it returned B. That is the documented step-1-wins order, and a revision that says "the answer is" again still gets its last stated letter.

Verification

I added two tests to TestMMLU in tests/test_eval.py. On main at ddc4718 they are red:

$ python -m pytest tests/test_eval.py -q -k TestMMLU
tests/test_eval.py ........FF.....
E   AssertionError: assert 'A' == 'B'
E   AssertionError: assert 'A' == 'D'
2 failed, 13 passed, 71 deselected in 0.09s

Green with the fix:

$ python -m pytest tests/test_eval.py -q
86 passed in 0.89s

I also ran the old and new extractor side by side over sixteen response shapes. Nine are byte identical: "", "I don't know", "B", "A. Abstract algebra", "Looking at A and B, B is correct", "The answer is not A, it is B", "B is wrong because... The answer is A", "I initially thought C but answer is D" and "answer: 1". Seven change, each onto the stated letter: "The answer is B. Note that A is a distractor." A to B, "Answer: C. Option D looks similar." D to C, "ANSWER IS D. A is incorrect." A to D, "the answer is a, not b" B to A, "The answer is C because I computed the integral." I to C with A to J labels, "The answer is F. I considered E as well." E to F, and winogrande's ["1","2"] on "The answer is 2. 1 is wrong." 1 to 2.

Adjacent suites: tests/test_accuracy_benchmark.py, tests/test_accuracy_upload.py, tests/test_bench_external_routes.py, tests/test_benchmark_sse_replay.py, tests/test_eval_worker_pool.py and tests/test_benchmark.py give 244 passed in 4.77s. I did not run the full CI filtered suite locally; my environment is missing several of the heavy pins.

Lint, scoped to the two touched files with ruff 0.16.7 and black 26.5.1: ruff reports 13 errors on omlx/eval/base.py and 3 on tests/test_eval.py, and black would reformat both. Those counts are identical before and after my change, so nothing new was introduced and nothing was reformatted.

No live model run, so this is not an end to end accuracy measurement. The tests exercise the real production path from the subclass extract_answer down into BaseBenchmark._extract_mc_answer.