fix(eval): find the choice letter when CJK text touches it - #4774

Summary

CMMLU, JMMLU and KMMLU score a correct answer as wrong when the model states it as a sentence in the benchmark's language and nothing separates the letter from the surrounding text. BaseBenchmark._extract_mc_answer returns an empty string, so the question counts as wrong (an external-API run reports it as parse_error).

>>> from omlx.eval.cmmlu import CMMLUBenchmark
>>> from omlx.eval.jmmlu import JMMLUBenchmark
>>> from omlx.eval.kmmlu import KMMLUBenchmark
>>> CMMLUBenchmark().extract_answer("答案是B", {})
''
>>> JMMLUBenchmark().extract_answer("正解はBです。", {})
''
>>> KMMLUBenchmark().extract_answer("정답은 B입니다.", {})
''

These are ordinary ways to give an answer. Chinese and Japanese put no spaces between words, and Korean particles and the copula attach straight to the letter. The CMMLU authors' own scorer has patterns for exactly these shapes, such as 答案(选项)?(是|为)：? ?([ABCD]) and 故?选择?：? ?([ABCD]) (extract_choice in mp_utils.py).

Root cause

omlx/eval/base.py:137-143. The fallback that takes the last valid letter relies on \b:

all_matches = re.findall(
    r"\b([" + pattern_letters + r"])\b",
    response_upper,
)

In a str pattern \w covers every Unicode letter, so Han characters, kana and Hangul are word characters. There is no word boundary inside "是B" or "B입", and the letter is never matched. Step 1 needs the English "answer is" cue and step 3 only looks at the first character, so all three steps come back empty.

Fix

A new last step runs only when steps 1 to 3 find nothing. It repeats the step 2 search with re.ASCII, which limits word characters to ASCII letters, digits and _. A letter next to CJK text now counts, and a letter inside an ASCII word or number still does not ("维生素B12", vitamin B12, still extracts nothing).

The step goes after the existing ones on purpose: every response that extracts a letter today extracts the same letter afterwards. A letter found by the existing steps still beats a later one that touches CJK text. "答案：B。A选项错误" ("Answer: B. Option A is wrong") stays B, and "B是正确的，A是错误的" ("B is correct, A is wrong") stays B through the first-character rule. Switching step 2 to ASCII boundaries outright would turn both into A.

All twelve benchmarks that call this helper get the change. Only responses that extract nothing today are affected, and those are always scored wrong today, so no score can go down.

Validation

Apple Silicon (arm64), macOS 26.7.1, Python 3.11.17. No model is needed; the change is in a pure text function.

tests/test_eval.py gains TestCJKMultipleChoice: seven responses whose letter touches Chinese, Japanese or Korean text, and three that pin the existing precedence. On main (d762927) with only the tests added:

$ python -m pytest tests/test_eval.py -k TestCJKMultipleChoice -q
E   AssertionError: assert '' == 'B'    (3 cases)
E   AssertionError: assert '' == 'C'    (2 cases)
E   AssertionError: assert '' == 'D'    (2 cases)
7 failed, 3 passed

With the change: 10 passed.

Adjacent suites: tests/test_eval.py gives 101 passed, and tests/test_eval_worker_pool.py with tests/test_mbpp_extract_code.py gives 46 passed. TestLocalStopReason and TestBenchmarkRegistry in tests/test_eval.py import modules that load MLX and could not run in my environment; neither passes CJK text to the extractor.

I also ran the old and new extractor side by side on 300,000 random strings mixing CJK, ASCII letters, digits and punctuation, and on 647,753 question and choice texts from all twelve bundled multiple-choice datasets. No input that extracted a letter before extracts a different one after.

ruff check and black --check report the same findings on both touched files before and after the change, so it adds none.

Checklist

• I read CONTRIBUTING.md.
• I understand every change in this PR and can explain it in review.
• For UI or UX changes, I attached before and after screenshots.

🤖 Generated with Claude Code