Skip to content

fix(reward): follow the GRPO reward contract in MathORM - #10266

Open
Lesereingrape wants to merge 1 commit into
modelscope:mainfrom
Lesereingrape:fix/math-orm-grpo-reward-contract
Open

Lesereingrape wants to merge 1 commit into
modelscope:mainfrom
Lesereingrape:fix/math-orm-grpo-reward-contract

Conversation

@Lesereingrape

Copy link
Copy Markdown
Contributor

What is changed / 改动内容

MathORM.__call__ now follows the reward-function contract that the GRPO dispatch uses, like every other
built-in ORM: it receives the completion texts and the dataset solution column, and still unwraps
InferRequest objects for programmatic callers (the same isinstance(..., str) branch ReactORM uses).

Why / 原因

--reward_funcs math crashed as soon as rewards were computed, because MathORM was the only registered ORM
left assuming List[InferRequest] while swift/rl_core/grpo_algorithm.py::compute_rewards_per_func passes
completions: List[str] positionally plus the dataset columns as kwargs:

  • solution column (what the math datasets and the reward function guide document) → TypeError: MathORM.__call__() missing 1 required positional argument: 'ground_truths'
  • a ground_truths column → AttributeError: 'str' object has no attribute 'messages' at swift/rewards/orm.py:444

Closes #10265

Test / 测试

  • New TestMathORMCompletions cases in tests/utils/test_rewards.py (completion strings, solution kwarg,
    InferRequest): the module runs FAILED (errors=2) on dbc395ff without the fix and OK with it.
  • End to end through the real dispatch (compute_rewards_per_func) with a solution column: crash before the
    change, [[1.0], [0.0]] after it for \boxed{3} vs 4 against \boxed{3}.
  • tests/utils/test_math_orm_expressions.py, which calls MathORM with InferRequest objects, keeps passing.
  • flake8, yapf and isort clean on both files.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] --reward_funcs math crashes: MathORM still expects InferRequest objects while the GRPO reward dispatch passes completion strings

1 participant