Fix UnboundLocalError in reward.py: compute_score cannot compute any reward - #7

Problem

compute_score() in recipe/deepresearch/reward.py — the reward function the released RL recipe is configured to use — raises UnboundLocalError on every call, before any scoring happens. On a clean checkout of main (962e10b), no reward can be computed for any task type.

Root cause

training_scripts/rl/recipe/deepresearch/reward.py, compute_score(), lines 2103–2114.

Two open-ended-eval locals are initialised from themselves:

# line 2103-2106
openended_eval_llm_address = kwargs.pop(
    "openended_eval_llm_address",
    kwargs.pop("openended_eval_llm_addresses", openended_eval_llm_addresses),  # <-- read here
)
...
# line 2109-2114
openended_eval_llm_model = str(
    kwargs.pop(
        "openended_eval_llm_model",
        os.environ.get("OPENENDED_EVAL_LLM_MODEL_NAME", openended_eval_llm_model),  # <-- read here
    )
).strip() or openended_eval_llm_model                                              # <-- and here

Neither openended_eval_llm_addresses nor openended_eval_llm_model is a parameter of compute_score (its signature has no openended_* params), and both are assigned later in the body — line 2182 and line 2109 respectively. Assignment anywhere in a function body makes a name local for the whole body, so these reads hit an unbound local.

Python evaluates a call's arguments before the call, so the default expressions in dict.pop(...) and os.environ.get(...) are evaluated eagerly whether or not the key is present. That makes the first two reads unconditional:

• 2105 always raises. No value in kwargs can avoid it.
• 2112 always raises once 2105 is fixed — setting OPENENDED_EVAL_LLM_MODEL_NAME does not help, because the default argument is still evaluated.
• 2114 raises whenever the resolved model string is empty. That is the documented launcher's default: run_deepresearch_fully_async_megatron.sh:794 exports OPENENDED_EVAL_LLM_MODEL_NAME="${OPENENDED_EVAL_LLM_MODEL_NAME:-}", so str("").strip() is falsy and the or branch is taken.

All three therefore need fixing, and they sit above the if not extracted_answer early exit at 2131 and above every scoring branch, so nothing downstream is ever reached.

pyflakes flags exactly these three reads:

reward.py:2105:52: undefined name 'openended_eval_llm_addresses'
reward.py:2112:61: undefined name 'openended_eval_llm_model'
reward.py:2114:18: undefined name 'openended_eval_llm_model'

Impact

Affected — the entire RL reward path. compute_score is the configured reward function: config/deepresearch_trainer.yaml sets custom_reward_function.name: compute_score (line 52) and path: recipe/deepresearch/reward.py (line 50), and run_deepresearch_fully_async_megatron.sh:1188 sets the same path at launch. Because the failing read precedes all routing, every task type dies identically — objective default scoring, objective scoring via the 864 per-task scripts in eval_scripts/, open-ended rubric scoring, BrowseComp and HLE — as do both public wrappers re-exported from __init__.py. 6/6 entry-point invocations raise (see below). Consequently the inline-citation component and the base_score * base_score_weight + citation_score_added combination at line 2317 never execute, and the fallback ladder at 2182–2189 is dead code today.

With reward_loop_class_name=DeepResearchRewardLoopManager (set at run_deepresearch_fully_async_megatron.sh:1199), reward_loop_manager.py:142 awaits compute_score inside a try that catches only asyncio.TimeoutError, so the UnboundLocalError propagates out of the reward loop rather than being caught.

Unaffected. compute_score_sync() (line 1402) does not contain this pattern and is fine. _compute_base_score() is fine — it takes both names as ordinary required parameters (lines 1820–1821) and is called correctly at 2214–2215. Nothing under evaluation/ is affected: the offline benchmark graders are separate code and do not import reward.py. Inference (inference/) is unaffected.

Present since the initial release: git blame puts all three lines in 002b75d (2026-05-20); the two later commits to this file (9ac3b3e, ac4422e) did not touch this region. Nothing in the repo exercises this function: no test imports recipe/deepresearch/reward.py, and the other custom_reward_function.name=compute_score occurrences under recipe/ (genrm_remote, r1_ascend, spo, retool) each point custom_reward_function.path at their own reward file, not at this one. run_deepresearch_fully_async_megatron.sh:1188 is the only place that path is recipe/deepresearch/reward.py.

Fix

Three tokens, restoring the defaults the surrounding code already expects. This is the minimal change that makes both reads legal:

• Line 2105 → None. The fallback ladder at 2182–2189 is written to receive None: it assigns openended_eval_llm_addresses = openended_eval_llm_address, optionally regenerates from IPs/ports, then does if openended_eval_llm_addresses is None: openended_eval_llm_addresses = eval_llm_addresses. None is the value that makes that ladder behave as documented ("falls back to objective addresses", line 2181).
• Lines 2112 and 2114 → eval_llm_model. The structurally identical block 17 lines above, inline_citation_eval_llm_model (2092–2097), uses eval_llm_model in both the env-default position and the or fallback. This block appears to be a copy of it with one rename too many.

No behaviour change beyond making the function runnable: when a caller supplies openended_eval_llm_address/openended_eval_llm_model (or the env var), the resolved values are exactly what the existing code intended.

Verification

Reproduction script (repro_reward_unboundlocal.py, attached below) imports the unmodified reward.py from a clean checkout and calls each entry point with a rollout whose answer exactly matches the ground truth. reward.py imports no verl/torch/ray, so this needs only aiohttp + stdlib and no GPU, cluster or API key.

Before (clean checkout of main @ 962e10b):

  RAISED objective task, default scoring    -> UnboundLocalError: local variable 'openended_eval_llm_addresses' referenced before assignment
  RAISED browsecomp task                    -> UnboundLocalError: local variable 'openended_eval_llm_addresses' referenced before assignment
  RAISED hle task                           -> UnboundLocalError: local variable 'openended_eval_llm_addresses' referenced before assignment
  RAISED open-ended task                    -> UnboundLocalError: local variable 'openended_eval_llm_addresses' referenced before assignment
  RAISED compute_score_wrapper              -> UnboundLocalError: local variable 'openended_eval_llm_addresses' referenced before assignment
  RAISED compute_score_wrapper_sync         -> UnboundLocalError: local variable 'openended_eval_llm_addresses' referenced before assignment

6/6 invocations raised before any scoring ran.

After this patch:

  ok     objective task, default scoring    -> score=0.75
  ok     browsecomp task                    -> score=0.0
  ok     hle task                           -> score=0.0
  ok     open-ended task                    -> score=0.75
  ok     compute_score_wrapper              -> score=0.75
  ok     compute_score_wrapper_sync

0/6 invocations raised before any scoring ran.

0.75 is base_score 1.0 * base_score_weight 0.75 with the citation component at 0.0, i.e. the full path now runs and returns a real score. BrowseComp and HLE return 0.0 with error: browsecomp_missing_api_key / hle_missing_judge_model because this environment has no judge credentials — that is the expected diagnostic, and it confirms routing now reaches those branches instead of crashing.

Patching only line 2105 leaves the second bug live, which is why both are in this patch:

  RAISED objective task, default scoring    -> UnboundLocalError: local variable 'openended_eval_llm_model' referenced before assignment

python -m pyflakes training_scripts/rl/recipe/deepresearch/reward.py reports the three undefined name errors above before the patch and none after it.

#!/usr/bin/env python3
"""Reproduction for the UnboundLocalError in recipe/deepresearch/reward.py::compute_score.
Requires only aiohttp + stdlib. Exits 0 if compute_score works, 1 if it raises."""
import asyncio, importlib.util, os, sys

HERE = os.path.dirname(os.path.abspath(__file__))
RECIPE = os.path.join(HERE, "training_scripts", "rl", "recipe", "deepresearch")
if not os.path.isdir(RECIPE):
    sys.exit(f"run me from the repo root (looked for {RECIPE})")
sys.path.insert(0, os.path.join(HERE, "training_scripts", "rl"))
sys.path.insert(0, RECIPE)

spec = importlib.util.spec_from_file_location("quest_reward", os.path.join(RECIPE, "reward.py"))
reward = importlib.util.module_from_spec(spec)
spec.loader.exec_module(reward)

SOLUTION = "<think>reasoning</think>\n<answer>Paris</answer>"
GT = {"answer": "Paris"}
CASES = [("objective task, default scoring", "quest_obj"), ("browsecomp task", "browsecomp"),
         ("hle task", "hle"), ("open-ended task", "openended")]

async def main():
    failures = 0
    for label, ds in CASES:
        try:
            r = await reward.compute_score(data_source=ds, solution_str=SOLUTION, ground_truth=GT, extra_info={})
            print(f"  ok     {label:34s} -> score={r.get('score')}")
        except Exception as e:
            failures += 1
            print(f"  RAISED {label:34s} -> {type(e).__name__}: {e}")
    try:
        r = await reward.compute_score_wrapper(data_source="quest_obj", solution_str=SOLUTION, ground_truth=GT, extra_info={})
        print(f"  ok     {'compute_score_wrapper':34s} -> score={r.get('score')}")
    except Exception as e:
        failures += 1
        print(f"  RAISED {'compute_score_wrapper':34s} -> {type(e).__name__}: {e}")
    return failures

failures = asyncio.run(main())
try:
    reward.compute_score_wrapper_sync(data_source="quest_obj", solution_str=SOLUTION, ground_truth=GT, extra_info={})
    print(f"  ok     {'compute_score_wrapper_sync':34s}")
except Exception as e:
    failures += 1
    print(f"  RAISED {'compute_score_wrapper_sync':34s} -> {type(e).__name__}: {e}")

print(f"\n{failures}/6 invocations raised before any scoring ran.")
sys.exit(1 if failures else 0)