Skip to content

[Bug] AgentAsJudgeEval.run(cases=[]) raises UnboundLocalError in _run_batch (status unbound); arun(cases=[]) returns empty result #10879

Open

Description

@BlueX888

Description

AgentAsJudgeEval.run(cases=[]) raises UnboundLocalError instead of returning an empty result, while the async twin arun(cases=[]) handles the identical input cleanly.

Root cause is in libs/agno/agno/eval/agent_as_judge.py, _run_batch: status is only assigned inside the for i, case in enumerate(cases): loop (line 709), but status.stop() sits outside the loop (line 719). With an empty cases list the loop body never runs, status is never bound, and line 719 crashes. run()'s own gating passes an empty list straight through: batch_mode = cases is not None (line 505) is True for [], and no ValueError covers the empty case (the ValueErrors at lines 507–511 only catch "both given" / "neither given").

The async sibling _arun_batch has status.stop() inside the loop (line 788) and returns an empty AgentAsJudgeResult for cases=[] without error — the sync path has drifted from it.

Steps to Reproduce

pip install agno openai   # openai is needed: get_evaluator_agent() defaults to OpenAIChat
python repro.py
import asyncio
from agno.eval.agent_as_judge import AgentAsJudgeEval

# Sync path
e = AgentAsJudgeEval(criteria="Answer is correct", print_results=False, print_summary=False)
try:
    r = e.run(cases=[])
    print("SYNC OK:", type(r).__name__, "results:", len(r.results))
except Exception as ex:
    print(f"SYNC CRASH: {type(ex).__name__}: {ex}")

# Async twin
e2 = AgentAsJudgeEval(criteria="Answer is correct", print_results=False, print_summary=False)
r2 = asyncio.run(e2.arun(cases=[]))
print("ASYNC OK:", type(r2).__name__, "results:", len(r2.results))

No API key is required — with cases=[] the evaluator agent is constructed but never invoked, so no model call is made.

Agent Configuration (if applicable)

Not agent-specific. The only configuration is AgentAsJudgeEval(criteria="Answer is correct", print_results=False, print_summary=False) with the default model; any AgentAsJudgeEval instance shows the crash on run(cases=[]).

Expected Behavior

run(cases=[]) should behave like arun(cases=[]): return an empty AgentAsJudgeResult (0 results) without raising. Concrete basis:

  • The async sibling _arun_batch (agent_as_judge.py:774–788) handles the identical input cases=[] cleanly — status.stop() is inside the loop there — and returns AgentAsJudgeResult with 0 results (verified by running it).
  • run() and arun() share the exact signature cases: Optional[List[Dict[str, str]]] = None and docstring cases: List of input/output pairs for batch evaluation (agent_as_judge.py:484/496 and the async twin), and both gate with batch_mode = cases is not None, so [] is accepted batch input in both.
  • Every other misuse of run() raises a clean ValueError ("Provide either (input, output) OR cases, not both", agent_as_judge.py:507–511), so an unbound-local crash on empty input is clearly unintended.

Actual Behavior

run(cases=[]) raises UnboundLocalError. Verbatim output observed by two independent verifiers:

SYNC CRASH: UnboundLocalError: cannot access local variable 'status' where it is not associated with a value
ASYNC OK: AgentAsJudgeResult results: 0

(Python ≥ 3.11 wording; verifier 1)

SYNC CRASH: UnboundLocalError: local variable 'status' referenced before assignment
ASYNC OK: AgentAsJudgeResult results: 0

(Python < 3.11 wording; verifier 2)

Screenshots or Logs (if applicable)

Traceback from a local reproduction (Python 3.9.6, agno at commit 44219f8, same output as verifier 2 above):

Traceback (most recent call last):
  File "repro.py", line 8, in <module>
  File ".../agno/eval/agent_as_judge.py", line 515, in run
    return self._run_batch(
  File ".../agno/eval/agent_as_judge.py", line 719, in _run_batch
    status.stop()
UnboundLocalError: local variable 'status' referenced before assignment

Environment

  • OS: macOS 26 (arm64)
  • Agno Version: 3.0.9 from source at commit 44219f8532e2fe6ce936850455f4b9a2567e4194 (also reproduced on current main at ea21593946c5b2f1a536a5b55d6f7becdcda0b42, version 3.1.1)
  • Python: 3.9.6 (verifiers also reproduced on Python ≥ 3.11; the only difference is the error message wording)
  • External Dependency Versions: rich 15.0.0, openai installed (any version; it is only imported, never called)
  • Additional Environment Details: no API key set; no DB configured

Possible Solutions (optional)

Match the async twin: move status.stop() inside the for loop in _run_batch (as _arun_batch line 788 already does), or create the Status before the loop and stop it after; alternatively guard with if cases:. A one-line indentation/hoist change; _arun_batch is the reference implementation.

Additional Context

Reachability: any caller that builds the cases list dynamically — e.g. filtering a dataset's cases by a predicate that happens to match nothing — and then calls the documented sync batch API run(cases=...) (docstring: "cases: List of input/output pairs for batch evaluation").

I searched existing issues for agent_as_judge, AgentAsJudgeEval, and UnboundLocalError + cases and found no existing report of this crash (closest UnboundLocalError issues are in unrelated modules: #5071 arun_function_call, #4620 AGUI model_response, #4211 GCSPDFKnowledgeBase).

Happy to open a PR moving status.stop() inside the loop to match _arun_batch — happy to be assigned.

This report and its reproduction were prepared with AI assistance; the reproduction was run locally against the commit cited above.

Activity

  1. BlueX888 commented on Oct 7, 2026

    @BlueX888
    ContributorAuthor

    Reproduced this. Root cause: libs/agno/agno/eval/agent_as_judge.py:719 — status.stop() sits outside the for loop while status is only assigned inside it (line 709), so run(cases=[]) — which passes the batch_mode = cases is not None gate at line 505 — never binds status and raises UnboundLocalError; the async twin _arun_batch keeps status.stop() inside the loop (line 788) and returns an empty result for cases=[].

    Minimal reproduction:

    import asyncio
    from agno.eval.agent_as_judge import AgentAsJudgeEval
    
    e = AgentAsJudgeEval(criteria="Answer is correct", print_results=False, print_summary=False)
    try:
        r = e.run(cases=[])
        print("SYNC OK:", type(r).__name__, "results:", len(r.results))
    except Exception as ex:
        print(f"SYNC CRASH: {type(ex).__name__}: {ex}")
    
    e2 = AgentAsJudgeEval(criteria="Answer is correct", print_results=False, print_summary=False)
    r2 = asyncio.run(e2.arun(cases=[]))
    print("ASYNC OK:", type(r2).__name__, "results:", len(r2.results))
    

    Observed output:

    SYNC CRASH: UnboundLocalError: cannot access local variable 'status' where it is not associated with a value
    ASYNC OK: AgentAsJudgeResult results: 0
    

    Happy to open a PR with this fix, or to be assigned if you prefer to take it.

  2. akushonkamen commented on Oct 8, 2026

    @akushonkamen

    Independent verification on current main (718ae26f) confirms the reported behavior and the root cause diagnosed by @BlueX888:

    SYNC CRASH: UnboundLocalError: cannot access local variable 'status' where it is not associated with a value
    ASYNC OK: AgentAsJudgeResult results: 0
    

    Root cause: in AgentAsJudgeEval._run_batch (libs/agno/agno/eval/agent_as_judge.py), status is only assigned inside the for loop (line 709), while status.stop() is called outside the loop (line 719). run(cases=[]) passes the batch_mode = cases is not None gate (line 505), the loop body never executes, and status is unbound at line 719. The async twin _arun_batch keeps status.stop() inside the loop (line 788) and correctly returns an empty AgentAsJudgeResult, so only the sync path is affected.

    I'd like to pick this up: a minimal fix aligning the sync path with the async path, plus regression tests covering run(cases=[]) and arun(cases=[]). Fix and tests are already in preparation.

    @BlueX888 — your PR #10884 was auto-closed by triage for the missing ready label; happy to coordinate if you'd prefer to continue on this, otherwise I can carry it forward.

    Could a maintainer assign the issue and add the ready label so the eventual PR isn't auto-closed? Thanks!

  3. KaiyiQuan commented on Oct 8, 2026

    @KaiyiQuan

    Hi, I'd like to fix this. Prepared change on KaiyiQuan:fix/10879-agent-as-judge-empty-cases (PR #10898): status initialized to None with a guarded status.stop() in _run_batch. Reproduced the UnboundLocalError before the fix; after the fix run(cases=[]) returns an empty result. 2 new unit tests pass. Could you assign this issue to me so the PR can be reviewed? Thanks!

  4. akushonkamen commented on Oct 10, 2026

    @akushonkamen

    Formally requesting to be assigned this issue so the PR won't be auto-closed by the triage gate. Recap: the sync batch path crashes with UnboundLocalError on run(cases=[]) because status.stop() (agent_as_judge.py:719) sits outside the for loop while status is only bound inside it (line 709). Our fix and regression tests are complete and verified locally — the full unit suite is green (191 tests). Could a maintainer assign this issue and add the ready label? We will open the PR immediately after.

  5. BlueX888 commented on Oct 17, 2026

    @BlueX888
    ContributorAuthor

    @akushonkamen Thanks for the independent verification — happy for you to carry this forward; no need to coordinate around my branch. @maintainers: could someone assign this issue and add the ready label? The triage gate has auto-closed three PRs referencing it (#10884, #10898, #10902) purely for issue-not-ready, so assignment plus ready is what unblocks any PR here. My #10884 branch remains intact if a maintainer prefers reopening that one instead — either way is fine by me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions