Repository navigation
Description
Description
AgentAsJudgeEval.run(cases=[]) raises UnboundLocalError instead of returning an empty result, while the async twin arun(cases=[]) handles the identical input cleanly.
Root cause is in libs/agno/agno/eval/agent_as_judge.py, _run_batch: status is only assigned inside the for i, case in enumerate(cases): loop (line 709), but status.stop() sits outside the loop (line 719). With an empty cases list the loop body never runs, status is never bound, and line 719 crashes. run()'s own gating passes an empty list straight through: batch_mode = cases is not None (line 505) is True for [], and no ValueError covers the empty case (the ValueErrors at lines 507–511 only catch "both given" / "neither given").
The async sibling _arun_batch has status.stop() inside the loop (line 788) and returns an empty AgentAsJudgeResult for cases=[] without error — the sync path has drifted from it.
Steps to Reproduce
pip install agno openai # openai is needed: get_evaluator_agent() defaults to OpenAIChat
python repro.pyimport asyncio
from agno.eval.agent_as_judge import AgentAsJudgeEval
# Sync path
e = AgentAsJudgeEval(criteria="Answer is correct", print_results=False, print_summary=False)
try:
r = e.run(cases=[])
print("SYNC OK:", type(r).__name__, "results:", len(r.results))
except Exception as ex:
print(f"SYNC CRASH: {type(ex).__name__}: {ex}")
# Async twin
e2 = AgentAsJudgeEval(criteria="Answer is correct", print_results=False, print_summary=False)
r2 = asyncio.run(e2.arun(cases=[]))
print("ASYNC OK:", type(r2).__name__, "results:", len(r2.results))No API key is required — with cases=[] the evaluator agent is constructed but never invoked, so no model call is made.
Agent Configuration (if applicable)
Not agent-specific. The only configuration is AgentAsJudgeEval(criteria="Answer is correct", print_results=False, print_summary=False) with the default model; any AgentAsJudgeEval instance shows the crash on run(cases=[]).
Expected Behavior
run(cases=[]) should behave like arun(cases=[]): return an empty AgentAsJudgeResult (0 results) without raising. Concrete basis:
- The async sibling
_arun_batch(agent_as_judge.py:774–788) handles the identical inputcases=[]cleanly —status.stop()is inside the loop there — and returnsAgentAsJudgeResultwith 0 results (verified by running it). run()andarun()share the exact signaturecases: Optional[List[Dict[str, str]]] = Noneand docstringcases: List of input/output pairs for batch evaluation(agent_as_judge.py:484/496 and the async twin), and both gate withbatch_mode = cases is not None, so[]is accepted batch input in both.- Every other misuse of
run()raises a cleanValueError("Provide either (input, output) OR cases, not both", agent_as_judge.py:507–511), so an unbound-local crash on empty input is clearly unintended.
Actual Behavior
run(cases=[]) raises UnboundLocalError. Verbatim output observed by two independent verifiers:
SYNC CRASH: UnboundLocalError: cannot access local variable 'status' where it is not associated with a value
ASYNC OK: AgentAsJudgeResult results: 0
(Python ≥ 3.11 wording; verifier 1)
SYNC CRASH: UnboundLocalError: local variable 'status' referenced before assignment
ASYNC OK: AgentAsJudgeResult results: 0
(Python < 3.11 wording; verifier 2)
Screenshots or Logs (if applicable)
Traceback from a local reproduction (Python 3.9.6, agno at commit 44219f8, same output as verifier 2 above):
Traceback (most recent call last):
File "repro.py", line 8, in <module>
File ".../agno/eval/agent_as_judge.py", line 515, in run
return self._run_batch(
File ".../agno/eval/agent_as_judge.py", line 719, in _run_batch
status.stop()
UnboundLocalError: local variable 'status' referenced before assignment
Environment
- OS: macOS 26 (arm64)
- Agno Version: 3.0.9 from source at commit
44219f8532e2fe6ce936850455f4b9a2567e4194(also reproduced on currentmainatea21593946c5b2f1a536a5b55d6f7becdcda0b42, version 3.1.1) - Python: 3.9.6 (verifiers also reproduced on Python ≥ 3.11; the only difference is the error message wording)
- External Dependency Versions: rich 15.0.0, openai installed (any version; it is only imported, never called)
- Additional Environment Details: no API key set; no DB configured
Possible Solutions (optional)
Match the async twin: move status.stop() inside the for loop in _run_batch (as _arun_batch line 788 already does), or create the Status before the loop and stop it after; alternatively guard with if cases:. A one-line indentation/hoist change; _arun_batch is the reference implementation.
Additional Context
Reachability: any caller that builds the cases list dynamically — e.g. filtering a dataset's cases by a predicate that happens to match nothing — and then calls the documented sync batch API run(cases=...) (docstring: "cases: List of input/output pairs for batch evaluation").
I searched existing issues for agent_as_judge, AgentAsJudgeEval, and UnboundLocalError + cases and found no existing report of this crash (closest UnboundLocalError issues are in unrelated modules: #5071 arun_function_call, #4620 AGUI model_response, #4211 GCSPDFKnowledgeBase).
Happy to open a PR moving status.stop() inside the loop to match _arun_batch — happy to be assigned.
This report and its reproduction were prepared with AI assistance; the reproduction was run locally against the commit cited above.
Activity
BlueX888 commentedon Oct 7, 2026
Reproduced this. Root cause: libs/agno/agno/eval/agent_as_judge.py:719 — status.stop() sits outside the for loop while status is only assigned inside it (line 709), so run(cases=[]) — which passes the batch_mode = cases is not None gate at line 505 — never binds status and raises UnboundLocalError; the async twin _arun_batch keeps status.stop() inside the loop (line 788) and returns an empty result for cases=[].
Minimal reproduction:
Observed output:
Happy to open a PR with this fix, or to be assigned if you prefer to take it.
akushonkamen commentedon Oct 8, 2026
Independent verification on current
main(718ae26f) confirms the reported behavior and the root cause diagnosed by @BlueX888:Root cause: in
AgentAsJudgeEval._run_batch(libs/agno/agno/eval/agent_as_judge.py),statusis only assigned inside theforloop (line 709), whilestatus.stop()is called outside the loop (line 719).run(cases=[])passes thebatch_mode = cases is not Nonegate (line 505), the loop body never executes, andstatusis unbound at line 719. The async twin_arun_batchkeepsstatus.stop()inside the loop (line 788) and correctly returns an emptyAgentAsJudgeResult, so only the sync path is affected.I'd like to pick this up: a minimal fix aligning the sync path with the async path, plus regression tests covering
run(cases=[])andarun(cases=[]). Fix and tests are already in preparation.@BlueX888 — your PR #10884 was auto-closed by triage for the missing
readylabel; happy to coordinate if you'd prefer to continue on this, otherwise I can carry it forward.Could a maintainer assign the issue and add the
readylabel so the eventual PR isn't auto-closed? Thanks!KaiyiQuan commentedon Oct 8, 2026
Hi, I'd like to fix this. Prepared change on
KaiyiQuan:fix/10879-agent-as-judge-empty-cases(PR #10898):statusinitialized toNonewith a guardedstatus.stop()in_run_batch. Reproduced the UnboundLocalError before the fix; after the fixrun(cases=[])returns an empty result. 2 new unit tests pass. Could you assign this issue to me so the PR can be reviewed? Thanks!akushonkamen commentedon Oct 10, 2026
Formally requesting to be assigned this issue so the PR won't be auto-closed by the triage gate. Recap: the sync batch path crashes with
UnboundLocalErroronrun(cases=[])becausestatus.stop()(agent_as_judge.py:719) sits outside theforloop whilestatusis only bound inside it (line 709). Our fix and regression tests are complete and verified locally — the full unit suite is green (191 tests). Could a maintainer assign this issue and add thereadylabel? We will open the PR immediately after.BlueX888 commentedon Oct 17, 2026
@akushonkamen Thanks for the independent verification — happy for you to carry this forward; no need to coordinate around my branch. @maintainers: could someone assign this issue and add the
readylabel? The triage gate has auto-closed three PRs referencing it (#10884, #10898, #10902) purely forissue-not-ready, so assignment plusreadyis what unblocks any PR here. My #10884 branch remains intact if a maintainer prefers reopening that one instead — either way is fine by me.