fix: output_validator ctx.retry reflects global retry counter, not per-tool count#4527
Conversation
…t per-tool count
When an @agent.output_validator fails, the RunContext passed to it had
ctx.retry set to the per-tool retry counter via ToolManager's
_build_tool_execution_context:
retry=self.ctx.retries.get(call.tool_name, 0)
This is correct for function tools (each tool has its own budget), but
wrong for output validators, which share the global output-validation
budget (max_result_retries / output_retries).
Consequence: if any retry-consuming event occurs *before* the output
tool is called (empty model response, unknown tool call, text validation
failure), the global counter is already > 0 but the validator sees
ctx.retry = 0. Validators that use ctx.retry to adjust behaviour on the
last attempt (e.g. 'be lenient on final try') are blind to these
already-consumed retries.
Fix: immediately before calling tool_manager.execute_tool_call for an
output tool, replace validated.ctx with a copy whose retry/max_retries
reflect the global state (ctx.state.retries / ctx.deps.max_result_retries).
Add regression test test_output_validator_sees_global_retry_count that
fires an empty-response before the output tool call and asserts the
validator receives ctx.retry=1, not ctx.retry=0.
Fixes pydantic#4385
|
Thanks for your interest in this issue! However, there are already open PRs addressing issue #4385: #4325. To avoid duplicate efforts, this PR has been closed. If you'd like to contribute, you can review the existing PRs or share your thoughts on issue #4385. If you believe the existing PRs are inactive, please comment on the issue and a maintainer can reassess. |
github-actions
Bot
added
the
bug
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
| validated = replace( | ||
| validated, | ||
| ctx=replace(validated.ctx, retry=ctx.state.retries, max_retries=ctx.deps.max_result_retries), | ||
| ) |
There was a problem hiding this comment.
🚩 Dual retry budget for output tools: global counter and per-tool counter both track failures
Output tool retries are tracked by two independent mechanisms:
- Global counter (
ctx.state.retries/ctx.state.increment_retries): Managed in_agent_graph.py:1034whenToolRetryErroris caught. This enforces themax_result_retriesbudget. - Per-tool counter (
ToolManager.failed_tools+ToolManager.ctx.retriesdict): Managed in_tool_manager.py:359-360whenModelRetryis caught inside_execute_tool_call_impl. This enforcesvalidated.tool.max_retriesvia_check_max_retriesat_tool_manager.py:147-151.
Both counters use the same cap value (output_retries / max_result_retries), so they effectively race to the same limit. This PR's fix only modifies what validated.ctx.retry (scalar) exposes to output validators — it does NOT change either retry-counting mechanism. The dual tracking is pre-existing and works because both counters increment in lockstep for output-only retry scenarios. However, when non-output retry events (empty responses, unknown tools) increment the global counter without affecting the per-tool counter, the per-tool _check_max_retries could allow more attempts than max_result_retries intends. This is mitigated by the global increment_retries check in _agent_graph.py, which fires first.
Was this helpful? React with 👍 or 👎 to provide feedback.
1 participant
Problem
Fixes #4385.
@agent.output_validatorfunctions received actx.retrythat was set by theToolManager's_build_tool_execution_context:This is correct for function tools (each tool has its own retry budget) but wrong for output validators, which share the global output-validation budget (
output_retries/max_result_retries).The discrepancy means: if any retry-consuming event occurred before the output tool is called — empty model response, unknown tool call, text validation failure — the global counter is already
> 0but the validator still seesctx.retry = 0. Validators that usectx.retryto adjust behavior on the last attempt are blind to the already-consumed retry budget.The text-output path already does the right thing (line 821 of
_agent_graph.py):The tool-output path was missing this step.
Fix
Immediately before calling
tool_manager.execute_tool_callfor an output tool, replacevalidated.ctxwith a copy whoseretry/max_retriesreflect the global state:Test
Added
test_output_validator_sees_global_retry_countwhich:ModelResponse(parts=[]))— this consumes 1 global retryctx.retry == 1(global), notctx.retry == 0(per-tool)