Repository navigation
Conversation
_generate_early_stopping_response read the memory before finalize ran, so the early stopping "generate" call only saw the original user message: the assistant and tool messages of the current run still lived in the agent's scratchpad, and the model was asked to summarize observations it never received. Finalize the agent before fetching the memory, matching what parse_agent_output already does when the last step made no tool call. The tool calls of the last step are never run once early stopping fires, so that message has no tool response to pair with. Drop it from the request, otherwise the prompt ends with an assistant tool_calls message that OpenAI/Anthropic-compatible APIs reject. Applied to both BaseWorkflowAgent and AgentWorkflow.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
_generate_early_stopping_responsebuilds the final "generate" call frommemory.aget(), but it never callsfinalizefirst.For the
FunctionAgent/AgentWorkflowflow, everything produced during a run lives in the agent's scratchpad untilfinalizeflushes it into memory (function_agent.py:180). The early stopping path read memory before that happened, so the final LLM call received only the original user message plus the early stopping prompt. The model was asked to summarize observations it had never been given, and the response it generated ignored everything the tools returned during the run.This was the one place that reads memory without finalizing.
parse_agent_outputalready does it the other way, with a comment spelling out why (base_agent.py:565):Root cause, pre-fix:
llama-index-core/llama_index/core/agent/workflow/base_agent.py:483—_generate_early_stopping_responsereadsmemoryat line 487,finalizeis never called on this path.llama-index-core/llama_index/core/agent/workflow/multi_agent_workflow.py:487— same shape inAgentWorkflow.Fixes # (no issue filed — the defect, root cause and fix are described in this PR)
Changes
llama-index-core/llama_index/core/agent/workflow/base_agent.py—finalizethe agent before reading memory, and thread theAgentOutputthrough_generate_early_stopping_responsesofinalizecan be called.MessageRoleadded to an existing import.llama-index-core/llama_index/core/agent/workflow/multi_agent_workflow.py— same call for the current agent.tool_calls. Those calls are never executed once early stopping fires (the max-iterations check returns before the tool-call branch), so that message has no tool results to pair with. Sending it would end the request with an assistanttool_callsmessage followed by the early stopping system prompt — a shape OpenAI and Anthropic-compatible endpoints reject with "An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'". Without the fix the request never contained tool calls at all, so this only becomes reachable once the tool results are included.New Package?
Version Bump?
Did you bump the version in the
pyproject.tomlfile of the package you are updating? (Except for thellama-index-corepackage)Type of Change
How Has This Been Tested?
test_early_stopping_method_generate_includes_tool_results(single agent) andtest_early_stopping_method_generate_includes_tool_results_multi_agentdrive aFunctionAgentwhose mock LLM always requests a tool, withmax_iterations=2andearly_stopping_method="generate". Each tool call gets its owntool_id, so the tests can assert both that the early stopping request contains the tool output, and that every tool call in it has a matching tool result.RED on the pre-fix revision (
git checkout HEAD~1 -- <the two source files>):GREEN with the fix:
I also dumped the exact payload of the early stopping call with a throwaway script to confirm the shape. Before the fix it was
[user, system(early stopping prompt)]; after it is[user, assistant(tool_calls=call_1), tool(call_1), system(early stopping prompt)].Note: in this checkout 13 tests in
tests/readers/file/test_base.pyfail because the partial clone is missingtests/readers/file/data. They fail identically with the change reverted (13 failed, 10 passed) and are unrelated to this PR.Suggested Checklist:
uv run make format; uv run make lintto appease the lint godsAI assistance disclosure
Per the Transparency principle in CONTRIBUTING.md: this change and its tests were written with AI assistance (Claude Code), and I reviewed the result before submitting. Verification: I reproduced the defect first and confirmed the new tests fail on the pre-fix revision and pass with the fix; I ran
tests/agent(92 passed, 3 skipped),ruff check/ruff format --check,mypy llama_indexascore-typecheck.ymlruns it, and the repo's pre-commit hooks on the changed files. I inspected the generated early stopping request directly rather than relying on the mock LLM's behavior alone.