Skip to content

fix(agent): include tool results in the early stopping response - #23113

Open
BlueX888 wants to merge 1 commit into
run-llama:mainfrom
BlueX888:fix/prep-early-stopping-generate-loses-tool-results
Open

BlueX888 wants to merge 1 commit into
run-llama:mainfrom
BlueX888:fix/prep-early-stopping-generate-loses-tool-results

Conversation

@BlueX888

Copy link
Copy Markdown

Description

_generate_early_stopping_response builds the final "generate" call from memory.aget(), but it never calls finalize first.

For the FunctionAgent / AgentWorkflow flow, everything produced during a run lives in the agent's scratchpad until finalize flushes it into memory (function_agent.py:180). The early stopping path read memory before that happened, so the final LLM call received only the original user message plus the early stopping prompt. The model was asked to summarize observations it had never been given, and the response it generated ignored everything the tools returned during the run.

This was the one place that reads memory without finalizing. parse_agent_output already does it the other way, with a comment spelling out why (base_agent.py:565):

# important: messages should always be fetched after calling finalize, otherwise they do not contain the agent's response
output = await self.finalize(ctx, ev, memory)
messages = await memory.aget()

Root cause, pre-fix:

  • llama-index-core/llama_index/core/agent/workflow/base_agent.py:483 — _generate_early_stopping_response reads memory at line 487, finalize is never called on this path.
  • llama-index-core/llama_index/core/agent/workflow/multi_agent_workflow.py:487 — same shape in AgentWorkflow.

Fixes # (no issue filed — the defect, root cause and fix are described in this PR)

Changes

  • llama-index-core/llama_index/core/agent/workflow/base_agent.py — finalize the agent before reading memory, and thread the AgentOutput through _generate_early_stopping_response so finalize can be called. MessageRole added to an existing import.
  • llama-index-core/llama_index/core/agent/workflow/multi_agent_workflow.py — same call for the current agent.
  • Both paths additionally drop the assistant message of the last step when it carries tool_calls. Those calls are never executed once early stopping fires (the max-iterations check returns before the tool-call branch), so that message has no tool results to pair with. Sending it would end the request with an assistant tool_calls message followed by the early stopping system prompt — a shape OpenAI and Anthropic-compatible endpoints reject with "An assistant message with 'tool_calls' must be followed by tool messages responding to each 'tool_call_id'". Without the fix the request never contained tool calls at all, so this only becomes reachable once the tool results are included.

New Package?

  • Yes
  • No

Version Bump?

Did you bump the version in the pyproject.toml file of the package you are updating? (Except for the llama-index-core package)

  • Yes
  • No

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

  • I added new unit tests to cover this change
  • I believe this change is already covered by existing unit tests

test_early_stopping_method_generate_includes_tool_results (single agent) and test_early_stopping_method_generate_includes_tool_results_multi_agent drive a FunctionAgent whose mock LLM always requests a tool, with max_iterations=2 and early_stopping_method="generate". Each tool call gets its own tool_id, so the tests can assert both that the early stopping request contains the tool output, and that every tool call in it has a matching tool result.

RED on the pre-fix revision (git checkout HEAD~1 -- <the two source files>):

$ uv run pytest tests/agent/workflow/test_single_agent_workflow.py::test_early_stopping_method_generate_includes_tool_results tests/agent/workflow/test_multi_agent_workflow.py::test_early_stopping_method_generate_includes_tool_results_multi_agent
E       AssertionError: the early stopping response was generated without the tool results
E       assert False
tests/agent/workflow/test_single_agent_workflow.py:401: AssertionError
tests/agent/workflow/test_multi_agent_workflow.py:571: AssertionError
FAILED tests/agent/workflow/test_single_agent_workflow.py::test_early_stopping_method_generate_includes_tool_results
FAILED tests/agent/workflow/test_multi_agent_workflow.py::test_early_stopping_method_generate_includes_tool_results_multi_agent
2 failed, 73 warnings in 0.14s

GREEN with the fix:

$ uv run pytest <the two tests above>                                   2 passed, 73 warnings in 0.12s
$ uv run pytest tests/agent/ -q                                          92 passed, 3 skipped, 1915 warnings in 0.46s
$ uv run ruff check llama_index/core/agent/workflow/ tests/agent/workflow/
All checks passed!
$ uv run ruff format --check llama_index/core/agent/workflow/ tests/agent/workflow/
19 files already formatted
$ uv run -- mypy llama_index                                            # as core-typecheck.yml runs it
Success: no issues found in 479 source files
$ uv run pre-commit run --files <the four changed files>                # scoped to the changed files; same hook set `make lint` runs
ruff, ruff-format, mypy, codespell, prettier, end-of-file-fixer, trailing-whitespace, mixed-line-ending, detect-private-key — all Passed

I also dumped the exact payload of the early stopping call with a throwaway script to confirm the shape. Before the fix it was [user, system(early stopping prompt)]; after it is [user, assistant(tool_calls=call_1), tool(call_1), system(early stopping prompt)].

Note: in this checkout 13 tests in tests/readers/file/test_base.py fail because the partial clone is missing tests/readers/file/data. They fail identically with the change reverted (13 failed, 10 passed) and are unrelated to this PR.

Suggested Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added Google Colab support for the newly added notebooks.
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I ran uv run make format; uv run make lint to appease the lint gods

AI assistance disclosure

Per the Transparency principle in CONTRIBUTING.md: this change and its tests were written with AI assistance (Claude Code), and I reviewed the result before submitting. Verification: I reproduced the defect first and confirmed the new tests fail on the pre-fix revision and pass with the fix; I ran tests/agent (92 passed, 3 skipped), ruff check/ruff format --check, mypy llama_index as core-typecheck.yml runs it, and the repo's pre-commit hooks on the changed files. I inspected the generated early stopping request directly rather than relying on the mock LLM's behavior alone.

_generate_early_stopping_response read the memory before finalize ran, so
the early stopping "generate" call only saw the original user message: the
assistant and tool messages of the current run still lived in the agent's
scratchpad, and the model was asked to summarize observations it never
received.

Finalize the agent before fetching the memory, matching what
parse_agent_output already does when the last step made no tool call.

The tool calls of the last step are never run once early stopping fires, so
that message has no tool response to pair with. Drop it from the request,
otherwise the prompt ends with an assistant tool_calls message that
OpenAI/Anthropic-compatible APIs reject.

Applied to both BaseWorkflowAgent and AgentWorkflow.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant