Skip to content

fix(core): keep "Answer:" in return_direct tool output - #23147

Open
BlueX888 wants to merge 1 commit into
run-llama:mainfrom
BlueX888:fix/prep-react-finalize-strips-answer-from-return-direct-tool-output
Open

BlueX888 wants to merge 1 commit into
run-llama:mainfrom
BlueX888:fix/prep-react-finalize-strips-answer-from-return-direct-tool-output

Conversation

@BlueX888

Copy link
Copy Markdown

Description

ReActAgent.finalize strips everything up to and including the first 'Answer:' from the final response text, assuming that text was parsed from an LLM turn by ReActOutputParser. On the return_direct path, however, aggregate_tool_results (base_agent.py) builds the final response straight from the tool output, so a return_direct tool whose output contains 'Answer:' (e.g. Sources:\n doc-1: pricing faq\nAnswer: 42 dollars per seat) is silently truncated to 42 dollars per seat before the caller sees it.

This skips the cleanup when the final response text was fabricated from a return_direct tool observation rather than an LLM turn. The skip additionally requires the final text block to match the observation exactly: handoff tools are also return_direct=True, but after a handoff the next agent produces a fresh LLM response that differs from the handoff observation, so the 'Answer:' cleanup still applies to real LLM answers in multi-agent workflows.

New Package?

Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

  • Yes
  • No

Version Bump?

Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

  • Yes
  • No

Type of Change

  • Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Two new deterministic unit tests (mock LLMs, no network) in llama-index-core/tests/agent/workflow/test_react_agent.py:

  • test_react_agent_return_direct_preserves_answer_substring: a return_direct tool whose output contains 'Answer:' is returned verbatim.
  • test_react_agent_handoff_still_strips_answer_scaffolding: agent A hands off to agent B (handoff tools are also return_direct); B answers with Thought: ... / Answer: ... text and the final response must keep the cleanup (B final answer 42, no raw scaffolding).

Commands and verbatim summaries (run in llama-index-core with llama-index-workflows==2.23.3):

  • With the fix: python -m pytest tests/agent/workflow/test_react_agent.py -q -> 3 passed, 69 warnings in 0.09s
  • Without the fix (git checkout HEAD~1 -- llama-index-core/llama_index/core/agent/workflow/react_agent.py): test_react_agent_return_direct_preserves_answer_substring fails on the original bug: AssertionError: assert '42 dollars per seat' == 'Sources:\n ...lars per seat' -> 1 failed, 2 passed, 69 warnings in 0.20s. The handoff test passes on the unpatched tree (it guards the fix, not the original bug).
  • Teeth of the handoff test: with the skip broadened to any return_direct observation (dropping the text-match condition, which would wrongly cover handoff tools) the handoff test fails: AssertionError: assert 'Thought: I k...nal answer 42' == 'B final answer 42' -> 1 failed, 2 passed, 69 warnings in 0.15s.
  • Full module suite: python -m pytest tests/agent/workflow/ -q -> 67 passed, 3 skipped, 1890 warnings in 0.47s
  • Lint: uvx [email protected] check <changed files> -> All checks passed!; uvx [email protected] --check <changed files> -> 2 files would be left unchanged.; pre-commit run --files <changed files> (ruff, ruff-format, mypy, codespell, whitespace hooks) -> all Passed.

Suggested Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added Google Colab support for the newly added notebooks.
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I ran uv run make format; uv run make lint to appease the lint gods

ReActAgent.finalize applied the ReAct parser cleanup (strip everything up
to and including the first "Answer:") to every response. The return_direct
path in aggregate_tool_results builds that response straight from the tool
output, so a tool result containing "Answer:" was silently truncated before
the caller saw it.

Skip the cleanup when the final response text was fabricated from a
return_direct tool observation instead of produced by ReActOutputParser.
Handoff tools are also return_direct, but they are followed by a fresh LLM
response that differs from the handoff observation, so requiring the final
text to match the observation keeps the cleanup active for that case.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant