Skip to content

fix: preserve user images in native tool call results #9401

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
mrubens merged 2 commits into from
Nov 19, 2025
Merged

fix: preserve user images in native tool call results #9401

mrubens merged 2 commits into from
Nov 19, 2025

Conversation

@daniel-lxs
Copy link
Member

@daniel-lxs daniel-lxs commented Nov 19, 2025 •
edited by ellipsis-dev bot
Loading

Problem

When native tool calls are enabled, user images sent with responses were being converted to placeholder text (image content) instead of being preserved and sent to the LLM. This caused the model to lose important visual context that users were providing.

Solution

Modified the pushToolResult function in presentAssistantMessage.ts to:

  • Preserve image blocks in tool_result content as arrays when images are present
  • Only convert to string when no images exist (for cleaner representation)
  • Maintain full compatibility with Anthropic API's tool_result format

Changes

  • Modified: src/core/assistant-message/presentAssistantMessage.ts

    • Updated pushToolResult function to detect and preserve image blocks
    • Images are kept as Array<TextBlockParam | ImageBlockParam> when present
    • Falls back to string conversion when no images for simpler representation

  • Added: src/core/assistant-message/__tests__/presentAssistantMessage-images.spec.ts

    • Comprehensive test coverage for image handling in both native and XML protocols
    • 4 test cases covering various scenarios

Testing

All tests pass, including:

  • ✅ Image preservation in native protocol tool results
  • ✅ String conversion when no images (native protocol)
  • ✅ Image preservation in XML protocol (existing behavior)
  • ✅ Empty tool result handling

Impact

  • Native Protocol: Images are now correctly preserved and sent to the LLM
  • XML Protocol: No changes - continues to work as before
  • Backwards Compatible: No breaking changes to existing functionality

Important

Fixes image preservation in native tool call results by updating pushToolResult in presentAssistantMessage.ts to handle image blocks correctly.

  • Behavior:
    • pushToolResult in presentAssistantMessage.ts now preserves image blocks in tool results for native protocol.
    • Converts to string only if no images are present.
    • Ensures compatibility with Anthropic API's tool_result format.
  • Testing:
    • Added presentAssistantMessage-images.spec.ts with 4 test cases for image handling in native and XML protocols.
    • Tests cover image preservation, string conversion, XML protocol behavior, and empty tool result handling.
  • Impact:
    • Native protocol now correctly preserves images for LLM.
    • XML protocol remains unchanged.
    • No breaking changes to existing functionality.

This description was created by Ellipsis for b5dc163. You can customize this summary. It will automatically update as commits are pushed.

@daniel-lxs
fix: preserve user images in native tool call results
b5dc163 
When native tool calls are enabled, user images sent with responses
were being converted to placeholder text '(image content)' instead of
being preserved and sent to the LLM.

This fix:
- Preserves image blocks in tool_result content as arrays when images
  are present
- Only converts to string when no images exist (for cleaner
  representation)
- Maintains compatibility with Anthropic API's tool_result format

Also adds comprehensive test coverage for image handling in both
native and XML protocols.
@dosubot dosubot bot added size:L This PR changes 100-499 lines, ignoring generated files. bug Something isn't working labels Nov 19, 2025
@roomote
Copy link
Contributor

roomote bot commented Nov 19, 2025 •
edited
Loading

Rooviewer Clock   See task on Roo Cloud

Review completed. No issues found - the implementation correctly preserves images in native protocol tool results while maintaining backward compatibility.

Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues.

@daniel-lxs daniel-lxs marked this pull request as draft November 19, 2025 17:32
@hannesrudolph hannesrudolph added the Issue/PR - Triage New issue. Needs quick review to confirm validity and assign labels. label Nov 19, 2025
@daniel-lxs
fix: properly handle images in native tool results
e1749bc 
Images should be added as separate blocks after tool_result,
not inside the tool_result content itself. This follows the
Anthropic API specification where tool_result content must
be a string, and images are separate content blocks in the
user message.

Updated tests to verify images are correctly added as
separate blocks in the user message content.
@daniel-lxs daniel-lxs moved this from Triage to PR [Needs Review] in Roo Code Roadmap Nov 19, 2025
@daniel-lxs daniel-lxs marked this pull request as ready for review November 19, 2025 19:01
@roomote
Copy link
Contributor

roomote bot commented Nov 19, 2025 •
edited
Loading

Rooviewer Clock   See task on Roo Cloud

Review completed. No issues found - the implementation correctly preserves images in native protocol tool results while maintaining backward compatibility.

Mention @roomote in a comment to request specific changes to this pull request or fix all unresolved issues.

@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Nov 19, 2025
@hannesrudolph hannesrudolph added PR - Needs Review and removed Issue/PR - Triage New issue. Needs quick review to confirm validity and assign labels. labels Nov 19, 2025
@mrubens mrubens merged commit 61fc391 into main Nov 19, 2025
23 checks passed
@mrubens mrubens deleted the fix/preserve-images-native-tool-calls branch November 19, 2025 19:39
@github-project-automation github-project-automation bot moved this from New to Done in Roo Code Roadmap Nov 19, 2025
@github-project-automation github-project-automation bot moved this from PR [Needs Review] to Done in Roo Code Roadmap Nov 19, 2025
mini2s added a commit to zgsm-ai/costrict that referenced this pull request Nov 20, 2025
@mini2s @mrubens
@daniel-lxs @kevint-cerebras @jr @roomote @ellipsis-dev @roomote @heyseth @elianiva @hannesrudolph @github-actions @brunobergher @Thibault00 @requesty-JohnCosta27 @dmarkey @SannidhyaSah @cte @sebastiand-cerebras @dleffel @ajjuaire @renovate
Roo to main (#677)