Skip to content

Python: Add tool call/result content types and update connectors and samples - #2971

Merged
Eduard van Valkenburg (eavanvalkenburg) merged 22 commits into
mainfrom
copilot/add-tool-call-content-types
Jan 8, 2026
Merged

Python: Add tool call/result content types and update connectors and samples#2971
Eduard van Valkenburg (eavanvalkenburg) merged 22 commits into
mainfrom
copilot/add-tool-call-content-types

Conversation

Copilot AI commented Dec 19, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor

Motivation and Context

MEAI introduced new content types for tool calls/results (code interpreter, MCP server, image generation) and enhancements to hosted file metadata; Python core needed parity so connectors can emit/consume these shapes.

Description

  • Types
    • Added CodeInterpreterToolCallContent, CodeInterpreterToolResultContent, ImageGenerationToolCallContent, ImageGenerationToolResultContent, MCPServerToolCallContent, MCPServerToolResultContent; extended HostedFileContent with optional media_type and name; updated Contents union and serialization.
    • FunctionApprovalRequestContent now accepts MCPServerToolCallContent in addition to standard function calls.
  • Tools
    • Added HostedImageGenerationTool with typed options/additional properties and translate those options to OpenAI Responses parameters.
  • Connectors
    • Updated OpenAI Responses parsing to emit paired tool call/result contents for code interpreter, image generation, and MCP server calls (streaming and non-streaming), even when outputs are empty, ensuring consistent pairing and typing; added option translation for image generation tools and removed raw dict image-generation remapping so callers rely on the tool type; MCP outputs now use the shared content parser.
    • Updated OpenAI Assistants to surface code interpreter and MCP server tool calls.
    • Updated Anthropic connector to emit MCP server and code interpreter tool call/result contents while keeping generic tool_use blocks as FunctionCallContent for non-code-execution tools, fixed missing ChatResponse/CitationAnnotation imports and code-block handling to satisfy type checks, and ensured MCP parsing/typing is consistent.
  • Samples
    • Refreshed OpenAI image generation and code interpreter samples to use the new tool call/result contents and proper tool lists.
  • Tests
    • Expanded core type/tool tests, OpenAI response/assistants tests, Anthropic tests, and added coverage for MCP server tool call acceptance in function approval requests; added guards to skip optional lightning tests when agentlightning is not installed.

Example:

call = CodeInterpreterToolCallContent(call_id="c1", inputs=[TextContent(text="print('hi')")])
result = CodeInterpreterToolResultContent(call_id="c1", outputs=[TextContent(text="log"), UriContent(uri="https://img")])
img_tool = HostedImageGenerationTool(options={"response_format": "uri", "image_size": "1024x1024"})

Contribution Checklist

  • The code builds clean without any errors or warnings
  • The PR follows the Contribution Guidelines
  • All unit tests pass, and I have added new tests where possible
  • Is this a breaking change? If yes, add "[BREAKING]" prefix to the title of the PR.
Original prompt

This section details on the original issue you should resolve

<issue_title>.NET: Python: add *ToolCallContent and *ToolCallResultContent</issue_title>
<issue_description>MEAI added several new content types: dotnet/extensions#6964 and dotnet/extensions#6664 and dotnet/extensions#6749

Below the .Net versions are linked to make it easy to reference.

CodeInterpreterContent

Update HostedFileContent

  • Updates HostedFileContent with an optional MediaType and Name, matching the corresponding properties on DataContent and UriContent.

HostedMCP

Image Generation

Computer Use
TBD
</issue_description>

<agent_instructions>Reference the dotnet links to figure out the structure and then implement them in the right way in the _tools.py for tools and _types.py for types. Make sure to also add the new content types to the Contents union typevar. There should be only python changes and they should be tested, so add test cases for all the types in test_types.py and for tools in test_tools. Everything should be inside the core package, except for some handling of these new types in different connectors, like Anthropic.</agent_instructions>

Comments on the Issue (you are Copilot in this section)

💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI changed the title [WIP] Add ToolCallContent and ToolCallResultContent types Add Python content types for tool calls/results and image generation tool support Dec 19, 2025
eavanvalkenburg
Eduard van Valkenburg (eavanvalkenburg) reviewed Dec 19, 2025
View reviewed changes

@eavanvalkenburg Eduard van Valkenburg (eavanvalkenburg) left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

looking good so far, there are also samples with image generation and code, can you update those as well Copilot

Comment thread python/packages/core/agent_framework/_types.py Resolved
Comment thread python/packages/core/agent_framework/openai/_responses_client.py Resolved
Comment thread python/packages/core/agent_framework/openai/_responses_client.py Resolved
Comment thread python/packages/core/agent_framework/openai/_responses_client.py Resolved
@markwallace-microsoft Mark Wallace (markwallace-microsoft) added the python Usage: [Issues, PRs], Target: Python label Dec 19, 2025
@github-actions github-actions Bot changed the title Add Python content types for tool calls/results and image generation tool support Python: Add Python content types for tool calls/results and image generation tool support Dec 19, 2025

Copilot AI commented Dec 19, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

looking good so far, there are also samples with image generation and code, can you update those as well Copilot

Updated the OpenAI image generation and code interpreter samples to use the new tool call/result content shapes and proper tool lists (1079faa).

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

  • api.openai.com
    • Triggering command: /usr/bin/python python -m pytest packages/core/tests/core/test_types.py packages/core/tests/core/test_tools.py packages/core/tests/openai/test_openai_responses_client.py (dns block)

If you need me to access, download, or install something from one of these locations, you can either:

eavanvalkenburg
Eduard van Valkenburg (eavanvalkenburg) reviewed Dec 19, 2025
View reviewed changes

@eavanvalkenburg Eduard van Valkenburg (eavanvalkenburg) left a comment •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

some additional comments, and the McpServer contents also need handling, also those names should be MCP instead of Mcp (python standard vs dotnet standard) Copilot

Comment thread python/packages/core/agent_framework/_tools.py Resolved
Comment thread python/packages/core/agent_framework/openai/_responses_client.py Resolved
Copilot AI changed the title Python: Add Python content types for tool calls/results and image generation tool support Add Python tool call/result content types and update OpenAI samples Dec 19, 2025
eavanvalkenburg
Eduard van Valkenburg (eavanvalkenburg) reviewed Dec 19, 2025
View reviewed changes

@eavanvalkenburg Eduard van Valkenburg (eavanvalkenburg) left a comment •
edited
Loading

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

couple of notes, and OpenAI Assistants also supports Code Interpreter, so that needs handling. Anthropic also support HostedMCPTool and Code interpreter, so those types need handling as well. Copilot

Comment thread python/packages/core/agent_framework/_types.py Resolved
Comment thread python/packages/core/agent_framework/_types.py Resolved
Comment thread python/packages/core/agent_framework/openai/_responses_client.py Resolved
Comment thread python/packages/core/agent_framework/openai/_responses_client.py Resolved
@eavanvalkenburg

Eduard van Valkenburg (eavanvalkenburg) commented Dec 19, 2025

Copy link
Copy Markdown
Member

couple of notes, and OpenAI Assistants also supports Code Interpreter, so that needs handling. Anthropic also support HostedMCPTool and Code interpreter, so those types need handling as well. Copilot

Copilot please work on this

Copilot AI commented Dec 19, 2025 •
edited
Loading

Copy link
Copy Markdown
Contributor Author

some additional comments, and the McpServer contents also need handling, also those names should be MCP instead of Mcp (python standard vs dotnet standard) Copilot

Updated MCP content naming, parsing, and connector handling; also added option translation for image generation tools as requested (3866935).

couple of notes, and OpenAI Assistants also supports Code Interpreter, so that needs handling. Anthropic also support HostedMCPTool and Code interpreter, so those types need handling as well. Copilot...

Added code interpreter and MCP handling across Responses, Assistants, and Anthropic connectors to emit the new content types (3866935).

Copilot AI changed the title Add Python tool call/result content types and update OpenAI samples Add Python tool call/result content types and update connectors and samples Dec 19, 2025
@eavanvalkenburg

Eduard van Valkenburg (eavanvalkenburg) commented Dec 19, 2025

Copy link
Copy Markdown
Member

Copilot FunctionApprovalRequestContent also needs to allow MCPServerToolCallContent as the function_call inside of it.

Copilot AI commented Dec 19, 2025

Copy link
Copy Markdown
Contributor Author

Copilot FunctionApprovalRequestContent also needs to allow MCPServerToolCallContent as the function_call inside of it.

FunctionApprovalRequestContent now accepts MCPServerToolCallContent (mapping or instance) as the function_call and keeps it typed (3904d09).

@eavanvalkenburg

Eduard van Valkenburg (eavanvalkenburg) commented Dec 19, 2025

Copy link
Copy Markdown
Member

Copilot python/packages/core/agent_framework/openai/_responses_client.py:348 is not needed, folks should use the Tool type

Copilot AI commented Dec 19, 2025

Copy link
Copy Markdown
Contributor Author

Copilot python/packages/core/agent_framework/openai/_responses_client.py:348 is not needed, folks should use the Tool type

Removed the raw dict image_generation remapping; callers should use the tool type directly (f7c856c).

Copilot AI and others added 17 commits January 8, 2026 13:54
@eavanvalkenburg
Handle MCP naming, image options mapping, and connector tool content
fa4424c 
Co-authored-by: eavanvalkenburg <[email protected]>
@eavanvalkenburg
Allow MCP call in function approval request
a2baea2 
Co-authored-by: eavanvalkenburg <[email protected]>
@eavanvalkenburg
Remove raw image_generation tool remapping
382e668 
Co-authored-by: eavanvalkenburg <[email protected]>
@eavanvalkenburg
Restore Anthropic tool_use to function calls unless code execution
f105119 
Co-authored-by: eavanvalkenburg <[email protected]>
@eavanvalkenburg
Fix lint issues for hosted file docstring and MCP parsing
f899704 
Co-authored-by: eavanvalkenburg <[email protected]>
@eavanvalkenburg
Import ChatResponse types in Anthropic client
2c18f74 
Co-authored-by: eavanvalkenburg <[email protected]>
@eavanvalkenburg
Fix Anthropics citation type imports and MCP typing for handoff/tools
33bbfd1 
Co-authored-by: eavanvalkenburg <[email protected]>
@eavanvalkenburg
Skip lightning tests without agentlightning and fix function call import
df2b783 
Co-authored-by: eavanvalkenburg <[email protected]>
@eavanvalkenburg
fix lint on lab package
f5494b8
@eavanvalkenburg
rebuilt anthropic parsing
6ba2354
@eavanvalkenburg
redid anthropic parsing
ff58948
@eavanvalkenburg
typo
65d6277
@eavanvalkenburg
updated parsing and added missing docstrings
f3fe05d
@eavanvalkenburg
fix tests