Skip to content

Python: [Bug]: foundry_hosting returns "Content(type=data)" instead of the generated image in image_generation_call.result #9350

Closed

Description

Observed Behavior

When an agent hosted with ResponsesHostServer (agent-framework-foundry-hosting) produces an image with the image generation tool, the image_generation_call output item it sends has result: "Content(type=data)" instead of the image.

The OpenAI chat client stores the generated image as the outputs of an image_generation_tool_result content, as a data URI Content (data:image/png;base64,...). The host passes str(content.outputs) to the SDK's output_item_image_gen_call(result_base64), so the result is the Content repr, and clients that decode result as base64 get garbage.

Expected Behavior

image_generation_call.result is the base64 image, as in the OpenAI Responses API (the SDK parameter is result_base64). The sibling package hosting-responses already extracts it this way in _image_generation_result (agent_framework_hosting_responses/_parsing.py).

Steps to Reproduce

  1. Stream an image_generation_tool_result with a data URI image through the foundry_hosting output tracker (this is what happens for a hosted agent that uses image generation).
  2. Look at the result of the emitted image_generation_call item.

Minimal Reproduction

import asyncio

from agent_framework import Content
from agent_framework_foundry_hosting._responses import _OutputItemTracker
from azure.ai.agentserver.responses.aio import ResponseEventStream


async def main() -> None:
    stream = ResponseEventStream(response_id="resp_img")
    stream.emit_created()
    stream.emit_in_progress()
    tracker = _OutputItemTracker(stream)
    image = Content.from_uri("data:image/png;base64,iVBORw0KGgo=")
    result = Content.from_image_generation_tool_result(image_id="ig_1", outputs=image)

    events = [event async for event in tracker.handle(result)]
    item = next(event for event in events if event["type"] == "response.output_item.done")["item"]
    print(item["type"], repr(item["result"]))  # image_generation_call 'Content(type=data)' (expected 'iVBORw0KGgo=')


asyncio.run(main())

The cause is in agent_framework_foundry_hosting/_responses.py, _OutputItemTracker.handle: self._stream.output_item_image_gen_call(str(content.outputs)).

Error Messages and Stack Traces

No error. The output item carries the text Content(type=data) instead of the image.

Package Versions

agent-framework-foundry-hosting: 1.0.0b261008 (main at b5e5521), agent-framework-core: 1.21.0

Python Version

Python 3.12

Additional Context

I'd like to fix this. The change is a small helper that returns the base64 part of a data URI (the same logic as hosting-responses), used in place of str(content.outputs), plus a regression test.

AI assistance: I used an AI coding assistant for the analysis and to draft this report. I reproduced and verified the behavior myself.

Metadata

Metadata

Labels

pythonUsage: [Issues, PRs], Target: Python

Type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions