Skip to content

Live AsyncSession.send_tool_response fails with TypeError when FunctionResponse.parts carries inline bytes #3022

Description

@DouweM

Environment details

  • Programming language: Python 3.14 (also reproduced on 3.10)
  • OS: Linux
  • Package version: google-genai 2.25.0

Steps to reproduce

The Live API's function responses accept multimodal parts, and the Gemini 3.x Live models read an image sent that way. For example, gemini-3.8-live and gemini-3.1-flash-live-preview correctly describe a PNG, JPEG or WebP returned from a tool. However, AsyncSession.send_tool_response can't send such a response. It serializes with _common.convert_to_dict, which leaves FunctionResponseBlob.data as raw bytes, and then passes the result to json.dumps:

import asyncio

from google import genai
from google.genai import types


async def main():
    client = genai.Client()
    tool = types.Tool(function_declarations=[types.FunctionDeclaration(name='take_photo', description='Take a photo.')])
    config = types.LiveConnectConfig(response_modalities=['AUDIO'], tools=[tool])
    async with client.aio.live.connect(model='gemini-3.8-live', config=config) as session:
        await session.send_client_content(
            turns=types.Content(role='user', parts=[types.Part(text='Take a photo and describe it.')])
        )
        async for message in session.receive():
            if message.tool_call:
                call = message.tool_call.function_calls[0]
                await session.send_tool_response(
                    function_responses=types.FunctionResponse(
                        id=call.id,
                        name=call.name,
                        response={'output': 'Photo taken.'},
                        parts=[
                            types.FunctionResponsePart(
                                inline_data=types.FunctionResponseBlob(data=b'\x89PNG...', mime_type='image/png')
                            )
                        ],
                    )
                )
                break


asyncio.run(main())
TypeError: Object of type bytes is not JSON serializable

A minimal repro that needs no API key:

import json

from google.genai import _common, _transformers as t, types

response = types.FunctionResponse(
    id='call_1',
    name='take_photo',
    response={'output': 'Photo taken.'},
    parts=[types.FunctionResponsePart(inline_data=types.FunctionResponseBlob(data=b'\x89PNG', mime_type='image/png'))],
)
json.dumps({'tool_response': _common.convert_to_dict(t.t_tool_response(response), convert_keys=True)})
# TypeError: Object of type bytes is not JSON serializable

Expected behavior

The bytes are base64-encoded, as send_client_content does for Blob.data: it serializes with model_dump(mode='json'), which honors the models' ser_json_bytes='base64'.

Workarounds

  • Serialize the message with the SDK's own types and send it on the session's socket:

    message = types.LiveClientMessage(tool_response=types.LiveClientToolResponse(function_responses=[response]))
    await session._ws.send(message.model_dump_json(by_alias=True, exclude_none=True))

    This works, but it reaches into the private _ws.

  • types.FunctionResponseBlob.model_construct(data=base64.b64encode(data).decode(), mime_type=...) also gets the message sent, but convert_to_dict then emits PydanticSerializationUnexpectedValue (Expected bytes) on every call.

Suggested fix

Serialize the tool response the way send_client_content does, with model_dump(mode='json', exclude_none=True), before json.dumps, so that nested bytes are base64-encoded.

The deprecated AsyncSession.send(input=FunctionResponse(...)) path has a related problem. It rebuilds each function response from name, response and id only, so it silently drops parts (and scheduling).

Activity

  1. added
    type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.
    priority: p2Moderately-important priority. Fix may not be included in next release.
    on Sep 29, 2026
  2. kkorpal commented on Sep 30, 2026

    @kkorpal

    Hi @DouweM, thanks for the awesome bug report and for catching that serialization issue with the google-genai SDK!
    I will file a bug report internally. Thanks

  3. added 5 commits that reference this issue on Oct 20, 2026
    9d5e1f0
    2fd4079
    c3bb22c
    07f63bf
    a7bbff6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

priority: p2Moderately-important priority. Fix may not be included in next release.type: bugError or flaw in code with unintended results or allowing sub-optimal usage patterns.

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions