Repository navigation
Description
Environment details
- Programming language: Python 3.14 (also reproduced on 3.10)
- OS: Linux
- Package version:
google-genai2.25.0
Steps to reproduce
The Live API's function responses accept multimodal parts, and the Gemini 3.x Live models read an image sent that way. For example, gemini-3.8-live and gemini-3.1-flash-live-preview correctly describe a PNG, JPEG or WebP returned from a tool. However, AsyncSession.send_tool_response can't send such a response. It serializes with _common.convert_to_dict, which leaves FunctionResponseBlob.data as raw bytes, and then passes the result to json.dumps:
import asyncio
from google import genai
from google.genai import types
async def main():
client = genai.Client()
tool = types.Tool(function_declarations=[types.FunctionDeclaration(name='take_photo', description='Take a photo.')])
config = types.LiveConnectConfig(response_modalities=['AUDIO'], tools=[tool])
async with client.aio.live.connect(model='gemini-3.8-live', config=config) as session:
await session.send_client_content(
turns=types.Content(role='user', parts=[types.Part(text='Take a photo and describe it.')])
)
async for message in session.receive():
if message.tool_call:
call = message.tool_call.function_calls[0]
await session.send_tool_response(
function_responses=types.FunctionResponse(
id=call.id,
name=call.name,
response={'output': 'Photo taken.'},
parts=[
types.FunctionResponsePart(
inline_data=types.FunctionResponseBlob(data=b'\x89PNG...', mime_type='image/png')
)
],
)
)
break
asyncio.run(main())TypeError: Object of type bytes is not JSON serializable
A minimal repro that needs no API key:
import json
from google.genai import _common, _transformers as t, types
response = types.FunctionResponse(
id='call_1',
name='take_photo',
response={'output': 'Photo taken.'},
parts=[types.FunctionResponsePart(inline_data=types.FunctionResponseBlob(data=b'\x89PNG', mime_type='image/png'))],
)
json.dumps({'tool_response': _common.convert_to_dict(t.t_tool_response(response), convert_keys=True)})
# TypeError: Object of type bytes is not JSON serializableExpected behavior
The bytes are base64-encoded, as send_client_content does for Blob.data: it serializes with model_dump(mode='json'), which honors the models' ser_json_bytes='base64'.
Workarounds
-
Serialize the message with the SDK's own types and send it on the session's socket:
message = types.LiveClientMessage(tool_response=types.LiveClientToolResponse(function_responses=[response])) await session._ws.send(message.model_dump_json(by_alias=True, exclude_none=True))
This works, but it reaches into the private
_ws. -
types.FunctionResponseBlob.model_construct(data=base64.b64encode(data).decode(), mime_type=...)also gets the message sent, butconvert_to_dictthen emitsPydanticSerializationUnexpectedValue(Expected bytes) on every call.
Suggested fix
Serialize the tool response the way send_client_content does, with model_dump(mode='json', exclude_none=True), before json.dumps, so that nested bytes are base64-encoded.
The deprecated AsyncSession.send(input=FunctionResponse(...)) path has a related problem. It rebuilds each function response from name, response and id only, so it silently drops parts (and scheduling).
Activity
kkorpal commentedon Sep 30, 2026
Hi @DouweM, thanks for the awesome bug report and for catching that serialization issue with the google-genai SDK!
I will file a bug report internally. Thanks
fix: Base64-encode inline bytes in Live send_tool_response
fix: Base64-encode inline bytes in Live send_tool_response
fix: Base64-encode inline bytes in Live send_tool_response
fix: Base64-encode inline bytes in Live send_tool_response
fix: Base64-encode inline bytes in Live send_tool_response