fix(formatter): put parallel Gemini function responses in one user turn - #2631

AgentScope Version

2.0.8 (main, base commit 0727cbf)

Description

Fixes #2628. That issue was split out of the review on #2613, and this change covers the Gemini side.

GeminiChatFormatter.format gives every tool result its own user content. The append is at src/agentscope/formatter/_gemini_formatter.py:232, inside the elif isinstance(block, ToolResultBlock): branch that opens at line 221. One result per turn is fine on its own, but parallel tool calls and their results all land in a single assistant Msg, because append_context in src/agentscope/state/_state.py extends the tail assistant message while the reply_id matches. The formatter therefore sees two ToolCallBlocks followed by two ToolResultBlocks in one message and turns the two results into two separate user contents.

Gemini's parallel function calling contract counts the function response parts answering a model turn against the function call parts of that turn, and refuses a mismatch with 400 INVALID_ARGUMENT, "Please ensure that the number of function response parts is equal to the number of function call parts of the function call turn." What the formatter returns is handed to google-genai verbatim as contents (src/agentscope/model/_gemini/_model.py:320), so the shape goes out on the wire and the agent loop stops on the turn after any parallel tool call. I have not run this against the live API, so the evidence below is the request body the formatter builds, not a captured response.

When one of the parallel results carries media the ordering is wrong as well. The promoted media turn is appended straight after the result it came from, so the model turn is followed by a functionResponse, then the media turn, then the second functionResponse.

GeminiMultiAgentFormatter has the same problem, since its _format_tool_sequence calls GeminiChatFormatter.format at src/agentscope/formatter/_gemini_formatter.py:366, so the fix below covers it with no edit of its own.

AnthropicChatFormatter carried this bug and it was fixed in #1894. The comment at src/agentscope/formatter/_anthropic_formatter.py:183-189 spells out why an extra ToolResultBlock must not open a new user message. This change does the same for Gemini, with the same scope: results accumulate within a single Msg, so two results that arrive as two separate Msg objects still produce two turns, exactly as on the Anthropic side.

Repro on main, one user message then an assistant message holding two ToolCallBlocks and their two ToolResultBlocks:

[0] role=user  parts=['text']
[1] role=model parts=['function_call', 'function_call']
[2] role=user  parts=['function_response']
[3] role=user  parts=['function_response']

After the change:

[0] role=user  parts=['text']
[1] role=model parts=['function_call', 'function_call']
[2] role=user  parts=['function_response', 'function_response']

The same two calls with a PNG in the first result, on main:

[0] role=model parts=['function_call', 'function_call']
[1] role=user  parts=['function_response']
[2] role=user  parts=['text', 'text', 'inline_data', 'text']
[3] role=user  parts=['function_response']
[4] role=model parts=['text']

After the change, which is the ordering #2613 proposes for the OpenAI family formatters:

[0] role=model parts=['function_call', 'function_call']
[1] role=user  parts=['function_response', 'function_response']
[2] role=user  parts=['text', 'text', 'inline_data', 'text']
[3] role=model parts=['text']

Change

A contiguous run of ToolResultBlocks now accumulates into tool_result_parts, and media promoted out of those results into promoted_parts. A small _flush_tool_results helper writes the run out as one user content followed by one media content, and it runs both when a block that is not a tool result interrupts the run and at the end of the message. Nothing else moves: the function_call formatting and its repair path, the thinking, hint and data branches, the role mapping, the convert_tool_result_to_string output and the promoted media parts themselves are untouched, and the if parts: flush that already opens the tool result branch stays where it is. A message with a single tool result, with or without media, formats identically to before, which is why nine of the ten existing tests pass unchanged.

Tests

test_parallel_tool_results_share_one_user_turn in tests/formatter_gemini_test.py builds the assistant message with two calls and two results and asserts the whole formatted list, mirroring the sibling tests in that file. test_chat_formatter_parallel_tool_media_after_tool_msgs covers the ordering half: the first of the two results carries a base64 PNG, and the assertion pins the media turn after both function responses rather than between them. It takes the name and shape #2613 gives the same case in the other formatter test files. test_chat_formatter_complex_multi_step asserted call_1 and call_2 in two separate user turns, which encoded the bug; correcting it is a five line deletion that merges the two turns. Run against main's formatter, those three fail and the other nine pass; on this branch all twelve pass.

The formatter files are listed one by one in the second command because pytest tests/ -k formatter -q aborts here with 18 collection errors: the vdb, workspace, storage, realtime and tui extras are not installed.

python -m pytest tests/formatter_gemini_test.py -q
# 12 passed   (the same file against main's formatter: 3 failed, 9 passed)
python -m pytest tests/formatter_gemini_test.py tests/formatter_anthropic_test.py \
  tests/formatter_openai_chat_test.py tests/formatter_openai_response_test.py \
  tests/formatter_openai_response_tool_result_test.py tests/formatter_dashscope_test.py \
  tests/formatter_deepseek_test.py tests/formatter_xai_test.py tests/formatter_ollama_test.py \
  tests/formatter_volcengine_test.py tests/formatter_moonshot_test.py tests/message_test.py -q
# 127 passed, 4 subtests passed

# black, flake8, mypy and pylint on the two changed files, run with the arguments .pre-commit-config.yaml passes them
# 2 files would be left unchanged / no output / Success: no issues found in 2 source files / 10.00/10

macOS, Python 3.11.15, agentscope 2.0.8, black 23.3.0, flake8 6.1.0, mypy 1.7.0, pylint 3.0.2. The new tests import only agentscope.formatter and agentscope.message and make no network call, so they need nothing CI does not already install.

Checklist

Please check the following items before code is ready to be reviewed.

• An issue has been created for this PR
• I have read the CONTRIBUTING.md
• Docstrings are in Google style
• Related documentation has been updated (e.g. links, examples, etc.) in documentation repository
• Code is ready for review