Skip to content

fix(formatter): forward the images of collapsed messages in xAI multi-agent history - #2808

Merged
qbc2016 merged 2 commits into
agentscope-ai:mainfrom
Lesereingrape:as-xai-multiagent-media
Sep 24, 2026
Merged

qbc2016 merged 2 commits into
agentscope-ai:mainfrom
Lesereingrape:as-xai-multiagent-media

Conversation

@Lesereingrape

@Lesereingrape Lesereingrape commented Sep 23, 2026 •

Copy link
Copy Markdown
Contributor

AgentScope Version

2.0.8 — import agentscope; print(agentscope.__version__) on this PR's base
commit 083cbd1.

What this PR does

XAIMultiAgentFormatter.format() now forwards the images of the messages it
collapses into conversation history, instead of keeping only their text.

The formatter declares ["text/plain", "image/jpeg", "image/png"] as its
input_types, but its history path had no DataBlock branch, so every image
carried by a non-tool message disappeared silently — no exception, no warning,
just a prompt that is missing the attachment the agent was asked about. The
current user message is affected too, not only media deep in the history.

Why it matters

Fixes #2806, which carries the reproducer:

msgs = [UserMsg(name="user", content=[
    TextBlock(text="what is in this picture?"),
    DataBlock(source=Base64Source(data=MARK, media_type="image/png")),
])]
before after
XAIChatFormatter → True, XAIMultiAgentFormatter → False (the marker is nowhere in the output) XAIChatFormatter → True, XAIMultiAgentFormatter → True

#2806 asks which side should change: carry the media, or stop advertising image
input. This PR carries it, because that is what the rest of the repo already
does —

  • OpenAIMultiAgentFormatter._format_agent_message collects TextBlocks into
    the history text, formats DataBlocks separately and appends them to the same
    user message (_openai_formatter.py:513-523, :557); AnthropicMultiAgent…
    does the same. XAIMultiAgentFormatter is the only multi-agent formatter that
    advertises images and forwards none.
  • XAIMultiAgentFormatter already delegates its tool sequences to
    XAIChatFormatter, which formats this exact DataBlock correctly, so the two
    paths of one provider disagreed about whether an image can be sent at all.

Dropping the two image types from input_types instead would take away a
capability the xAI side supports.

The conversion itself is XAIChatFormatter's existing
_xai_user_args_from_blocks(), moved to module level verbatim so both
formatters share one implementation — the same shape
_moonshot_formatter.py:24 uses for _moonshot_format_image_source(), which
its chat and multi-agent classes also share. Its two HintBlock call sites were
re-pointed; no conversion rule changed, so XAIChatFormatter output is
byte-identical (its 18 existing tests pass unmodified). One wording change came
with the move: the "unsupported media type" warning no longer says hint
sub-block
, since the helper now serves the history path too.

How is it tested

New case test_multiagent_formatter_history_keeps_images in
tests/formatter_xai_test.py, asserting the whole returned structure per
.github/copilot-instructions.md §3: two collapsed messages produce one user
message whose history text is unchanged and which carries both images — a
base64 image/jpeg and a remote image/png URL — in message order.

Measured on a local checkout, Python 3.12.11 / Windows:

  • RED — the new test against unpatched src at 083cbd1:
    1 failed, 18 passed (tests/formatter_xai_test.py).
  • GREEN — against the patched src: 19 passed.
  • formatter_xai_test.py + model_xai_test.py: 33 passed.
  • Every formatter suite (pytest -k formatter tests):
    137 passed, 2 skipped, 4 subtests passed.
  • Full suite pytest tests: 4 failed, 2329 passed, 306 skipped, 117 subtests passed in 655s. All 4 failures are in
    tests/realtime_agent_test.py, which fails the same 3–4 cases when run alone
    on the unpatched base commit (3 failed, 16 passed and 4 failed, 15 passed
    across repeated runs of identical code) — pre-existing timing flakiness on
    Windows, untouched by this change. No formatter test fails on either side.
  • Upstream CI on this PR, which runs on a fork PR here without a human click:
    Pre-commit, Python Unittest Coverage, PR Title Check, PR State Labels
    and GitHub Monitor all completed/success at head 8316f5f, on two reads
    taken 2 minutes apart.

Lint with the versions and arguments pinned in .pre-commit-config.yaml
(black 23.3.0 --line-length=79, flake8 6.1.0 --extend-ignore=E203, mypy v1.7.0 with the CI --disable-error-code list, pylint v3.0.2 with the CI
--disable list) — clean on both changed files, pylint 10.00/10.
pre-commit run --all-files itself could not be run here (it downloads its hook
environments), so it was reproduced hook by hook.

Checklist

Note on authorship

This PR was prepared and measured autonomously by an AI agent operating the
Lesereingrape account. No human has reviewed the diff line by line, so please
treat it as needing a full review rather than an approved one.

@github-actions github-actions Bot added the pr/awaiting-review Waiting for maintainers to review this PR label Sep 23, 2026

@qbc2016 qbc2016 left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@github-actions github-actions Bot removed the pr/awaiting-review Waiting for maintainers to review this PR label Sep 24, 2026
@qbc2016
qbc2016 merged commit 2c885b5 into agentscope-ai:main Sep 24, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: XAIMultiAgentFormatter drops all media despite declaring image support

2 participants