Skip to content

fix: normalize AG-UI snapshot message content instead of str() - #3020

Open
BlueX888 wants to merge 2 commits into
vibrantlabsai:mainfrom
BlueX888:fix/prep-agui-snapshot-content-str-coercion
Open

fix: normalize AG-UI snapshot message content instead of str()#3020
BlueX888 wants to merge 2 commits into
vibrantlabsai:mainfrom
BlueX888:fix/prep-agui-snapshot-content-str-coercion

Conversation

@BlueX888

@BlueX888 BlueX888 commented Sep 19, 2026 •
edited
Loading

Copy link
Copy Markdown

Issue Link / Problem Description

AGUIEventCollector._handle_messages_snapshot builds content with str(getattr(msg, "content", "")) (src/ragas/integrations/ag_ui.py:620 on main). Two documented shapes of ag_ui.core message content are destroyed by that coercion:

  1. None becomes the string "None". AssistantMessage.content is Optional[str] with default None (ag-ui-protocol 0.1.22), which is what a tool-call-only assistant turn carries. str(None) yields the four-character string "None", so the message content is recorded as "None" rather than empty. That value reaches SingleTurnSample.response through build_sample → extract_response, so metrics get scored against the literal text None.
  2. Multimodal content becomes its pydantic repr. UserMessage.content is Union[str, List[InputContentPart]]. str([TextInputContent(...)]) produces a repr, so HumanMessage.content becomes [TextInputContent(type='text', text='...')] instead of the user's actual text.

Reproduction on main:

from ag_ui.core import AssistantMessage, MessagesSnapshotEvent, TextInputContent, UserMessage
from ragas.integrations.ag_ui import convert_messages_snapshot

snapshot = MessagesSnapshotEvent(messages=[
    UserMessage(id="msg-1", content="What's the weather?"),
    AssistantMessage(id="msg-2"),  # content=None
])
convert_messages_snapshot(snapshot)[1].content
# main: 'None'      expected: ''

snapshot = MessagesSnapshotEvent(messages=[
    UserMessage(id="msg-1", content=[TextInputContent(type="text", text="What is in this screenshot?")]),
])
convert_messages_snapshot(snapshot)[0].content
# main: "[TextInputContent(type='text', text='What is in this screenshot?')]"
# expected: "What is in this screenshot?"

Changes Made

  • src/ragas/integrations/ag_ui.py — added _normalize_content(), which maps None to "", passes str through unchanged, and flattens a multimodal content list to the text of its TextInputContent parts. The other input-part types (image/audio/video/document/binary) have no text field and are skipped; anything else keeps the previous str() behaviour as a fallback. _handle_messages_snapshot now calls it instead of str(...).
  • tests/unit/integrations/test_ag_ui.py — two regression tests, one per shape. The multimodal test skips with a reason when the installed ag-ui-protocol has no TextInputContent (the repository's minimum pin, 0.1.9).

The streaming path is untouched: it already builds content by joining deltas in _handle_text_message_content.

Testing

How to Test

  • Automated tests added/updated
  • Manual testing steps: not required — the two regression tests cover both shapes end to end

Both tests fail on main and pass with the change:

$ git checkout origin/main -- src/ragas/integrations/ag_ui.py
$ python -m pytest tests/unit/integrations/test_ag_ui.py -k "none_content_is_empty or multimodal_content_keeps_text" -q
FF                                                                       [100%]
=================================== FAILURES ===================================
___________________ test_snapshot_with_none_content_is_empty ___________________
>       assert messages[1].content == ""
E       AssertionError: assert 'None' == ''
E         + None

tests/unit/integrations/test_ag_ui.py:252: AssertionError
_______________ test_snapshot_with_multimodal_content_keeps_text _______________
>       assert messages[0].content == "What is in this screenshot?"
E       assert "[TextInputCo...creenshot?')]" == 'What is in this screenshot?'
E
E         - What is in this screenshot?
E         + [TextInputContent(type='text', text='What is in this screenshot?')]

tests/unit/integrations/test_ag_ui.py:274: AssertionError
=========================== short test summary info ============================
FAILED tests/unit/integrations/test_ag_ui.py::test_snapshot_with_none_content_is_empty
FAILED tests/unit/integrations/test_ag_ui.py::test_snapshot_with_multimodal_content_keeps_text
2 failed, 43 deselected in 0.14s

$ git checkout HEAD -- src/ragas/integrations/ag_ui.py
$ python -m pytest tests/unit/integrations/test_ag_ui.py -k "none_content_is_empty or multimodal_content_keeps_text" -q
..                                                                       [100%]
2 passed, 43 deselected in 0.07s

$ python -m pytest tests/unit/integrations/test_ag_ui.py -q
45 passed in 0.09s

$ python -m pytest tests/unit/integrations -q
72 passed in 0.11s

With the version guard in 68db694: on ag-ui-protocol==0.1.9, which has no TextInputContent, the multimodal test is skipped and the file reports 44 passed, 1 skipped; on versions that provide TextInputContent it reports 45 passed.

Lint/format (the Makefile format target):

$ ruff format --check src/ragas/integrations/ag_ui.py tests/unit/integrations/test_ag_ui.py --config pyproject.toml
2 files already formatted

$ ruff check src/ragas/integrations/ag_ui.py tests/unit/integrations/test_ag_ui.py --config pyproject.toml
All checks passed!

Types: pyright -p pyproject.toml src reports no findings in src/ragas/integrations/ag_ui.py. The 6 errors / 11 warnings it reports elsewhere (langsmith.py, tracing/langfuse.py, tracing/mlflow.py, plus reportMissingModuleSource warnings) are present on main and unchanged by this PR.

References

  • Related code: _handle_text_message_content (src/ragas/integrations/ag_ui.py:322) builds streaming content by joining deltas, so only the snapshot path needed fixing; build_sample (src/ragas/integrations/ag_ui.py:1265) is where the coerced value lands in SingleTurnSample.response.
  • External reference: ag-ui-protocol AssistantMessage / UserMessage content types in ag_ui.core.
  • Note: fix: read nested function name/arguments from AG-UI snapshot tool calls #3011 also edits _handle_messages_snapshot but fixes a different defect (tool-call name/arguments read from the nested function object). The hunks are disjoint; whichever lands second will need a trivial rebase.

Screenshots/Examples (if applicable)

Not applicable — no UI change.

@BlueX888
fix: normalize AG-UI snapshot message content instead of str()
5a8d574 
str() turned content=None (AG-UI assistant messages, e.g. tool-call-only
turns) into the literal string 'None', and rendered multimodal
UserMessage content as its pydantic repr. Both leaked into
SingleTurnSample.response and metric inputs.
@contextablemark

contextablemark commented Oct 11, 2026

Copy link
Copy Markdown
Contributor

Thanks for the detailed write-up, and for flagging the overlap with #3011 in the references. As the author of the original AG-UI integration (#2395), I reproduced both defects you describe and validated this patch locally. I support merging it alongside #3011 — they fix distinct bugs in the same handler and combine cleanly.

Validation (exact patch at 5a8d574):

Unit level:

  • The content=None regression test is red on unpatched main and green with the patch, on both ag-ui-protocol==0.1.9 and 0.1.22.
  • The multimodal regression test is red on main and green on 0.1.22 (see version note below).
  • Manually confirmed the reported paths: AssistantMessage.content=None became the string 'None' and flowed through extract_response() into build_sample()'s SingleTurnSample.response; multimodal content became its pydantic repr.
  • Ruff lint/format and pyright are clean for ag_ui.py with the patch applied.

Live end-to-end:

One test issue on the minimum pinned version: the repo pins ag-ui-protocol>=0.1.9, and on 0.1.9 UserMessage.content is str-only with no TextInputContent, so test_snapshot_with_multimodal_content_keeps_text raises ImportError there (I verified: 44 passed / 1 failed on 0.1.9, 45 passed on 0.1.22). The runtime _normalize_content() handles the missing shape safely via getattr, so only the test needs a guard — a try/except around the import with pytest.skip when unavailable would keep the suite green on the minimum pin.

Linked issue: happy to file one capturing both shapes (with the repros from your description) if the maintainers prefer the issue-first workflow.

@anistark — you reviewed and merged the original integration; could you take a look when you have a chance, or point us to the current maintainer for src/ragas/integrations/?

— Mark

@contextablemark contextablemark mentioned this pull request Oct 11, 2026
Open
@contextablemark

contextablemark commented Oct 11, 2026

Copy link
Copy Markdown
Contributor

Linked issue: #3060 (filed per the offer above, with fresh minimal repros on main for both shapes — the content=None one verified on ag-ui-protocol 0.1.9 and 0.1.22).

This was referenced Oct 11, 2026
Open
Closed
@BlueX888

BlueX888 commented Oct 12, 2026

Copy link
Copy Markdown
Author

Thanks @contextablemark — especially for catching the 0.1.9 import failure and confirming the counts on both pins.

Pushed 68db694: the multimodal test now wraps the TextInputContent import in try/except and pytest.skips with a factual reason when the installed ag-ui-protocol doesn't provide it. The runtime path is untouched — as you noted, _normalize_content() already handles the missing shape via getattr, and TextInputContent was only referenced by the test.

Verified: on ag-ui-protocol==0.1.9 → 44 passed, 1 skipped (this test, matching your numbers); on ==0.1.22 → 45 passed with this test running and passing. Ruff clean.

@contextablemark

contextablemark commented Oct 12, 2026

Copy link
Copy Markdown
Contributor

Thanks for the compatibility guard in 68db694. I reviewed the change and reran the AG-UI test file against the updated PR head:

  • ag-ui-protocol==0.1.9: 44 passed, 1 skipped, with only the unsupported multimodal test skipped.
  • ag-ui-protocol==1.0.0: 45 passed, including the multimodal test.

That resolves the minimum-version test failure I reported. The runtime normalization is unchanged, so my earlier validation and support for this fix still stand. #3060 tracks the underlying defects.

For clarity, I’m the author of the original AG-UI integration, not a Ragas maintainer; I can provide validation and recommendations but cannot approve a merge on the project’s behalf. The project appears to have stalled on the maintenance side: the latest merge I could verify was February 24, 2026, and there have been no merges in the last 90 days, although contributors remain active. I don’t know whether maintenance has paused or moved to someone else.

The compatibility fix addresses my outstanding request. Hopefully a current maintainer can review and merge this work.

— Mark

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

AG-UI MessagesSnapshot: str() coercion corrupts message content (None -> 'None', multimodal -> pydantic repr)

2 participants

@BlueX888 @contextablemark