fix: read nested function name/arguments from AG-UI snapshot tool calls by BlueX888 · Pull Request #3011 · vibrantlabsai/ragas

BlueX888 · 2026-09-15T14:17:17Z

Issue Link / Problem Description

Fixes AG-UI MessagesSnapshot: convert_messages_snapshot drops tool call name and arguments (ToolCall(name='unknown_tool', args={})) #3010

convert_messages_snapshot() rebuilt tool calls from a MessagesSnapshotEvent by reading tc.name / tc.args through getattr(..., default), but ag_ui.core.ToolCall has neither: the name and the (JSON-string) arguments live under ToolCall.function. Both getattr calls therefore always returned their defaults, so every tool call recovered from a snapshot came back as ToolCall(name='unknown_tool', args={}) with the real tool name and arguments discarded.

Changes Made

src/ragas/integrations/ag_ui.py — AGUIEventCollector._handle_messages_snapshot (line 636): read name / arguments from the nested function object, and json.loads the arguments string. Empty arguments become {}. A string that is not valid JSON keeps the existing {"raw_args": ...} fallback already used by the streaming path (_handle_tool_call_end, line 406).
tests/unit/integrations/test_ag_ui.py — new regression test test_messages_snapshot_preserves_tool_call_name_and_args asserting that a snapshot tool call converts to ToolCall(name="get_weather", args={"city": "SF"}).

The writer in the same file (convert_messages_to_ag_ui, line 915) already emits the nested FunctionCall(name=..., arguments=json.dumps(...)) shape, so a Ragas → AG-UI → Ragas round-trip now preserves tool names and arguments.

Testing

How to Test

Automated tests added/updated
Manual testing steps: not required — the regression test covers the reporter's repro end to end

The new test fails on main and passes with the fix:

$ git checkout origin/main -- src/ragas/integrations/ag_ui.py
$ python -m pytest tests/unit/integrations/test_ag_ui.py::test_messages_snapshot_preserves_tool_call_name_and_args -q
E       AssertionError: assert [ToolCall(nam...ol', args={})] == [ToolCall(nam...city': 'SF'})]
E         At index 0 diff: ToolCall(name='unknown_tool', args={}) != ToolCall(name='get_weather', args={'city': 'SF'})
FAILED tests/unit/integrations/test_ag_ui.py::test_messages_snapshot_preserves_tool_call_name_and_args
1 failed in 0.14s

$ python -m pytest tests/unit/integrations/test_ag_ui.py::test_messages_snapshot_preserves_tool_call_name_and_args -q
1 passed in 0.12s

$ python -m pytest tests/unit/integrations/test_ag_ui.py -q
44 passed in 0.06s

Lint/format (per the Makefile run-ci-format-check / run-ci-check targets):

$ ruff format --check src/ragas/integrations/ag_ui.py tests/unit/integrations/test_ag_ui.py --config pyproject.toml
2 files already formatted

$ ruff check src/ragas/integrations/ag_ui.py tests/unit/integrations/test_ag_ui.py --config pyproject.toml
All checks passed!

Types: pyright -p pyproject.toml src reports no findings in src/ragas/integrations/ag_ui.py. The 20 errors it reports elsewhere are present on main too and are unchanged by this PR.

Note: -p no:phoenix is only needed in this local virtualenv, where the arize-phoenix pytest plugin fails to import (ModuleNotFoundError: No module named 'phoenix.evals.models'); it is unrelated to this change.

Behaviour before / after

snapshot = MessagesSnapshotEvent(messages=[
    AssistantMessage(id="a1", content="Checking", tool_calls=[
        AGToolCall(id="tc1", function=FunctionCall(name="get_weather", arguments='{"city":"SF"}'))]),
])

# main:    ToolCall(name='unknown_tool', args={})
# this PR: ToolCall(name='get_weather', args={'city': 'SF'})
convert_messages_snapshot(snapshot)[0].tool_calls

References

Related issue: AG-UI MessagesSnapshot: convert_messages_snapshot drops tool call name and arguments (ToolCall(name='unknown_tool', args={})) #3010
Related code: _handle_tool_call_end (src/ragas/integrations/ag_ui.py:406) already parses function.arguments JSON on the streaming path; convert_messages_to_ag_ui (src/ragas/integrations/ag_ui.py:915) already writes the nested shape.
External reference: ag-ui-protocol ToolCall / FunctionCall models (ag_ui.core).

Screenshots/Examples (if applicable)

Not applicable — no UI change.

MessagesSnapshotEvent tool calls are ag_ui.core.ToolCall objects, which store the name and arguments under `function` (arguments as a JSON string). Reading `tc.name` / `tc.args` always fell back to the defaults, so every tool call reconstructed from a snapshot came back as name='unknown_tool' with empty args.

contextablemark · 2026-10-11T16:19:17Z

Thanks for catching this and putting together the fix. As the author of the original AG-UI integration (#2395), I support this PR as the resolution for #3010.

I reproduced the loss of tool names and arguments, then locally validated the exact patch at 51e4d0da. The snapshot reader was incorrectly accessing top-level name/args; reading function.name and decoding function.arguments correctly matches the AG-UI schema and our existing writer.

Validation results:

All 44 AG-UI integration tests passed.
An additional 14 local regression cases passed with both ag-ui-protocol==0.1.9 and 0.1.22. All 14 failed against the original code.
These covered round trips, multiple tool calls, tool results and metadata, nested/Unicode arguments, empty strings, malformed JSON, and valid non-object JSON.
Ruff lint/format checks passed, and Pyright reported no errors or warnings for ag_ui.py.

Between this PR and #3047, I prefer this smaller, focused change. It handles the supported nested schema without additional flat-format compatibility branches, preserves the raw_args fallback, and treats empty arguments as {}, consistent with the streaming handler.

It would be useful to include the round-trip and argument edge cases in the committed tests as well. My validation was scoped to the AG-UI integration, not the full repository suite.

@anistark — you helped review and merge the original integration. Could you take a look when you have a chance, or point us to the current maintainer for this area?

— Mark

BlueX888 · 2026-10-12T02:45:12Z

Thanks @contextablemark — glad this shape won out, and thanks for the explicit preference over #3047.

Pushed 30cd756 committing the round-trip and argument edge cases from your validation list:

round trip through the writer: nested function.name + JSON function.arguments, read back as ToolCall(name=..., args={...})
multiple tool calls, order preserved
tool results with metadata (built as a snapshot directly — convert_messages_to_ag_ui intentionally skips ToolMessage; noted in the docstring)
nested/Unicode arguments, including an empty-string argument value
empty arguments → {}
malformed JSON → raw_args fallback
valid non-object JSON (array / string / number) → parsed raw_args

All 7 new tests fail against the pre-fix reader with the ToolCall(name='unknown_tool', args={}) signature; with the fix, 51 passed on both ag-ui-protocol==1.0.0 and ==0.1.9. Ruff clean.

contextablemark · 2026-10-12T07:03:00Z

Thanks for adding the regression coverage in 30cd756. I reviewed the new tests and reran the AG-UI test file against the updated PR head: 51 passed on both ag-ui-protocol==0.1.9 and ==1.0.0.

The added round-trip, multiple-call, metadata, nested/Unicode, empty-argument, malformed-JSON, and non-object-JSON cases address my earlier suggestion. The runtime fix is unchanged, and I continue to support #3011 as the resolution for #3010.

For clarity, I’m the author of the original AG-UI integration, not a Ragas maintainer; this is contributor validation, not merge approval. The project appears to have stalled on the maintenance side: the latest merge I could verify was February 24, 2026, with no merges in the last 90 days, despite continued contributor activity. I can’t tell whether that reflects a pause or a change in ownership. The outstanding check-docs failure also needs maintainer attention; it should not be mistaken for a failure of these regression tests.

I hope a current maintainer can pick this up. Thanks again for following through.

— Mark

BlueX888 · 2026-10-25T19:07:35Z

Up to date with main (298b682) and applies cleanly; no rebase needed. This fixes AG-UI messages-snapshot tool calls losing their function name and arguments, which read back as ToolCall(name='unknown_tool', args={}). Tests pass: 51 AG-UI tests on ag-ui-protocol==0.1.9 and ==1.0.0, including the round-trip and argument edge cases. The check-docs failure is the fork checkout step in claude-docs-check.yml, not this patch. Happy to adjust or split the change if that helps review.

BlueX888 mentioned this pull request Sep 19, 2026

fix: normalize AG-UI snapshot message content instead of str() #3020

Open

1 of 2 tasks

test: add AG-UI snapshot tool-call round-trip and argument edge cases

30cd756

Navigation Menu

Repository navigation

fix: read nested function name/arguments from AG-UI snapshot tool calls - #3011

fix: read nested function name/arguments from AG-UI snapshot tool calls#3011
BlueX888 wants to merge 2 commits into
vibrantlabsai:mainfrom
BlueX888:fix/prep-agui-snapshot-toolcall-name-args-lost

BlueX888 commented Sep 15, 2026

Uh oh!

contextablemark commented Oct 11, 2026

Uh oh!

BlueX888 commented Oct 12, 2026

Uh oh!

contextablemark commented Oct 12, 2026

Uh oh!

BlueX888 commented Oct 25, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

BlueX888 commented Sep 15, 2026

Issue Link / Problem Description

Changes Made

Testing

How to Test

Behaviour before / after

References

Screenshots/Examples (if applicable)

Uh oh!

contextablemark commented Oct 11, 2026

Uh oh!

BlueX888 commented Oct 12, 2026

Uh oh!

contextablemark commented Oct 12, 2026

Uh oh!

BlueX888 commented Oct 25, 2026

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants