Skip to content

fix(anthropic): keep client tool call intact when server tool deltas stream - #23462

Open
BlueX888 wants to merge 2 commits into
run-llama:mainfrom
BlueX888:fix/prep-anthropic-stream-stale-tool-call-corrupted-by-server-tool-delta
Open

BlueX888 wants to merge 2 commits into
run-llama:mainfrom
BlueX888:fix/prep-anthropic-stream-stale-tool-call-corrupted-by-server-tool-delta

Conversation

@BlueX888

@BlueX888 BlueX888 commented Oct 11, 2026 •

Copy link
Copy Markdown

Description

stream_chat / astream_chat in llama-index-llms-anthropic never reset cur_tool_call when a tool block ended. When a streamed turn contains a client tool_use block followed by a server-side tool block (e.g. web_search_20250305 passed via tools, or mcp_servers), the finished client ToolUseBlock stayed assigned to cur_tool_call, so the guard that is meant to ignore server-tool input_json_delta events (llama_index/llms/anthropic/base.py:562-566) did not fire. The server tool's partial JSON was appended to the stale cur_tool_json and written back via cur_tool_call.input (base.py:567-579 on main). For a client tool call that streamed no arguments the write succeeds and the finished call ends up with the server tool's arguments; for one that streamed its own arguments the combined buffer does not parse and the except ValueError: pass keeps the earlier input. The non-streaming path (_get_blocks_and_tool_calls_and_thinking) is not affected.

The fix resets cur_tool_call = None once the finished call has been appended, in the ContentBlockStopEvent branch of both the sync (base.py:615) and async (base.py:873) generators, so the existing guard recognizes the server tool's input_json_delta and ignores it. Client tool calls now keep the input the API delivered.

Fixes #23459

New Package?

Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

  • Yes
  • No

Version Bump?

Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

  • Yes
  • No

Type of Change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

How Has This Been Tested?

  • I added new unit tests to cover this change
  • I believe this change is already covered by existing unit tests

Two mock-based regression tests (test_stream_chat_server_tool_delta_does_not_corrupt_client_tool_call, test_astream_chat_server_tool_delta_does_not_corrupt_client_tool_call) stream a client tool_use block followed by a server_tool_use block whose input arrives as InputJSONDelta, and assert the client tool call keeps its original arguments (tool_kwargs == {}). Both tests fail on current main:

FAILED tests/test_llms_anthropic.py::test_stream_chat_server_tool_delta_does_not_corrupt_client_tool_call
FAILED tests/test_llms_anthropic.py::test_astream_chat_server_tool_delta_does_not_corrupt_client_tool_call
AssertionError: assert {'query': 'weather in London'} == {}

Commands run from llama-index-integrations/llms/llama-index-llms-anthropic:

  • python -m pytest tests/test_llms_anthropic.py -k server_tool_delta_does_not_corrupt — 2 passed (2 failed before the fix)
  • python -m pytest — 54 passed, 15 skipped

At the repo root: uv run make format and uv run make lint — all pre-commit hooks passed (ruff, ruff-format, mypy, codespell, etc.).

Suggested Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added Google Colab support for the newly added notebooks.
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I ran uv run make format; uv run make lint to appease the lint gods

…stream

In stream_chat/astream_chat, cur_tool_call was never reset when a tool
block ended. A following server-side tool block (e.g. web_search via the
tools/mcp_servers args) streams its input as InputJSONDelta, which then
fell through to the client tool-call branch because cur_tool_call still
pointed at the finished client ToolUseBlock. The server tool's partial
JSON was appended to the stale cur_tool_json and, when it parsed,
silently replaced the completed client tool call's input.

Reset cur_tool_call when the tool block stops so that the existing
server-side tool guard recognizes these deltas and ignores them.
…rrupted-by-server-tool-delta

Resolve pyproject.toml version conflict:
- upstream main bumped llama-index-llms-anthropic to 0.12.2 (run-llama#23305,
  Claude Sonnet 5.5 support, already published to PyPI), our branch
  had bumped it to 0.12.1 for this fix
- take upstream's 0.12.2 as the new baseline and bump to 0.12.3 so
  this bug fix ships as a fresh release

base.py auto-merged cleanly; both cur_tool_call = None resets in
stream_chat/astream_chat (the actual fix for stale server-tool
input_json_delta corrupting completed client tool calls) are intact,
as are the added regression tests in tests/test_llms_anthropic.py.

@VANDRANKI VANDRANKI left a comment •

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Community review (not a maintainer, does not clear the merge gate). I read the full diff and ran it.

What I ran, with anthropic 1.13.0 and the integration installed editable from this checkout:

  • The two new tests (test_stream_chat_server_tool_delta_does_not_corrupt_client_tool_call and the async one) pass on the PR branch.
  • I put the PR's tests on top of main's base.py. Both fail there: the client tool call get_current_time ends up with tool_kwargs == {'query': 'weather in London'}, which is the web_search input from the later server tool block. So the bug is real on main and the two-line change fixes it.
  • The full llama-index-llms-anthropic test directory on the PR branch: 54 passed, 15 skipped.

Why it happens, as I read it: cur_tool_call stays set to the finished ToolUseBlock after ContentBlockStopEvent. A later InputJSONDelta from a ServerToolUseBlock therefore passes the isinstance(cur_tool_call, ToolUseBlock) check, gets appended to cur_tool_json, and parse_partial_json writes the result into the old block's input. Resetting cur_tool_call = None at block stop in both gen() functions (sync and async) closes that.

One scoping note from my own probe, so the description is not read too broadly: I also tried a client tool that streams its arguments ({"a": 1} via InputJSONDelta) followed by a server tool. Main did not corrupt that case. I read the reason as cur_tool_json becoming {"a": 1}{"query": "x"}, which parse_partial_json should reject, with the except ValueError: pass keeping the earlier input, but I did not instrument that. So the corruption on main is limited to a client tool with empty input, which is what the PR test uses. Two client tools in a row, and a client tool followed by a text block, gave the same output on main and on the PR.

I did not run against the live API. The events are hand-built from the anthropic types. No issues found with the change.

@BlueX888

Copy link
Copy Markdown
Author

Updated the description to record the scope you found: the overwrite only hits a client tool call that streamed no arguments; when the client tool streamed its own arguments the combined buffer fails parse_partial_json and the except ValueError keeps the earlier input. Package suite at 9f35d61: 54 passed, 15 skipped. Code and head unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: Anthropic stream_chat: server-tool input_json_delta after a completed client tool_use block overwrites the finished client tool call's arguments

2 participants