Skip to content

fix(websearch): sync tool_choice when converting web_search tools#31375

Merged
shivamrawat1 merged 5 commits into
litellm_internal_stagingfrom
litellm_web_search_interception_fix
Jun 27, 2026
Merged

fix(websearch): sync tool_choice when converting web_search tools#31375
shivamrawat1 merged 5 commits into
litellm_internal_stagingfrom
litellm_web_search_interception_fix

Conversation

@shivamrawat1

@shivamrawat1 shivamrawat1 commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

Relevant issues
Fixes #27721
Fixes LIT-4039

Summary
Claude Code talking to LiteLLM proxy over /v1/messages with websearch_interception hit two bugs on the same path: search sub-requests 400'd before any search ran, and streaming clients got application/json back after search instead of SSE, which looked like truncated output. This PR fixes both so native web search works end-to-end through the proxy

Problem
Claude Code uses the Anthropic Messages API with a native web_search tool and often forces it via tool_choice: {"type": "tool", "name": "web_search"}. It also sends stream: true

LiteLLM websearch interception renames that tool to litellm_web_search and runs Tavily (or another configured search tool) inside an agentic loop. Two things broke on that path:

  1. Search never started (HTTP 400)
    Tools were renamed but tool_choice still pointed at web_search. Anthropic returned Tool 'web_search' not found in provided tools. Claude Code showed "Did 0 searches". A second gap made this worse: tool_choice is a named param on anthropic_messages(), not in **kwargs, so the websearch pre-request hook never saw it to rewrite it

  2. Streaming clients got JSON after search ([Bug]: websearch_interception silently truncates streaming response on /v1/messages — follow-up call always uses stream=False #27721)
    The pre-hook correctly converts stream: true → false for the agentic loop and sets websearch_interception_converted_stream so the response can be re-streamed later. After the agentic loop finished, the proxy returned a plain dict (content-type: application/json) instead of SSE. Clients expecting text/event-stream mis-parsed it and answers appeared cut off mid-generation with no error. FakeAnthropicMessagesStreamIterator only ran when the LLM didn't call the search tool; agentic-loop results bypassed the wrap entirely

Fix
Sync forced tool_choice with converted tools
Forward tool_choice explicitly into pre-request hooks. After converting web_search → litellm_web_search, repoint forced tool_choice via _sync_forced_tool_choice() so upstream Anthropic sees a consistent tool name

Re-wrap agentic loop responses as SSE for streaming clients
Add _maybe_websearch_fake_stream_wrap() and call it when the agentic plan returns and on the final handler return. If the client originally requested streaming (websearch_interception_converted_stream on the logging object) and the result is a dict, wrap it in FakeAnthropicMessagesStreamIterator so the proxy emits proper Anthropic SSE

Agentic follow-up calls stay non-streaming internally so LiteLLM can still detect chained tool_use across loop iterations; only the wire format back to the client is converted to SSE

Screenshots / Proof of Fix
Start proxy with websearch_interception and a Tavily search tool configured

Search runs (no 400):

curl -sS -X POST 'http://localhost:4000/v1/messages?beta=true' \
  -H 'Authorization: Bearer sk-1234' \
  -H 'Content-Type: application/json' \
  -H 'anthropic-version: 2023-06-01' \
  -d '{
    "model": "claude-sonnet-4-5",
    "max_tokens": 4096,
    "stream": false,
    "messages": [{"role": "user", "content": "What are the top AI news headlines today?"}],
    "tools": [{"type": "web_search_20250305", "name": "web_search", "max_uses": 1}],
    "tool_choice": {"type": "tool", "name": "web_search"}
  }'

Before: HTTP 400. After: HTTP 200 with web_search_tool_result and real URLs

Streaming wire format (#27721):

curl -N -D /tmp/headers.txt -sS -X POST 'http://localhost:4000/v1/messages?beta=true' \
  -H 'Authorization: Bearer sk-1234' \
  -H 'Content-Type: application/json' \
  -H 'anthropic-version: 2023-06-01' \
  -d '{
    "model": "claude-sonnet-4-5",
    "max_tokens": 4096,
    "stream": true,
    "messages": [{"role": "user", "content": "What are the top AI news headlines today?"}],
    "tools": [{"type": "web_search_20250305", "name": "web_search", "max_uses": 1}],
    "tool_choice": {"type": "tool", "name": "web_search"}
  }'

grep -i content-type /tmp/headers.txt
Before: content-type: application/json. After: content-type: text/event-stream with event: / data: SSE chunks

Files changed
litellm/integrations/websearch_interception/handler.py — _sync_forced_tool_choice()
litellm/llms/anthropic/experimental_pass_through/messages/handler.py — forward tool_choice into hooks
litellm/llms/custom_httpx/llm_http_handler.py — _maybe_websearch_fake_stream_wrap()

Screenshot 2026-06-25 at 5 46 28 PM Screenshot 2026-06-25 at 6 09 09 PM

Claude Code forces native web search via tool_choice pointing at web_search
while websearch_interception renames the tool to litellm_web_search, causing
Anthropic 400s. Forward tool_choice into pre-request hooks and rewrite forced
tool_choice to match the converted tool name.

Co-authored-by: Cursor <[email protected]>
@CLAassistant

CLAassistant commented Jun 26, 2026

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ shivamrawat1
❌ Shivam Rawat


Shivam Rawat seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@greptile-apps

greptile-apps Bot commented Jun 26, 2026

Copy link
Copy Markdown
Contributor

Greptile Summary

This PR fixes two bugs on the websearch interception path for Anthropic /v1/messages requests: a 400 error caused by tool_choice naming a tool that had been renamed, and streaming clients receiving application/json instead of text/event-stream after the agentic search loop completed.

  • tool_choice sync: tool_choice is now forwarded explicitly into _execute_pre_request_hooks (it was a named param that hooks never saw), and _sync_forced_tool_choice rewrites it to litellm_web_search after the tool definition is converted, so Anthropic receives a consistent tool name.
  • SSE re-wrap: _maybe_websearch_fake_stream_wrap is extracted as a shared static helper and applied at both exit points of the agentic loop in _call_agentic_completion_hooks (plan execution and no-loop fallback) as well as in async_anthropic_messages_handler, ensuring streaming clients always get SSE regardless of which code path produced the final response.

Confidence Score: 5/5

Safe to merge — both fixes are narrowly scoped to the websearch interception path and do not touch non-websearch requests.

The two code paths changed are guarded by the websearch_interception_converted_stream flag (for the SSE re-wrap) and by the presence of web-search tools in the request (for the tool_choice sync), so neither change activates on ordinary requests. Double-wrapping is prevented by the __aiter__ check. Unit tests cover all new logic, are mock-only, and do not weaken any existing assertions.

No files require special attention.

Important Files Changed

Filename Overview
litellm/integrations/websearch_interception/handler.py Adds _tool_name and _sync_forced_tool_choice helpers; calls the latter inside async_pre_request_hook after tool conversion. Both methods are straightforward and well-tested.
litellm/llms/anthropic/experimental_pass_through/messages/handler.py Forwards tool_choice explicitly into _execute_pre_request_hooks so it lands in hook kwargs and can be rewritten; extracts it back via request_kwargs.pop. Change is additive and doesn't affect callers that don't set tool_choice.
litellm/llms/custom_httpx/llm_http_handler.py Extracts stream-re-wrapping logic into _maybe_websearch_fake_stream_wrap and applies it in both the post-agentic-plan path and the non-agentic fallback path. Double-wrapping is guarded by __aiter__ check; behavior for non-websearch requests is unchanged (flag not set → passthrough).
tests/test_litellm/integrations/websearch_interception/test_websearch_interception_handler.py Adds five new unit tests covering _sync_forced_tool_choice (forced repoint, untouched, extra fields, non-forced passthrough) and an end-to-end hook test. All tests use mocks — no real network calls.
tests/test_litellm/llms/custom_httpx/test_llm_http_handler.py Adds four unit tests for _maybe_websearch_fake_stream_wrap covering the wrap, no-op (flag unset), no-op (already streaming), and no-op (None logging_obj) paths. All mock-based.

Reviews (3): Last reviewed commit: "test(websearch): cover tool_choice sync ..." | Re-trigger Greptile

Comment thread litellm/integrations/websearch_interception/handler.py
Comment thread litellm/integrations/websearch_interception/handler.py
@codecov

codecov Bot commented Jun 26, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

Shivam Rawat and others added 2 commits June 25, 2026 18:06
…lients

When websearch interception converts stream=true to false for the agentic
loop, dict responses from the loop were returned as application/json even
though the client requested SSE. Wrap those responses in
FakeAnthropicMessagesStreamIterator so /v1/messages streaming callers
(e.g. Claude Code) receive text/event-stream after search completes.

Fixes #27721

Co-authored-by: Cursor <[email protected]>
…P006

Add regression tests for both websearch interception fixes: _sync_forced_tool_choice
repointing a forced web_search tool_choice to litellm_web_search (the 400 fix) and
_maybe_websearch_fake_stream_wrap re-wrapping agentic loop dict responses as SSE for
streaming clients (#27721). Switch the new helper annotations to builtin dict/list so
the ruff UP006 strict-rule ceiling stays within budget.

Co-authored-by: Cursor <[email protected]>
@shivamrawat1

shivamrawat1 commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator Author

@greptile-ai review again holistically.

@shivamrawat1

Copy link
Copy Markdown
Collaborator Author

@greptile-apps review again holistically with all commits

@mateo-berri mateo-berri left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM; thanks!

shivamrawat1 and others added 2 commits June 26, 2026 19:23
Remove the duplicate _maybe_websearch_fake_stream_wrap helper left by a bad merge that caused a SyntaxError in CI, and route all call sites through _maybe_wrap_in_fake_stream instead.

Co-authored-by: Cursor <[email protected]>
@shivamrawat1 shivamrawat1 merged commit de82f78 into litellm_internal_staging Jun 27, 2026
119 of 121 checks passed
@shivamrawat1 shivamrawat1 deleted the litellm_web_search_interception_fix branch June 27, 2026 02:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: websearch_interception silently truncates streaming response on /v1/messages — follow-up call always uses stream=False

3 participants